GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning
Authors: Longxi Gao
Li Zhang
Pengzhi Gao
Wei Liu
Jian Luan
Mengwei Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Mobile GUI Agent
Reinforcement Learning
Vision Language Model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Training effective Vision-Language Models (VLMs) for GUI agents typically depends on large-scale annotated datasets, whose collection is both labor-intensive and error-prone. We introduce ***K***-step GUI Transition, a self-supervised inverse dynamics task in which VLMs learn GUI dynamics by predicting the initial action that causes a transition between two GUI states. This approach eliminates the need for natural language instructions and enables scalable dataset construction from existing GUI trajectories or automated exploration. Building on this task, we propose **GUI-Shift**, a reinforcement learning (RL) framework that combines rule-based optimization with data filtering to improve VLM performance. We conduct extensive experiments using multiple VLM backbones across five benchmarks, spanning GUI task automation (AndroidControl, GUI Odyssey, AndroidWorld) and GUI grounding (ScreenSpot-v2, ScreenSpot-Pro). Our results show that training on GUI-Shift generalizes well to both GUI automation and grounding tasks, yielding up to an 11.2% increase in GUI automation accuracy. This study underscores the potential of self-supervised RL to leverage unlabeled GUI trajectories and offers a scalable alternative to training with annotated samples. GUI-Shift will be open-sourced at: https://github.com/UbiquitousLearning/GUI-Shift.

Published as a conference paper at ICLR 2026
GUI-SHIFT: ENHANCING VLM-BASED GUI AGENTS THROUGH SELF-SUPERVISED REINFORCEMENT LEARN-
ING
Longxi Gaoi, Li Zhang', Pengzhi Gao", Wei Liu", Jian Luan', Mengwei Xu Beijing University of Posts and Telecommunications >Independent Researcher g x@obpt....... edu.cn
ABSTRACT
Training effective Vision-Languaan Models (VLMs) for GUI agents typically de- pends on large-scale annotated datasets, whose collection is both labor-intensive and error-prone. We introduce K K-step GUI Transition, a self-supervised inverse dynamics task in which VLMs Iearn GUI dynamics by predicting the initial action that causes transition between two GUI states. This approach climinates the need for natural language instructions and enables scalable dataset construction from ex- isting GUI trajectories automated exploration. Building on this task, we propose GUI-Shift, a reinforcement learning (RL) framework that combines rule-based op- timization with data filtering improve VLM performance. We conduct extensive experiments using multiple VLM backbones across five benchmarks, spanning GUI task automation (AndroidConttrl, GUI Odyssey, AndroidWorldy World) and GUI ground- ing (ScreenSpot-v2, ScreenSpot-Prod Our results show that training on GUI-Shift generalizes well both GUT automation and grounding tasks, yielding up to an 11.2% increase in GUI automation accuracy. This study underscores the potential of self-supervised RL to leverage unlabeled GUI trajectories and offers a scalable alternative to training with annotated samples. GUI-Shift will be open-sourced at: com http:///ggtttoooooooooooooooooan
INTRODUCTION
Mobile GUT agents (Gou et al. 2024 Hong et al. 2024 Qin et al. 2025 Wen et al. 2024 Yang et al. 2024) interpret natural language instructions and perform actions (e.g., click, scroll) directly on smartphone screens. They can control diverse apps as a human would, improving accessibility for users who are visually impaired, elderly. or have their hands occupied. Breakthroughs of vision language models (VLMs) (Bai et al., 2025 Chen et al. 2024 Xiaomi 20256) have reshaped the design paradigm of mobile GUI agents, transitioning from handcrafted heuristics to learned, vision- grounded policies. However, VLMs still struggle to deliver satisfactory accuracy (Dai et al.) 2025
Qin al. 2025 Rawles al. 2024 Zhang et al.) (025a)) especially when facing complex multi- step tasks. A common approach for enhancing VLMs is through supervised fine-tuning (SFT) on datasets containing GUI interaction trajectories paired with annottted task instructions (Li et al. 2024. Rawles et al., (023). Yet effective, collecting GUI trajectories with task instructions remains labor-intensive and error-prone (Deka et al. 2017 7, Rawles et al. (023). For example, the
AndroidConttol (Li et al. 2024) dataset takes one year of paid annotation effort to produce just 15,283 task demonstrations. Such high annotation cost limits the scalability of this paradigm.
In this study, we aim to address a fundamental challenge: how to train capable mobile GUI agents using large-scale, unlabeled GUI trajectories, rather than relying on costly humun-annotated in- structions. To tackle this, we propose a self-supervised training task, termed K K-step GUT Transition. Inspired by inverse-dynamics modeling in robotics and biomechanics (Brandfonbrener et al. 2023 Tian et al. 2024| Zapolsky & Drumwright 2017), where model predicts control commands linking two consecutive physical states, our task treats screenshots as states and GUI actions as commands. Each training sample in K-step GUI Transition consists of two screenshots, S, and Setk, where
"Corresponding author.

---

Published as a conference paper at ICLR 2026
Task Comparison
Learning Process
Self-supervised Reinforcement Learning
Sh stop GUI Transhion
Implidtly Learning Stenal from GUI Dyramics am given two screenshots.
Training Promot
You are given two screenshets:
the state 5(1);
The first image shows a shopping app's category view many products.
a future state S(++k) after k consecutive actions from Stt). Your task to infer the first GUI action that initiates the
transition from towerd S(t=k).
Return the action insice <answers..</answee
The second image is the detail page of specific product- - "Jumanji The Game in Roal Wooden Box".
Data Fitteingg S
Current state: STO
Targer state: STEAk)
Visual Language Madel
k-stop away screen the visual gool. Annototion-free freel
Viston Encodor
Projector
Language Model
So, the gool the transition to neach detail page.
Undate
Rollout N Resporses
canswers aaction /cddk: 5500 "y" 1800//arswer
Action Predistion
conswor: wwtctatoottoeeooooooooao htppp////p/ppp/pppppppppo/p "X" 1003, V 2041s/bnover>
Instruction-eppeen dependent Task
sshuld click search and type the product name. 5o, think firtt step should ber click the search box.
View the detail page of the product:
canswers "action wectiin_typee/ooe hyppe// "Y" 550 210
"Jumanji The Game in Real Woo dan Box"
Human/Al annotated: high cost
canswon: "wwctoocctttt""""" "screl" "dinection" upls)answer>
Figure I: Overview of the GUI-Shift framework. Left: K-step GUT Transition replaces annotated instructions with the target state St+k. enabling scalable data construction through automated offline exploration. Middle: The model learns GUI dynamics by predicting the action that causes the transition. Right: GUI-Shift achieves superssud training by applying GRPO GUI Transition.
S1+k results from executing K actions starting from St. The VLM is trained to predict the first action that transforms S₁ into St-1. This design offers two key advantages: (1) Explicit state-change signal. Each sample contains a pair of GUI screenshots, enabling the model to utilize inter-screen visual differences and temporal cues, rather than learning from single screen. (2) Efficient data utilization at scale. Since ground-truth actions are embedded in GUT trajectories (Rawles et al.) [0033]
Li al. 2024), no predefined instructions or manual annotations are needed. Moreover, for any k. a GUI trajectory with n screens can yield up to n k training samples, enabling scalable data construction. These benefits make K Ktsp GUI Transition a strong candidate for self sulf-supervised GUI agent training.
With the sulf-supervised training task, it is essential to determine how to effectively enhance VLMs. In GUT tasks, multiple action parameters can often be functionally equivalent and result in the same next state. For example, any coordinate within a button's 's bounding box valid for a click, and textual inputs may be accepted in various formats or with different keywords. This multiplicity makes supervised fine-tuning (SFT) suboptimal, as it enforces a single reference action in a static dataset through the cross-entropy loss, penalizing all other valid alternatives and therefore providing misleading learning signals. To address this limitation, we adopt Group Relative Policy Optimization (GRPO) (Shao et al. 2024), which samples diverse plausible actions, evaluates them using a task- specific reward function, and ranks them based on group-normalized advantages. For example, in click actions, rewards assigned if the sampled point lies within the target bounding box, offering more tolerant and informative optimization signal. Overall, GRPO provides a more effective training paradigm for GUI agents by encouraging exploration and increasing robustness action variability.
To this end, we present GUI-Shift, a self-supervised reinforcement learning (RL) framework that applies GRPO K-step GUI Transition. Figure TaaueeTilluua Iillustrates the overview of the GUI-Shift framework. To select data matched to the model's 's learning ability, we adopt a unified action sampling and scoring mechanism during both data filtering and training stages. For each sample, the VLM generates N action predictions, each scored based on format and action correctness. Only samples containing both correct and incorrect predictions among the N predictions are selected for training. After training, the VLM acquires GUI-specific capabilities and serves
more effective backbone for GUI agents. VLMs enhanced with GUI-Shift also have the ability to generalize well to GUI task automation and GUI grounding tasks without further alignment fine-tuning.
We apply GUI-Shift to train four VLMs: Qwen2.5-Ve 5L-7BB (Bai et al. 2025) Intern VL3-8B (Chen et al. 2024), Mimo VL-7B-SFT (Xiaomi (025b), and Mimo VL-7B-LLL RL. each using 2K samples for four K -step GUI Transition variants (K E 21,2, 3, 4)). We evaluate the VLMs on five benchmarks: AndroidControl (Li et al. 2024), GUI Odyssey (Lu et al. 2024), and AndroidWorld (Rawles et al.
2

---

Published as a conference paper at ICLR 2026
2024) for GUI task automation, and ScreenSpot-v2 (Wu et al. 2024b) and ScreenSpot-Prr (Li et al. 2025a for GUI grounding. Overall, VLMs enhanced with GUI-Shift show notable improvements over their base versions. For example, GUI-Shift-Qwen achieves up to 1 11.2% higher accuracy AndroidCoontrol-Higll and 2.5% on ScreenSpot-v2, yielding 70.4% and 89.0% overall accuracy on the respective benchmarks. We also conduct comprehensive ablation studies to examine the effects of data filtering, task formulation, reasoning configurations, and training paradigms. Results show that using the target state S as visual instruction offers an effective alternative human- or Al-annotated textual instructions for training GUI agents. Our contributions are summarized below.
(1) We introduce K-step GUI Transition, a training task that leverages abundant unlabeled GUI trajectories to enhance VLMs used in GUI agents.
(2) We propose GUI-Shift, a self-supervised RL framework that bridges the gap between GUI dynamics modeling and action-level GUI learning, mitigating the limitation of SFT in handling action multiplicity and poor generalization in GUI tasks.
(3) Experiments across four VLMs and five benchmarks show that VLMs enhanced with GUI- Shift exhibit generalization in both GUI automation and grounding tasks, with up to 11.2% accuracy gains.
2 RELATED WORK
2.1 MOBILE GUI AGENTS
Recent progress in mobile GUI agents has been driven by VLMs trained via SFT on large-scale datasets. These models learn to map instructions to GUI actions using fooluuttoon-oolooiing tasks, making high-quality annotations essential. Despite the availability of diverse GUI datasets (Deka et al., 2017 Gao al.] 2024]] Li et al. 2024 Lu et al. 2024 Rawles et al. 2023), the quantity of high-quality annotations remains insufficient for robust training and usually requires significant human effort to scale. To reduce annotation costs, prior pipelines often incorporate out-of-domain image-caption pairs (Hong et al. 2024 Wang al. 2024) and supplement training with web and desktop data to improve pross-plations generalization (Cheng et al.) (024). As a result, the overall scale of training data tends be large: Uground (Gou et 2024) uses 1.3M screenshots to train visual grounding model; OS-Atlas (Wu et al. (0224b) leverages 13M GUI elements for grounding pretraining. Some recent approaches have explored GUI state modeling. UI-TARS (Qin al. 2025) incorporates a state transition task, which focuses on describing visual changes between screenshots rather than predicting the underlying actions, resulting in gap with GUI task automation. Mobile VLM (Wu et al.) (024a) introduces an action prediction task between screenshots, but is restricted to one-step transitions and fine-tuning for downstream alignment and generalization. In
SFT. They still rely on annotation- this work, we propose K K-step GUI Transition, formulating k-step inverse dynamics objective that enables scalable training on large, unlabeled, and underutilized GUI datasets.
2.2 RULE-BASED REINFORCEMENT LEARNING
Rule-based RL has proven to be promising alternative to SFT. GRPO (Shao et al. 2024) uses re- ward model to score each response and computes group relative advantages instead of training critic model, whose size is comparable to the policy model, thereby significantly reducing computational cost. Reinforcement Learning with Verifiable Rewards (Lambert et al. 2024) further emphasize the use of verifiable answers to design reliable reward signals. DeepSeek-RI (Guo et al.) 2025) shows that simple format and accuracy rewards are sufficient to surpass the performance of instruction-tunee models. Several recent works have applied GRPO on GUI tasks: UI-RI (Lu et (202)) employs one-stage RL on 136 samples with step-level instructions. GUI-RI (Xia & Luo (025) expands this to 3K task-level instructions from five platforms. InfiGUI-RI (Liu et al.] 2025 adopts two stage SFT+RL pipeline and scales to 32K samples from both GUI and non-GU domains. UI-Venu (Gu) ct al. 2025) employs GRPO to two variants, using 107K samples for grounding and 350K for naviga- tion. While these works demonstrate the effectiveness of rule-based RL for GUI agents, they still rely on annotated instructions and require reasoning during training and inference. Different from these annotation-depent dependent training paradigms. GUI-Shift fine-tunes VLMs via one-stage RL on K K-step GUI Transition in a self-supervised manner, achieving competitive performance and demonstrating strong generalization across GUT task automation and GUI grounding benchmarks.
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
GUI-Shift a self-supervised RL framework designed to enhance VLM-based GUI agents through the K- K-step GUI Transition task. In this section, we first describe GRPO, the underlying training algorithm GUI-Shift. We then detail the reward design tailored to GUI action modeling, and present the complete GUI-Shift framework along with its rationale and advantages.
3.1 PRELIMINARIES
GRPO (Shao et al. 2024) offers a computationally efficient alternative to Proximal Policy Optimiza- tion (PPO) (Schulman et al., 2017). widely used actor-critic method. Instead of maintaining separate critic network for value estimation, GRPO computes normalized, group-wise advantages A; directly from reward scores, thereby removing the value function update and lowering computational cost. The GRPO objective in our framework is defined as follows:
(o) E P(Q), (o:) To.m (0 [0]
(min (ps As, clip (ps, - 1 +e) A.) - DKI. (Dor((TollTTet) (1)
To(o; g) Toada (o: 1))
A = fi memn((r)))) 12, ,ra))
where
std(fri,
.ra)
Specifically, for each question q in the training set, we sample a group of outputs {or ......, oG} from the old policy To... using high temperature decoding, and compute the group-wiss soup-wisee/o relative advantage A; for each output. clipped surrogate objective, along with KL divergence regularizer toward the reference policy Tefs is then used to update model parameters and ensure training stability.
3.2 REWARD DESIGN
The reward function plays a central role in guiding and stabilizing model optimization. In GUI action prediction, each answer is a structured action comprising verifiable action type and associated parameters, making the task well-suited to rule-based reward formulation. Following DeepSeek- R1 (Guo et al. 2025), we adopt mule-based reward tailored for GUI tasks, which combines format reward Ry to enforce output consistency and an action reward R. to evaluate action correctness:
= R + Rs
Format reward. To ensure that model outputs are well-structured and easy to parse, GUI-Shift requires the final answer to be enclosed in <answer> </ </answer> tags during training. Pre- dictions conforming to the expected format receive R 1; otherwise, R2 = 0. Unlike prior methods (Lu et al. 2025 Liu et al. 2025 Xia & Luo| 2025). GUI-Shift omits explicit reasoning traces in outputs, eliminating reasoning token generation and substantially reduces training time. For example, training Owen2.5-VLL-7B
2K K-step GUI Transition samples requires only 9 hours, compared to 17 hours with reasoning traces, without compromising downstream performance, as shown in Table
Action reward. We adopt a unified action space of eight types for both training and inference. The action space comprises eight types, each JSON object with action_lype and type-specific parameters: click and long_pnn press require target point; scroll requires direction; open_app requires an app name; input_tee text requires the input content; and navigate back, navigate_ home and wait require no parameters. The action reward R. is defined accordingly:
if S & < 22 and yı < ÿ S V2, t {click, long_press);
1, i = 1 and
L {open_app, input_text, scroll);
Rs
if =t, = {navigate_ _bck, navigate home, wait);
0. otherwise.
Here, and P denote the predicted action type and parameter, and p denote their ground-truth counterparts; J. ÿ the predicted coordinates, and [x),, 21, y2] is the ground-truth bounding box.
4

---

Published as a conference paper at ICLR 2026
3.3 GUI-SHIFT FRAMEWORK
K-step GUI Transition. While existing VLMs can parse individual GUI screens due to exposure to GUI data during pretraining, they still lack the temporal reasoning capabilities required for complex multi-step GUI tasks. To bridge this gap, we propose the K-step GUI Transition task, which asks the model to predict the first action that transitions given state to a future state Si+k. as shown in Figure _ Compared to annotated approaches, our task offers two key advantages. First, while annotated tasks require costly and error-prone textual annotations for each step, K K-step GUI Transition leverages state pairs directly extracted from GUI trajectories. The future state Si+K. obtained after executing K actions from S,, serves as an explicit visual goal, providing a supervision signal that is not only annotation-free but also more concrete and informative than textual instructions. Second, rather than mapping textual instructions to actions, our task compels the model to interpret and compare both the current and target GUI states, infer the transition goal, and identify the action that initiates the state change. Overall, by leveraging visual goals and requiring temporal reasoning across state pairs, this more challenging formulation fosters deeper understanding of GUT dynamics and provides a scalable, practical solution for robust GUI agent training.
Supe-supe RL. During training, for each sample, the model generates N candidate actions (N = 8 in our experiments), each evaluated by a rule-based reward that integrates format and action correctness, as detailed in Section 3.2 Group-wise normalized advantages are then computed, and optimization proceeds as outlined in Section 3.1 GRPO is particularly well-suited to GUI-Shift for three reasons: (I) Compared to PPO, eliminates the need for a separate value function, typically in the same size as the policy model. This substantially reduces computational overhead and better supports our efficiency objectives; (2) Compared SFT, it enables flexible reward assignment tailored to each action type. For instance, click actions are considered correct if the predicted point falls within the ground-truth bounding box rather than requiring an exact match in SFT, which better reflects practical GUI grounding requirements; (3) The N -ccandidate sampling mechanism encourages exploration and model can learn from optimal candidates while avoiding suboptimal ones.
Data filtering pipeline. To prepare high-quality K-step GUI Transition data, we perform data filtering using the same action sampling and scoring mechanism as in training. First, for each K (1, 2, 4), we construct pool of candidate state pairs (S,, St+k) from the original dataset. Next, for each pair, the model generates 8 responses using the same sampling temperature as in GRPO training. Each response then evaluated with the reward function described in Section (.2 Finally, we retain only those pairs with both correct and incorrect responses. By applying this filtering process each model independently, the final training set is both challenging and informative, and well aligned with the model's 's learning capacity.
Taken together, these design choices enable GUI-Shift to provide higher efficiency from three aspects: (1) Scalable data construction. Without relying on annotated instructions, GUI-Shift enables large- scale filtering of training data at minimal cost. For example, for MimooVL--BB-RE
we filtered 2,920
high-quality samples out of 8K original 1-step GUI Transition pairs, without any annotation waste. (2) Maximized data utilization. For each K, an N -immage trajectory can yield up to N K training pairs, maximizing data utilization for fixed-length GUI trajectories. (3) Reduced training cost. Without explicit reasoning traces during training, GUI-Shift avoids extra token decoding and reduces training time by nearly 50%, from 17 to 9 hours on 2K samples under our experimental setup.
4 EXPERIMENTS
In this section, we first detail the experimental setup, including data construction and training configurations (Section 4. We then evaluate models trained with K-step GUI Transition (k e (1,2,3, 43) on static GUI task automation and grounding benchmarks (Section 4.2) as well as end-to-end GUI task automation benchmarks (Section 4.3) Finally, conduct comprehensive ablation studies to analyze key design choices from four perspectives: data filtering, task formulation, reasoning configurations during RL., and training algorithms (Section 4.4
4.1 EXPERIMENTAL SETUP
Training configurations. Using the open-source VLM-RI (Shen et al.
2025 framework, we
fine-tune Qwen2.5-VI--B (Bai et al. 2025). Intern (Chen et al. 2024 MImoVL.-7B-
5

---

Published as a conference paper at ICLR 2026
SFT (Xiaomi 2025b), and MimoVL-7B-RD with the pipeline described in Section During training, only the language model is optimized while the vision encoder and projector are frozen. All experiments conducted on 8XNVIDIA H100 GPUs. Hyper-parameters are listed in Appendix B
Data construction. All training data are sourced from the training set of AndroidControl (Li ct al. 2024), which provides GUI trajectories paired with human-labeled instructions. These instructions enable both self-supervised GUI-Shift training and comparisons with VLMs trained using SFT (see 1.4tion44.400 for comparison of the two training approaches). Following the data filtering pipeline discussed in Section 3. we select 2K samples per K for each model. For Qwen2.5-VL-77 the proportion of samples with either entirely correct or entirely incorrect actions was exceptionally high. As result, we use unfiltered data for its training.
Section
4.2 STATIC BENCHMARKS AND RESULTS
task automation benchmarks: AndroidCon- trol (Li et al. 2024) and GUI Odyssey (Lu et al. 2024). AndroidControl provides two test settings: AndroidControl-Lonn which assesses step-level instruction following ability (e.g.. "Type the product name in the search box"), and AndroidControll- Hiigh which evaluates long-horizon task planning (c.g., "View the detail page of the product"), GUI Odyssey offers a more challenging evaluation, encom- passing both phone and tablet applications as well as cross-app scenarios. The test set includes 9,134 samples in AndroidControl and 27,493 in GUI Odyssey, covering six action types: click, song_prees scroll, navigate_ b_cck,_c,, navigale soteeee_ee,____ and input_lete _ext_ We report type match (TM) that represents the proportion of samples with the correct action type, and exact match (EM), which requires both the action type and all parameters to be correct. Metrics are computed using AgentCPM-GUI ZZhang cct al. 2025b) and GUIEvalKit (Xiaomi 2025a).
GUI task automation. We evaluate GUI-Shift
Table 1: Performance comparison on GUI task automation benchmarks: AndroidControl (AC-Low, AC-High) and GUI Odyssey. GUI-Shift achieves substantial improvements base models. Bold: the best result; underlined: the second best result. TM: type match; EM: exact match.
AC-Low
AC-High
GUI Odyssey
Training Samples
Model
TM
EM
TM
EM
TM
EM
Proprictary models
GPT-4o OOpenAl 2024
74.3
19.4
66.3
20.8
34.3
3.3
Models trained with annotations
SeeClick Cheng et al. 2024
1M
93.0
75.0
82.9
59.1
71.0
53.9
OS-Atlas-75 /B
al. 2024b
2.3M
93.6
85.2
85,2
71.2
62.0
Aguvis-7B Xu et al. 2024
1M
80.5
61.5
UI-TARS-75 - Qin al.
2025
98.0
90.8
83.7
72.5
94.6
87.0
UI-RI-3B Lu et 2025
136
88.5
57.9
45.4
$2.2
32.5
GUI-R1-7B [iaa & Luo.
2025
3K
85.2
66.5
71.6
51.7
65.5
38.8
InfiGUI-R1-BB (Lu al.
2025
32K
96.0
92.1
82.7
71.1
AgentCPM-GGU (Liu et al.
2025
470K
94.4
90.2
77.7
69.2
90.9
75.0
Ul-Venus-Navi-7B (Gu
2025
350K
97.1
92.4
86.5
76.1
87.3
71.5
Ours:
Qwen2.5-VL-77. as the base model
Qwen2.5-VL-7B Bai al.
2025
94.9
83.8
72.9
59.2
59.8
44,9
GUI-Shift-Qwen (k
2K
98.013.1
90.6168 85.9113.
70.4111.
78.5
54.8-9)
Ours: Intern VL3-8B as the base model
Intern VL33-B (Chen al.
2024
97.8
90.0
71.5
49.8
48.8
20.3
GUI-Shift-Intem (I 4)
2K
97.3
88.0120
78.5t7.
56.6168
59.6r108
23.3r3.
Ours: Mimo- VL-7B-SFT as the base model
Mimo-VL-7B-SFI Xiaomi GUI-Shifl-MmmoSSH (k
2025b
90.8
85.7
75.2
63.1
86.9
62.0
2K
98.6178
93.217. 87.2
73.4
86. 1
60.7.13
Ours: Mimo VL-7B- RL-RLAR as the base model
Mino-VVL-7BRRL (Xiommi SUU--hhft-Mimoe RL (Is
2025b
91.8
87.2
76.5
64.6
87.2
63.1
2K
98.9
93.216.0 86.9
71.7
84.812.4
59.5.36

---

Published as a conference paper at ICLR 2026
GUI-Shift generally improves performance over base models GUI task automation benchmarks. Table I presents the results of GUI-Shift, together with comparisons against both base models and models trained with annotations. GUI-Shift achieves notable gains across all four models, especially on AndroidControl-Higge GUI-Shift- wwen raises EM by 11 1.2% over Qwen2.5-VL.-75. while GUI- Shift-Mimo-SFi and GUI-Shift-Mmmo-RI reach gains of 10.3% and 7.1 7.1%, respectively. On GUT Odyssey, minor declines for GUI-Shift-Mimo-S the 1,381 tablet episodes in the test set, whose GUI layouts differ substantially from smartphones. Compared to models trained with annotations, GUI-Shift achieves comparable or even superior results on all benchmarks using only 2K stepp GUI Transition samples. Overall, these results underscore the robustness and effectiveness of our approach in GUI task automation.
and GUI-Shift-Mimo-RA RL likely result from
GUI grounding. We evaluate GUI-Shift on two GUI grounding benchmarks: ScreenSpot-v2 Wu 2024b) with 1,272 samples from mobile, desktop, and web platforms, and ScreenSpot (Li
et al.
al. 2025a with 1,581 high-resolution screenshots for fine- fine-grained evaluation. Evaluations for
base models and GUI-Shift are adapted from
(Li et al. 2025b).
GUI-Shifi consistently outperforms base models and surpasses most existing baselines on GUI grounding benchmarks. Table summarizes the overall and baseline results. Across all models, GUI-Shift delivers improved accuracy over base models, with the best variants reaching 2.5% and 1.5% gains on ScreenSpot-v2 and ScreenSpot-Pro respectively. Moreover, GUI-Shift surpasses all annotation-traiiee models except UI- Srruus-Ground7755 which is trained specifically for the GUI grounding using 107K annotated samples. These results demonstrate that models trained solely on unlabeled GUI Transition data can effectively transfer to challenging GUI grounding tasks.
Table 2:
Performance comparison on GUI grounding benchmarks: ScreenSpot-v2 and ScreenSpot- Pro. GUI-Shift exhibits strong generalization and achieves the second best result ScreenSpot-Pre Bold: the best result; underlined: the second best result.
ScreenSpot-r2
-Pro
Training
Mobile
Desktop
Web
Model
Samples
Avg.
Avg.
Text Icon
Text
Icon
Text
Icon
Models trained with annotations
CogAgennt-Je MeAAgent-88BB 8B
Wang et al.
2024
7.7
SeeClick-9.6B
Cheng al.
2024
1M
78.4
50.7
70.1
29.3
55.2
32.5
55.1
1.1
UGround-7B Gou
2024
1.3M
75.1
84.5
85.1
61.4
84.6
71.9
76.3
16.5
OS-Atlas-7B (Wu et al,
2024b
2.3M
95.2
75.8
90.7
63.6
90.6
77.3
84.1
18.9
ShowUI-2B Lin et al.
2024
256K
7.7
UI-TARS-7B (Qin el al.
2025
96.9
89.1
95.4
85.0
93.6
85.2
91.6
35.7
UI-RI-E-3B (Lu al. 2025
2K
83.0
97.1
85.0
91.7
77.9
95.4
89.5
33.5
InfiGUI-R--3B [Luu et al. 2025
32K
35.7
LPO Tang et al. |
2025
97.9
82.9
95.9
86.4
95.6
84.2
90.5
UI- Ul-Venus-Ground- Ground-rodd--111
Gu et al.
2025
107K
99.0
90.0
97.0
90.7
96.2
88.7
94.1
50.8
Ours: Qwen2.5 VL-7B as the base model
Qwcn2.5-VL--7B (Bai et al.
2025
98.3
86.3
88.7
67.1
92.7
81.8
87.7
26.4
GUI-Shift-Qwen (1 = 4)
2K
98.6
89.6
86.1
75.0
92.7
82.8
89.0m1.
27.1.u7
Ours: Intern VL3-8B as the base model
IntemVL3--8 (Chen et al. GUI-Shift-Inter (k =
2024
93.4
81.5
80.4
52.1
91.0
73.4
81.3
15.0
2K
93.8
83.4
80.4 51.4
91.0
73.4
81.6103
15.4-0.4
Ours: Mimo VL-7B SFT as the base model
Cimo-VIL-7B-SFT Xiaomi
2025b
96.6
84.4
92.8
80.0
88.9
76.8
87.6
39.8
GUI-Shift-Mimo Minbft--Mio-FFFUk
2K
98.3
87.7
92.3 82.1
94.0
79.8
90.1
40.7-0.9
Ours: Mino- VL-7B-RL as the base model
Mimo-VLL VL-7B-RL Xiaomi 2025b
98.3
86.3
90.2
80.7
92.7
75.4
88.4
40.2
GUI-Shift-Mine RL (R
2K
99.0
87.7 91.2 83.6
89.7
72.9
88.4
41,7-1.5
Detailed results for different K values are provided for GUT task automation and GUI grounding benchmarks in Appendix and Appendix respectively.
7

---

Published as a conference paper at ICLR 2026
4.3 END-TO-END BENCHMARKS AND RESULTS
Evaluation setup. Beyond the single-step evaluation on AndroidControl described in Section 4.2 additionally evaluate end-to-end performance. An episode is considered successful only if all steps are correct in both action type and parameters; otherwise, the entire trajectory is counted as failure. This stricter metric better reflects realistic GUI task automation. We also evaluate GUI-Shift on Android World (Rawles et al. 2024), more challenging interactive benchmark consisting of 116 tasks. We follow the official M3A agent protocol, where the model receives screenshots annotated with set-of- mark (Yang et al.) 2023) and generates one action and summary at each step. We evaluate Mimo- VL-7B-SFT and GUI-Shift-Mimo
(K E (1,2,3,4 4)) under this configuration.
Table 3: End-to-end performance AndroidControl (AC-Low, AC-High) across base models.
GUI-Shift improves over the base models in most cases. Bold: the best result.
Qwen2.5-VL. VL-7B
Intern VL3-8B
Mimo-VL-7B-SFY
Model
AC-Low
AC-High
AC-Low
AC-High
AC-Low AC-High
AC-Low
AC-High
Base
50.2
22.4
68.3
15.4
48.4
16.4
53.3
18.0
K=1
67.5
29.917.5
60.717.6
18.9
72.41240
32.11157
76.3-23.0
32.2/14.2
K=2
65.1
29.9
60.3
19.2
73.9
33.1116.7
71.8-18.5
34.6116.6
K=3
69.3+19.1
29.417.
61.0,7.
19.2
75.7
34.1
71.0~177
33.6115.6
K=4
67.6
28.716.3
61.1
21.9
74.2125.8
31.1
69.5-162
33.7
Pass @1
Pass @
Pass@5
30
(a)
(b)
(c)
25.9
25
24.1
19.0
20.7
19.8
18.1
21.6
21.6
15,5
14.7
14.7
12.9
Success
10
5-
0
Base
K=1
K-2
K=3
K=4
Base
K=1
K-2
K=3
K=4
Base
K=1
K=2
K=3
K=4
Model Variants
Figure 2: Performance AndroidWorld of the base model Mimo-VL-7B VL-7B-SFT and GUI-Shift-Mime SFT with different K. Evaluation follows the original M3A agent protocol. GUI-Shift consistently improves success rates across Pass@1 1, Pass@3, and Pass@5.
GUI-Shift improves end-to-end performance across benchmarks. Table presents end-to-end performance on AndroidControl, where GUI-Shift improves over the base models in most cases. For example,
improves from 48.4% 44 75.7% AndroidControl-LLovv and from 16.4% to 34.1% % on AndroidCControl-Higgh Mimo- VL--BBRI. and Qwen2.5-V1-77B also show gains across both settings. For EntemVL3-88 performance decreases on AndroidCControl-Lovd compared to the base model, while increasing on the more challenging AndroidConttol-Higg from 15 55,4% to 21.9%. Figure illustrates consistent improvements on AndroidWorld under Pass @ 1, Pass @ and @ GUI-Shift-Mimo-SS and from 18.14 % to 25.9% for K 4 at Pass @ These results indicate that GUI-Shift improves performance not only in single-step accuracy but also in end-to-end performance, demonstrating its potential for real-world GUT task automation.
increases the success rate from 6.0% to 16.4% for K 3 at Pass
4.4 ABLATION STUDY
Data filtering. We evaluate IntemVL3-88 MimoVL-7B-SFT,, and MimoVL-7B-RL trained with and without data filtering on both GUI task automation and GUI grounding benchmarks. For each K-step GUI Transition sample, each model generates 8 responses to build its candidate pool. retaining only those samples where predictions are both correct and incorrect. We select 2K training samples per model and & from each filtered pool and the original training set, respectively.
8

---

Published as a conference paper at ICLR 2026
Base Model
Fittering
Data Fitering
Android Control-Low
AndroidConttoo--High
GUI Odyssey
Scree Bcreen5pat-v2
ScreenSpat-Pro
InterviL 3-88
(N)
(9)
+12
68
Accuney
12.5
4.6
82-
+0
82
44
160
01
MMM--V--7B--F
Accurrry
+2.9
00.5
+1.7
+5.3
53
+22
+2.3
-2.2
60
-15
+06
63
01
MMM--VL- SEMN--V--BB-RR 7B-RLL
Accurnay (N)
-2.6
12.
02
+0.1
+2.0
1044
90
+0.0
4.1
10.
+2.0
405
Figure 3: Impact of Data filtering. Each model is fine-tuned on 2K K K-step GUI Transition samples.
Filtered data are more informative and challenging, and outperform unfiltered ones.
Data filtering improves accuracy on both GUI task automation and GUI grounding benchmarks. Figure 3 shows that models trained with filtered data achieve higher accuracy than those trained on unfiltered data in most cases. For example, on AndroidControl-Lon
achieves up to 4.8% higher accuracy (Figure fi.. K = 3), and on ScreenSpot-v2, up to 2.3% (Figure i)). K = 1). These results suggest that our filtering mechanism effectively selects more informative and challenging samples for GUI agent training. Moreover, since K K-step GUI Transition does not require numan-annotater instructions, this filtering process scales casily and incurs minimal cost.
Mimo-V Mimo-VL-7B.
Task formulation. We compare K-step GUI Transition with two annotated baselines. To ensure fairness, we do not apply data filtering, and all models in each comparison are trained on the same set of 2K samples, with identical current states S, and ground -ruth actions. The only difference is the instruction type: baselines pair S, with human-anma an-otateo task instruction or step instruction, while K-step GUI Transition uses the target state S the visual instruction.
Uxing St+k as the visual target outperforms using textual instructions as input. Table shows that VLMS trained with K-step GUI Transition achieves better performance than those with annotated tasks in most cases. For example, on AC-Low and AC-High, InternVLL-44 trained with GUI Transition achieves 4.0% and 3.6% higher EM accuracy, respectively, than when trained with task instructions. Qwen2.5-Ve V-LL7BBB also achieves the highest EM accuracy with GUI Transition across all benchmarks. These results indicate that Si+* provides more informative signal than human- annotated instructions.
Reasoning configurations. To verify the effect of reasoning during training, we compare models fine-tuned on 2K K-step GUI Transition data with and without <think>.. </think>.
Excluding reasoning boosts performance and efficiency for GUI-Shift. Table shows that omitting explicit reasoning requirements during training not only maintains but often improves performance. For Intern VL33-B,, training without reasoning achieves up to 7.9% higher EM on AndroidControl- High: Qwen2.5-VL-7 reasoning nearly halves training time cost, reducing from 17 to hours for Qwen2.5-VL. and from 15 to hours for Intern VL3-8B. These results confirm that excluding reasoning both improves performance and significantly enhances training efficiency.
shows consistent gains of about 2% across benchmarks. Moreover, removing
Training algorithms. For each K 21,2,3, 3, 4), fine-tune Twee2.5-VLL-7B and InterVVL3-8B on 2K identical K K-step GUI Transition data using SFT or GRPO, and evaluate them on AndroidControl.
GRPO is more suitable than SFT for K-step GUI Transition. Figure 4 shows that GRPO improves accuracy in most cases, whereas SFT consistently reduces accuracy compared to both the base models and GRPO, with drops up 65.1 1% relative to GRPO (Figure c), K 3). We attribute this to its

---

Published as a conference paper at ICLR 2026
Table 4: Performance GUI task automation under different training settings. GUI-Shift outper- forms models trained with textual instructions or with explicit reasoning requirements, Bold: the best result. TM: type match; GR: grounding accuracy for clicks; EM: exact match.
AndroidControl-Low
AndroidConttoo-Higgh
GUI Odyssey
Model
TM
GR
EM
TM
GR
EM
TM
GR
EM
Base model: Qwen2.5-VL-7B
Owen2.5-VL-7B
94.9
90.9
83.8
72.9
66.6
59.2
59.8
47.5
44.9
Task Instruction
97.913.0
93.5126
90.5-6.
85.31124
76.219.6
69.9
74.1
62.0F145
51,8169
Step Instruction
97.7t28
93.7t28
86.4-26
82,419.5
73.116.
67.2r8D
74.5t14.7
62.7115.
51.5166
Ours (w/ reasoning)
95.5
90.1
88.2-4.
83.8
75.6
69.0r9.8
74.0
63.5r16.0
51.616.7
Ours
98.0ç3.1
94.0/3.1
90.6-6.8
85.9
17.5
70.4
18.5
67.2
54.8
Base model:
InternVL3-88
VL3-8B
97.8
92.4
90.0
71.5
54.6
49.8
48.8
20.2
20.3
Task Instruction
95.9t1.9
92.810.4
85.3.47
79.718.2
65.8
54.614.8
57.318.5
31.5
26.516.2
Step Instruction
96.811.0
92.810.4
86.0.40
80.7
66.0F1144
54.5
62.61138
34.4
28.8
Ours (w/ reasoning)
97.310.
92.310.1
87.8.22
72.470,9
55.8r12
50.310.5
38.9199
18.8114
16.2
Ours
97.5
92.610.
89.3.07
78.617.
61.7m7
58.2
51.4126
21.4112
21.4t11
Owen2
IntemVL3-88
QwenVL
(b) on AC-High
AC-High
96
+32.2
+31.7
89
+52.0
+65.1 192.
+38.7
+33.2
+33.8
+32.4
55
+28.4
69
45
66
49
35
56
25
46
3
3
K
K
K
K
Figure 4: Comparison of training algorithms for the KCstep GUI Transition task (A 21, 3,4).
Qwen2.5-VLL-7B and Intern VL.3-8B are fine-tuned with 2K samples for each i and evaluated on AndroidControl. GRPO provides notable performance gains over SFT for all models and settings.
sensitivity to format mismatch between training and evaluation. These results confirm that GRPO is more effective choice for transferring K K-step GUI Transition knowledge to GUI task automation.
CONCLUSION
This study introduces GUI-Shift, self-supervised reinforcement learning framework for training VLM-based GUI agents without relying on costly annotations. Based on the K K-step GUI Transition training task, enhancing VLMS with GUI-Shift captures temporal dynamics between GUI states and provides a scalable, annotation-free training signal. Experiments across four VLMs and five benchmarks show consistent improvements, including up to 1 11.2% gains in GUI task automation accuracy and strong generalization to GUI grounding tasks. These results demonstrate that self- supervised RL can effectively exploit unlabeled GUI trajectories, offering practical and efficient alternative to training tasks with human- or AI- A--annttteed textual instructions.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (No. 62522202) and the Beijing Natural Science Foundation (No. L253005).
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint htt::2502.1392..3 2025.
David Brandfonbrener, Ofir Nachum, and Joan Bruna. Inverse dynamics pretraining learns good representations for multitask imitation. In NeurIPS, 2023.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185-24198, 2024.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing GUI grounding for advanced visual GUI agents. In ACL (1), pp. 9313-9332. Association for Computational Linguistics, 2024.
Gaole Dai, Shiqi Jiang, Ting Cao, Yuanchun Li, Yuqing Yang, Rui Tan, Mo Li, and Lili Qiu. Advancing mobile gui agents: A verifier-driven approach to practical deployment. arXiv preprint arXiv:2503.15937, 2025.
Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In UIST, pp. 845 854. ACM, 2017.
Longxi Gao, Li Zhang, Shihe Wang, Shangguang Wang, Yuanchun Li, and Mengwei Xu. Mobile- views: A large-scale mobile gui dataset. arXiv preprint arXiv:2409. 14337, 2024.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:24 2024.
Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, et al. Ui-venus technical report: Building pirf-peeformance ui agents with rft. arXiv preprint arXiv:2508. 10833, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: visual language model for GUI agents. In CVPR, pp. 14281-14290 IEEE, 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri. Shane Lyu, et al. T)"" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15122 2024.
Kaixin Li, Ziyang Meng. Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma. Zhiyong Huang, and Tat-Seng Seng Chua. Screenspot- pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981, 2025a.
Kaixin Li, Meng Ziyang, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: GUI grounding for professional high-resolution computer use. In Workshop on Reasoning and Planning for Large Language Models, 2025b. URL https
7/openrevie eww net. www./eeumm/mttteemmmo wtw.rrrmmmmxxxxxX..AA.E..E.
11

---

Published as a conference paper at ICLR 2026
Wei Li, William E. Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tya- magundlu, and Oriana Riva. On the effects of data scale on UI control agents. In NeurIPS, 2024.
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang. Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for gui visual agent. arXiv preprini arXiv:241 17466,, 2024.
Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-rl: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504. 14239. 2025.
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406. 08451, 2024.
Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guan- jing Xiong, and Hongsheng Li. Ui-rl: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620, 2025.
OpenAl. Gpt-4o system card. arXiv preprint arXiv:2410.1127.1 2024.
Yujia Qin, Yining Ye, Junjie Fang. Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprini arXiv:2501.12322 2025.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36:59708-59727 2023.
Christopher Rawles, Sarah Clinckemailic,, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405. 14573, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402..33300 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, al. Vlm-rl: A stable and generalizable rl rl-style large vision-language model. arXiv preprint 1rXiiv2504.006111 2025.
Jiaqi Tang, Yu Xia, Yi-Feng Wu, Yuwei Hu, Yuhui Chen, Qing-Guo Chen, Xiaogang Xu, Xiangyu Wu, Hao Lu, Yanqing Ma, et al. Lpo: Towards accurate gui agent interaction via location preference optimization. arXiv preprint arXiv:2506.0933 09373, 2025.
Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Pre- dictive inverse dynamics models are scalable learners for robotic manipulation. arXiv preprint
arXiv:2412.15109 2024.
Weihan Wang. Qingsong Lv,, Wenmeng Yu, Wenyi Hong. Ji Qi. Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Keqin Chen, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. In NeurIPS, 2024.
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Lim-powered task automation in android. In MobiCom, pp. 543-557. ACM, 2024.
Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, and Shuo Shang. Mobilevim: A vision-language model for better intra- and inter-ui understanding. In EMNLP (Findings), pp. 10231-10251. Association for Computational Linguistics, 2024a.
12

---

Published as a conference paper at ICLR 2026
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang. Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024b.
Xiaobo Xia and Run Luo. Gui-rl: A generalist rl-style vision-language action model for gui agents. arXiv preprint arXiv:2504. 10458, 2025.
Xiaomi. Guievalkit: Open-source evaluation toolkit for gui agents. https:///ithubbccm
#iLommm-eeserrchhhgllevaaaa 2025a. Accessed: 2025-09-25.
LLM-Core-Team Xiaomi. Mimo-vl technical report. arXiv preprint arXiv:2506.03569, 2025b.
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.0445% 2024.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mart prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 11441, 2023.
Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256 2024.
Samuel Zapolsky and Evan M. Drumwright. Inverse dynamics with rigid contact and friction. Auton. Robots, 41(4)831--63, 2017.
Li Zhang, Longxi Gao, and Mengwei Xu. Does soaaain-of-thouggh reasoning help mobile gui agent? an empirical study, arXiv preprint 26788 2025a.
Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, Jie Xie, Wei Zhou, Wang Xu, Yuanheng Zhang, Zhou Su, Zhongwu Zhai, Xiaoming Liu, Yudong Mei, Jianming Xu, Hongyan Tian, Chongyi Wang. Chi Chen, Yuan Yao, Zhiyuan Liu, and Maosong Sun. AgentCPM-GUI: Building mobile-use agents with reinforcement fine-tuning, arXiv preprint arXiv: arXiv:2506.0139// 2025b.
13

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS (LLMS)
We use LLMs for writing polishing and language refinement.
B TRAINING HYPER-PARAMETERS
The details for GUI-Shift are provided in Table 5
Table 5:
settings used for all GRPO training.
Hyper-parameter
Value
learning_rad rate
from 1e-6 to
temperature
0.9
num sum_generations
sum_train_epocht
max_promp__engthh
1024
_ox__ompletion_lengee
256
per lr__dvvvce___ate batch size
gradient accumulation _ttpps steps
(clipping parameter)
0.2
(KL coefficient)
0.04
14

---

Published as a conference paper at ICLR 2026
C TASK AUTOMATION RESULTS WITH AND WITHOUT DATA FILTERING
Table and Table 7 report the performance of GUI-Shift on AndroidControl and GUI Odyssey benchmarks using 2K Ksttep GUI Transition samples, with and without data filtering. Without data filtering, as shown in Table GUI-Shift trained with each & value outperforms the base models on most benchmarks. Specifically, for Qwen2.5-VL-77, GUI-Shift delivers consistent improvements across all benchmarks; for the other three models, GUI-Shift also improves task automation in most cases, especially on AndroidControl. These results demonstrate that conditioning on the future state S++E consistently provides effective supervision signal across different transition step sizes. We apply data filtering
Intern VI.3-8B, MimoVL-7B--FTT, and Vi With data filtering, as shown in Table 7 model performance further improves in most settings, confirming that filtering enhances data quality and strengthens VLM optimization.
Table Performance on task automation benchmarks: AndroidControl and GUI Odyssey. Models are fine-tuned with 2K k-step UI Transition samples for each & E {1, 3, 4]. without model-specific data filtering. The target states Si+k with different & values provide effective visual instructions for GUI agent training. TM: type match; GR: grounding accuracy for clicks; EM: exact match.
AndroidControl-LLww
AndroidConntrol-Highh
GUI Odyssey
Model
TM
GR
EM
TM
GR
EM
TM
GR
EM
Qwen2.5 -
94.9
90.9
83.8
72.9
66.6
59.2
59.8
47.5
44,9
GUI-Shift-Qwer (A: 1)
98.0ra.1
94.0ra.1
90.616.8
85,9-13a
77.5-109
70.4-112
78.5
67.2
54.8199
GUI-Shift-Qwer (A: 2)
98.0r3.1
93.7728
90.016.2
85.9-13a
76.9-103
70.8
79.4
68.6121.1
55.71108
GUI-Shift-QU -wen (f
97.913.0
93.612.7
91.5m 85.3-12A
77.7
70.0r10s
77.1
67.6120/
53.818.9
GUI-Shift-Qwen 4)
97.8129
92.8119
91.0172
84.8-119
77.1-105
69.6rm4
76.0r162
65.7
53.1
InternVL3-8B
97.8
92.4
90.0
71.5
$4.6
49.8
48.8
20.2
20.3
GUI-Shift-mtemm (A: 1)
98.110.3
92.810.4
89.6.0.4
73.1
56.0r14
52.Sr27
42.9
17.6126
17.9424
GUI-Shift-Inter (I 2)
97.610.2
92.910.5
87.912.1
76.3
59.8152
54.3145
59.41106
24.4M12
23.1
GUI-Shift-Intem
97.5
92.610.
89.3
78.6ma
61.7m
58.2184
51.4726
21.441.2
21.441.
GUI-Shift-Ineem 4)
90.3
92.910.5
88.012.
78.5
64.3r97
56.6168
59.6110s
25.915.7
23.313.0
Mimo-VL-7B-SFT
90.8 93.5
85.7
75.2
75.7
63.1
86.9
66.3
62.0
GUI-Shift- SiaifttMMim-SFF- (I: 1) 98.6178 93.8103
92.016.3
86.8
74.3
71.7186
85.411
67.1
61.3
GUI-ShiflMMmooSTI (k: 2)
98.5 92.7
90.0
87.0-118
73.9
70.3172
85.0119
66.0j03
59.5125
GUI-Shifl-MiooSTI (k: 3) 98.2
92.5
88.4127
85.5-10.3
72.2
14.0
86.0,09
65.4j09
58.51a5
GUI-Shift-Mimo SFT 4) 98.317.5
93.010.5
92.216.5
11.9
73.4123
72.4193
85.9110
67.7tL.
60.1
Mimo-VL--BB- VL--7B-RI
91.8 94.5
87.2
76.5
77.5
64.6
87.2
67.9
63.1
SUUlShift-Mimon RI. (k 98.7166 94.7
92.515.
86.7-102
77.0gas
70.8162
85.0jaa
67.8ja.
59.1448
GUI-Shift-Min Minio-LM RL (k- 98.716.
93.3 88.211.0
8
72.7U4.8
69.0r4.
85.3119
66.2
59.41a.
GUI-Shift-Mimo-P (k- 98.016.2
93.8
1
85.8193
73.6139
68.8142
85.9113
66.0,19
58.8143
GUI-Shifl-Mimo
98.516.7
93.0
89.5
86.6~10.1
73.0.4.
70.1
85.3
66.
57.515.
15

---

Published as a conference paper at ICLR 2026
Table 7: Performance task automation benchmarks: AndroidControl and GUT Odyssey. Compared to training without data filtering, applying filtering yields greater improvements in most cases. TM: type match; GR: grounding accuracy for clicks; EM: exact match.
AndroidConntrol-Lov
AndroidConntool-Hgg
GUI Odyssey
Model
TM
GR
EM
TM
GR
EM
TM
GR
EM
Intern VL3-8B
97.8
92.4
90.0
71.5
54.6
49.8
48.8
20.2
20.3
GUI-Shift-Inter (k 1)
96.2.1.6
92.6102
87.0.3.0
78.617.1
63.218.6
55.1
51.6128
24.5
22.1
GUI-Shift-Inter (k 2)
97.2,06
92.3,o
86.4
79.4,79
63.3
55.9t61
51.9ta.
24.414.
23.9
GUI-Shift-Inters (k 3)
96,3,15
92.7f03
87.7u2.3
79.918.4
65.5-109
57.6118
59.5+10.7
27.2
24.8
GUI-Shift-Intem (k = 4)
96.221.6
92.8f04
87.7u23
81.019.5
67.8-132
59.91101
57.0wx2
27.417.
25.715.
Mimo-VL VL-7B-SIT
90.8 93.5
85.7
75.2
75.7
63.1
86.9
66.3
62.0
GUI-ShiftMMimo-3 (k 1) 97.8r1. 93.8,03
92.2n6.5
84.619.4
75.7
70.2
81.8
64.3|20
56.2,5.8
Mii-Shift-MMio--- SFT (k 2) 98.6178 93.
92.977.
86.41112
75.8
72.0ps9
85.2.17
66.6103
60.3
GUI-ShiftMMmo-SFFT (A 3) 98.6718 94.0tos
93.2
87.2
75.6.0.1
73.4m10
86.1,08
66.3100
60.7
GUI-Shift-Minn SFT (A 4) 98.6 9305
92.7
85.81106
73.911.
71.618.
85.911.
67.5112
60.9
Mimo-VIL-7B-BB
91.8
94.5
87.2
76.5
77.5
64.6
87.2
67.9
63.1
-UT--Shit-Mimo-T RI (k 1)
98.9-7.1
94.3jo2
93.216.
86.91104
75.9
71.7f71
84.8,244
67.5
59.5
GUT-Shiit-Mimm-RRL (k 2)
97.7-5.9
93.710s
91.3+4.1
87.6
75.9
71.6770
84.9
65.4122
58.9
Sii-Shhhi--M--RRL 3) 97.3+5.
93.9106
91.1
8701
77.3,022
71.717.1
84.9123
67.6103
59.7
CUL-Shift-Mimo-
4) 96.8r5.0
94.2
90.9+37
87.0r10s
77.0,0.5
72.1475
84.7u25
68.3
59.9
16

---

Published as a conference paper at ICLR 2026
D GROUNDING RESULTS WITH AND WITHOUT DATA FILTERING
We report the accuracy of GUI-Shift on two GUT grounding benchmarks: ScreenSpot-v2 and ScreenSpot-Pro, under both filtered and unfiltered settings.
For ScreenSpot-v2 (Table 8), all models except GUI-Shift-Inten achieve consistent improvements their respective baselines without data filtering. With data filtering (Table 2p. GUI-Shift-Intern models also surpass their baselines, indicating the benefit of filtering.
For ScreenSpot-Pre (Table I0), all GUI-Shift- wwen models improve without data filtering, while other models exhibit mixed results across different k. We attribute the performance drop to the high-resolution desktop screenshots ScreenSpot-Preg which are not present in our training data. With data filtering (Table III. GUI-Shift- Miahiftt-imo-RRe Mimo--RL achieves consistent gains for all k.
Overall, GUI-Shift demonstrates strong generalization on GUI grounding, with performance gains in most cases, especially when data filtering is applied.
Table 8: Performance on GUI grounding benchmark: ScreenSpot-v2. Models are fine-tuned with 2K
k-step UT Transition samples each (1,2, 4), without model-specific data filtering.
Mobile
Desktop
Web
Model
Avg.
Text
Icon
Avg.
Text
Icon
Avg.
Text
Icon
Avg.
Qwen2.5-VL-P2
98.3
86.3
93.2
88.7
67.1
79.6
92.7
81.8
87.6
87.7
GUI-Shift-OO twen (k = 1)
98.6
87.7
94.0
88.1
71.4
81.1
92.7
81.8
87.6
88.4
GUI-Shift SUUlShift-Qwen -wen (k 2)
98.6
89.1
94.6
87.6
73.6
81.7
92.3
80.8
87.0
88.6ra.
GUI-Shift-Ower
3)
88.6
94.6
86.1
72.9
80.5
80.8
87.0
88.3
CUU-Shift-Qwen (k- -4
98.6
89.6
94.8
86.1
75.0
81.4
92.7
82.8
88.1
89.0+1.3
Intern VL3-8B
81.5
88.4
80.4
68.6
73.4
82.8
81.3
GUI-Shift- (k = 1)
94.1
81.5
88.8
77.3
52.1
66.8
91.5
73.4
83.1
81.1
GUI-Shift-Inten (A: 2)
93.8
80.6
88.2
80.4
52.1
68.6
91.5
72.4
82.6
81.1.0.2
GUI-Shift-Intem (k 3)
94.8
80.1
88.6
78.9
47.1
65.6
90.6
71.4
81.7
80.2
GUI-Shift-Intor 4)
94.1
82.0
89.0
79.4
54.3
68.9
91.9
71.9
82.6
81.5
Mimo-VL-7B VL-7B----BB
96.6
84.4
91.4
92.8
80.0
87.4
88.9
76.8
83.3
87.6
GUI-Shift Mii-ShhffMMM--SSSS
1)
97.2
86.3
92.6
91.8
79.3
86.5
74.9
83.3
87.8y0.2
GUI-Shift- Shift-Miim---ee
SFT (k: = 2)
95.9
84.4
91.0
92.8
82.9
88.6
91.9
75.9
84.4
88.1 10.5
GUT-Shifi-Mimo-ST (k = 96.6
86.3
92.2
91.8
82.9
88.0
90.6
77.3
84.4
88.4ras
GUT-Shiff-Mimo-Ti (k= = 44)
96.9
87.2
92.8
91.8
84.3
88.6
89.7
80.3
85.4
89.2
Mimo-VI.-7B-RI.
98.3
86.3
93.2
90.2
80.7
86.2
92.7
75.44
84.7
88.4
GUT-Shift-Mimo-R
RI. (h 1)
99.3
88.2
94.6
91.8
80.7
87.1
90.2
75.4
83.3
88.8
GUI-ShiftMMmo-RRI (k:
97.9
87.2
93.4
91.8
81.4
87.4
91
74.4
83.3
88.4ro.
GUI-Shift-Mimo, RL
3)
98.3
87.2
93.6
91.8
83.6
88.3
91
76.4
84.2
89.0ru.
CUU-SShit-Mimor RL L (k
99.3
87.7
94.4
91.8
81.4
87.4
90.6
75.4
83.5
88.810.
17

---

Published as a conference paper at ICLR 2026
Table 9: Performance on GUI grounding benchmark: ScreenSpot-v2. Model-specific data filtering applied InterVVL3-88 MimoVL--7B-SFI and MimoVL--7B-R For each model, we select 2K k-step UI Transition samples for each & E 3,44,344 from a pool of candidates.
Mobile
Desktop
Web
Model
Avg.
Icon
Avg.
Text
Icon
Avg.
Icon
Avg.
LntemVL3-88
93.4
81.5
88.4
80.4
52.1
68.6
91.0
73.4
82.8
81.3
GUI-Shift-Intem 1)
83.4
89.4
80.4
51.4
91.0
73.4
82.8
81.6,0.3
GUI-Shift-Intem (k 2)
94.5
82
89.2
78.9
51.4
67.4
91.9
72.4
82.8
81.3
LUI-Shiit-Inteae (A: = 3)
94.5
83,4
89.8
79.4
50.7
67.4 4
91.9
73.4
83.3
81.710.4
CUI-Shift-Inten (k
93.4
83.4
89.2
77.8
54.3
68.0
91.5
73.9
83.3
81.6r0.3
Mimo-VL-7B VL-7B-L---B
96.6
84.4
91.4
92.8
80.0
87.4
88.9
76.8
83.3
87.6
GUI-Shift SLUl-hiit-Mimo
-iim--Mi--------- (k: 1)
98.3
87.7
93.8
92.3
82.1
88.0
94.0
79.8
87.4
90.1
2)
87.7
94.0
93.3
83.6
79.8
86.7
90.3
SiifttfttM-io--- (k = 3)
96.6
85.3
91.8
91.8
81.4
87.4
89.3
80.3
85.1
88.4
GUI-Shift-Mo Shaif-Mimo-SSe SFT (k = 4)
97.6
83.9
91.8
92.3
82.9
88.3
91.5
79.3
85.8
88.8
Mimo-VL-7B- VL-7B-RL
98.3
86.3
93.2
90.2
80.7
86.2
92.7
75.4
84.7
88.4
GUI-Shift-Minn Sinift-Mimo-RR RL ( (k =)
99.0
87.7
94.2
91.2
83.6
88.0
89.7
72.9
81.9
88.4r0.
GUL-Shift-Mimo- RL (k
97.9
86.7
93.2
90.7
80.7
86.5
91.0
75.4
83.8
88.2
RL
99.0
87.2
94.0
91.8
80.7
87.1
91.0
76.8
84.4
88.9r0.5
GUI-Shift- Sohift-Mmm-RRe RL (k-
98.6
85.3
93
91.8
81.4
87.4
91.9
77.3
85.1
88 88.810.
Table 10: Performance on GUI grounding benchmark: ScreenSpot-Pre Models are fine-tuned with 2K step UI Transition samples for each E 2,,2,3, 4}, without model-specific data filtering.
CAD
Dev
Creative
Scientific
Office
OS
Model
Avg.
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
1ree2.5-VL-77
16.2
1.6
44.2
2.1
36.9
7.7
47.9
8.2
$3.7
18.9
36.4
7.9
26.4
GUI-Shift- Qwen (k
16.2
4.7
52.6
9.0
27.3
7.0
52.1
5.5
49.7
17.0
38.3
13.5
26.8t0.4
GUI-Shift-Qwan Owen (k
16.8
3.1
52.6
10.3
29.3
8.4
52.1
4.5
50.8
17.0
35.5
13.5
27.2108
Sii-Shiit-Qeer (A: 3)
15.7
52.6
30.3
50.7
5.5
48.0
15.1
35.5
26.5
GUI-Shift-Qwen (A:
17.3
3.1
51.9
9.7
30.3
7.0
54.2
5.5
49.7
17.0
33.6
12.4
27.1
LLternVL3-8B
8.6
4.7
27.3
27.3
0.7
24.3
4.5
32.2
3.4
15.0
CUI-Shift-Inters (A: 1)
10.2
1.6
27.3
4.8
26.8
0.7
23.6
3.6
33.9
7.5
15.0
2.2
15.410.4
GUI-Shift-Inten (A: 2)
7.6
3.1
27.3
3.4
28.3
0.7
21.5
4.5
33.3
7.5
12.1
2.2
14.9.0.1
CU1-Shift-Inters (A
9.6
3.1
23.4
5.5
26.3
0.7
20.1
1.8
29.9
3.8
11.2
1.1
13.7
GUI-Shift-Intee (f 4)
9.1
4.7
24.0
4.1
28.8
0.7
21.5
3.6
31.1
3.8
13.1
1.1
14.5
Hino-VLL-7B-SFT
47.2
23.4
48.7
9.0
48.0
13.3
70.8
27.3
64.4
39.6
36.4
15.7
39.8
GUI-Shift-Mimo-SFI SFF (k 1)
49.7
15.6
46.8
13.8
49.0
16.1
74.3
25.5
65.0
37.7
40.2
15.7
40.9-1.1
DUt-Shift-Mimo- (k 48.7
20.3
50.6
11.7
48.5
14.0
72.9
26.4
63.3
41.5
36.4
16.9
40.6r0.8
SuU--ShittMMmmo- (k: 45.7
18.8
12.4
45.5
70.1
28.2
65.5
39.6
35.5
19.1
39.6102
GUI-Shift-Mine SFT (k 4)
45.7
18.8
51.3
12.4
46.5
12.6
71.5
28.2
62.7
41.5
37.4
16.9
39.9
Mimo- VI.---VI-
48.2
14.1
46.8
11.0
47.0
14.0
71.5
27.3
66.7
39.6
39.3
19.1
40.2
GUI- Shift-Mimo RL (k 1)
48.7
15.6
13.8
13.3
72.2
30.0
66.7
42.1
21.3
41.7115
Se1-Shift-Miins RL (k 2)
46.7
14.1
47.4
13.8
49.0
12.6
70.1
27.3
65.5
39.6
41.1
21.3
40,5
GUT-Shift-Mimo-R RL (k =3)
49.7
12.5
47.4
11.0
47.0
11.2
70.8
26.4
65.5
45.3
35.5
20.2
39.9.03
LLT-Shift-Miman RL. (I 46.7
14.1
46.1
13.8
47.0
14.7
69.4
27.3
64.4
41.5
36.4
21.3
39.8.04
18

---

Published as a conference paper at ICLR 2026
Table 11:
Performance on GUI grounding benchmark: ScreenSpot-Pro Model-specific data filtering is applied to InternVL3-8B. MimoVL-7B-S3 and MimoVL-7B-RI For each model, we 2,3, 4) from a pool of candidates.
select 2K k-step UI Transition samples for each 21,
CAD
Dev
Creative
Scientific
Office
OS
Model
Avg.
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
InternVL3-81
8.6
4.7
27.3
27.3
0.7
24.3
4.5
32.2
11.2
3.4
15.0
GUI-Shift- Shift-lntem (A: 1)
11.2
3.1
26.6
5.5
28.8
0.7
20.1
4.5
33.9
7.5
13.1
1.1
15.4
GUI-Shift-Inten (A: 2)
8.1
1.6
26.0
4.1
26.3
0.7
25.7
1.8
31.1
5.7
14.0
1.1
14.5
GUT-Shift-Intem (k 3)
8.6
1.6
27.3
4.1
25.3
0.7
21.5
1.8
33.3
5.7
14.0
1.1
14,4.0.6
SU1-Shift-Intem (f:
8.1
1.6
27.9
4.8
26.8
0.0
22.9
2.7
32.8
3.8
10.3
2.2
14.5
Mimo-VL-7B--S
47.2
23.4
48.7
9.0
48.0
13.3
70.8
27.3
64.4
39.6
36.4
15.7
39.8
GUT-Shift-Mimo-SN SFT (A 1)
49.2
14.1
50.0
9.0
48.0
11.9
73.6
29.1
54.3
43.4
38.3
15.7
40.710.9
BUL-Shift-Mimo-TT (k
46.2
17.2
46.8
11.0
47.5
10.5
74.3
30.9
65.5
43.4
35.5
16.9
40.0
GUI-Shift-Mimo- (A = -3)
50.8
15.6
47.4
13.8
47.0
69.4
24.5
65.0
19.1
40.2+0.4
GUT-Shift-Minn SFT (A: 4)
44.2
18.8
53.9
11.0
43.9
11.2
72.2
26.4
62.1
47.2
40.2
12.4
39.410.
Mimo- Mimo-VI.-7Baae VI.-7B-RI
48.2
14.1
46.8
11.0
47.0
14.0
71.5
27.3
66.7
39.6
39.3
19.1
40.2
GUT-Shift-Mime RI. (k 1)
48.7
14.1
51.9
13.1
50.0
15.4
70.8
28.2
67.2
39.6
37.4
21.3
41.6r14
SUL--hift-Miins RL (k: 2)
47.2
18.8
47.4
14.5
49.0
15.4
72.2
29.1
66.1
43.4
39.3
19.1
41.3
GUI-Shift-Mo Mi-Shhit--Mio---- RL (k 3)
45.7
17.2
46.1
11.7
49.5
14.0
70.8
29.1
66.1
39.6
37.4
21.3
40.4
DUI-Shift-Mimar RI. (k: 4)
47.7
18.8
48.1
13.8
52.5
12.6
72.2
28.2
67.2
45.3
40.2
20.2
41.811.
E REWARD DISTRIBUTION OF MODEL-SPECIFEE DATA FILTERING
To examine the relationship between the K value, task difficulty, and model capability, analyze the reward distributions obtained during the data filtering stage. Our analysis based on 7,783 shared samples. For fair comparison, the current state Stt) and ground-truth actions remain identical across different K values. Figure presents the reward distributions for each model across different K values.
exactly the same samples for each K across all models, and
InternVL3-8B
MImo-VL-7BB-FF
MImo-VL-7B-RB
#Samples (x1000)
2
5
3
5
7
Couley Predictions (total reward=2)
Figure 5: Comparison of reward distributons during model-specific data filtering, The x-axis represents the number of fully-correct predictions, where x indicates all 8 rollouts are incorrect and x 8 indicates all 8 rollouts are correct. The y-axis reports how many K-step GUI Transition samples fall into each fully-correct count.
We provide detailed analysis from two perspectives:
From the perspective of K. (1) At x = O, where all 8 rollouts are incorrect, the sample counts follow K = 4 3 2 1, indicating that larger K values correspond higher difficulty. (2) At x = 8, where all predictions are correct, the trend reverses to K > > 3 > 4, showing that smaller K values easier for the models.
From the perspective of model capability, As shown in Table T and Table 2 Mimo-VL-7B-S3 VL-7B-SFT and Mimo VL-BB-RI. perform better than Intern VL.3-BB on four GUI-related benchmarks. We may
19

---

Published as a conference paper at ICLR 2026
reasonably consider the Mimo models to have stronger GUI capability. A similar pattern appears in Figure 5 both Mimo models have noticeably more samples at x = 8 than Intern VL3-8B, suggesting that models with stronger GUI capability perform better on the K-step GUI Transition task.
Overall, these findings reflects that the K-step GUI Transition task becomes more challenging as K increases, and that models with stronger GUI capability achieve better performance this task, which aligns with intuition.
F INTERNVL3-8B PERFORMANCE SCALING WITH TRAINING DATA SIZE
To investigate how training data size affects GUI-Shift performance, we scale the training set from 2K 6K samples using K-step GUI Transition data (K 1). Figure 6 illustrates the results AndroidConttol-Loww and AndroidConntol-HHighh
AndroidControl-Low
AndroidCCnntoo-Htggh
+1.7
88.6
57.0
Accuracy (%)
+1.6
88.1
56.5
+0.7
87.6
56.0
+0.6
87.1
+0.2
55.5
+0,4
86.6
-0.3
55.0
2
3
4
5
#Training Samples (x1000)
Figure 6: Performance on AndroidControl with Intern VL3-8B trained on 1-step GUI Transition data. Training samples are scaled from 2K 6K. Accuracy exhibits an overall upward trend with only minor fluctuations.
Specifically, relative to the 2K setting, GUI-Shift-Ine Shift-Intern shows an additional gain of 1.7% when trained with 6K samples AndroidControl-Lovw and a further 2.0% improvement when trained with 3K samples on AndroidControl High. In general, we observe an overall upward trend accuracy as the data size increases, with only small fluctuations across different scales.
20


Reviews:
[{"review_id": "XXls3R1qMa", "reviewer": "Reviewer_VvL8", "summary": "This paper proposes a self-supervised K-step GUI Transition task, which enables VLMs to learn GUI dynamics by predicting the initial action that causes a transition between two GUI states.\n\nThe approach eliminates the need for natural language instructions and enables scalable dataset construction from existing GUI trajectories. Building on this task, the author propose GUI-Shift with reinforcement learning (RL) framework. Experiments show that training on GUI-Shift generalizes well to both GUI automation and grounding tasks, yielding up to an 11.2% increase in GUI automation accuracy.", "questions": "1. How to assure the quality of training data? From fig. 1, the task has 2 target, task prediction and action prediction, action prediction can achieve by compared to ground-truth, but how to assure task prediction is aligned with the actual target.  \n2. Is there an explanation why the shift prediction task could improve the performance of grounding task, which seems to be irrelevant even when k=4.\n3. Could GUI-shift generalize to different scenarios and tasks, like out-of-domain tasks. \n4. Is there some connections between step K and task difficult or model’s capability. \n5. Could model tackle task if the trajectory includes significant state shift across screenshots like mobile application tasks (ac is relative simple) \n6. Could K-step GUI Transition task possesses scaling law compared to datasets constructed by general trajectory, further experiment is encourage if calculation resource is enough.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposed a novel GUI task, k-step GUI transition task, which enables self-supervised training and eliminate dataset construction.", "weaknesses": "Despite the efficiency of the method, mechanisms and deeper analysis are somewhat lacking - please see details in questions part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aeArcyldBn", "reviewer": "Reviewer_QngP", "summary": "GUI-Shift is a self-supervised RL framework that applies GRPO to the “K-step GUI Transition” task: given a current screen St and a future screen S_(t+k) taken from real trajectories, the model predicts the first action that initiates the transition. It optimizes with rule-verified rewards (format + action correctness) over a unified 8-type action space, and uses a sampling-and-scoring loop for both training and data filtering. Trained on 2K samples per backbone (k∈{1,2,3,4}) and evaluated on AndroidControl/GUI Odyssey (automation) plus ScreenSpot-v2/Pro (grounding), GUI-Shift reports notable gains—up to +11.2% accuracy on AndroidControl-High and +2.5% on ScreenSpot-v2—without extra alignment.", "questions": "see above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "GUI-Shift is practical and scalable because it builds supervision directly from real GUI trajectories (no manual labels) and trains with a GRPO sampling-and-ranking loop that better accommodates multiple plausible actions than single-label SFT, while its rule-verified rewards provide precise, automatic checks on action type and arguments. The unified eight-action interface and “keep only informative cases” filtering keep optimization focused and stable, and dropping explicit reasoning traces cuts training overhead without hurting downstream performance, together yielding consistent gains on mobile GUI automation and even modest improvements on GUI grounding.", "weaknesses": "The method treats the trajectory’s first action as the only gold label. Rule-based rewards allow coordinate variance (e.g., any point inside the bbox) but not type/argument alternatives, so equally valid first moves that still reach S_(t+k) get penalized, biasing against strategy-level equivalence.\n\nTraining/eval center on AndroidControl and GUI Odyssey (mobile). Grounding sets are included, but not multi-step control. Metrics are mostly single-step TM/EM, no end-to-end success rates, latency/token cost, or cross-OS/desktop multi-app scenarios—so external validity is limited.\n\nOverly literal rewards, weak goal alignment. Rewards favor exact action type/argument matches (e.g., verbatim input_text, specific scroll step) instead of measuring progress toward the target screen S_(t+k). This creates false negatives for synonymous inputs or alternative scrolls and teaches label imitation over goal achievement.\n\nFirst-step-only training/eval (short-horizon myopia). Given (St, S_(t+k)), the model learns only the first action, and metrics reflect single-step accuracy. This risks poor multi-step planning and error recovery during rollouts, weakening correlation with real agent performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W4Y4S2YrpE", "reviewer": "Reviewer_7wjQ", "summary": "The paper proposes GUI-Shift, a self-supervised reinforcement learning framework for training VLM-based GUI agents without manual task instructions. The core pretext task is K-step GUI Transition: given a current screenshot and a target future screenshot, the model predicts the first action that initiates the transition. Training uses GRPO with a simple, verifiable reward, and a model-specific data filtering step that retains samples where the model produces both correct and incorrect actions under stochastic decoding. The authors fine-tune four open VLMs and report gains over base models on AndroidControl, GUI Odyssey, ScreenSpot-v2, and ScreenSpot-Pro.", "questions": "(1) What fraction of candidate pairs survive filtering per model and per k? What is the inter-model overlap of filtered pairs?\n\n(2) In Table 3, were the task/step instruction baselines also trained with GRPO and the same binary-format reward?\n\n(3) If only click points are recorded, how is the box inferred, and how would GUI‑Shift operate on real logs that lack element metadata?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The K-step Transition objective is a compact way to leverage unlabeled trajectories at scale. The visual-goal formulation reduces reliance on noisy textual instructions and naturally encourages temporal reasoning.\n\n\nS2: GRPO avoids penalizing multiple valid clicks (any point in a control) and removes the need for a critic. The binary-format action reward is stable, inexpensive, and reproducible.\n\nS3: On AndroidControl-High, GUI-Shift-Qwen improves EM by +11.2% (to 70.4%), with other base models also benefiting. Despite training only on AndroidControl trajectories, models improve on ScreenSpot-v2 and ScreenSpot-Pro, demonstrating potential cross-platform generalization for GUI grounding.", "weaknesses": "W1: Training relies solely on the AndroidControl split, which undercuts the claim that the method scales to arbitrary “unlabeled trajectories.”\n\nW2: Performance dips on GUI Odyssey for some models are attributed to tablet layouts; ScreenSpot‑Pro (desktop, high‑res) gains are modest.\n\nW3: The work evaluates offline action accuracy and grounding. Demonstrating end‑to‑end success in a dynamic environment like AndroidWorld would strengthen claims about task automation robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VDRhTy2fDB", "reviewer": "Reviewer_hoLT", "summary": "The paper introduces a novel self-supervised task for VLMs used in GUI agents. It is inspired by inverse dynamics within unlabeled GUI trajectories. They also propose a self-supervised RL framework with advantages in handling action multiplicity and poor generalization in GUI tasks.", "questions": "• Formatting Suggestion: A minor formatting note: On page 4, Line 254, the label “(3)” appears misaligned.\n\n• Predicting Action Sequences: The current K-step GUI Transition task only requires the model to predict the first action initiating the transition from $S_t$ to $S_{t+k}$. However, the \"Action Prediction\" example in Figure 1 (Middle) implies the model reasons about a sequence of actions (\"...I should click the search box and type... So, I think the first step should be: click...\"). Have the authors considered or experimented with a more complex task objective, such as predicting the full sequence of $k$ actions required to reach $S_{t+k}$? If so, how did this compare to predicting only the initial action?\n\n• Generality of Data Filtering: The data filtering pipeline, which selects samples with both correct and incorrect predictions, is presented as a key component for improving performance (Figure 2). However, this filtering was not applied to the Qwen2.5-VL-7B model because it produced an \"exceptionally high\" proportion of all-correct or all-incorrect samples5. This suggests the filtering heuristic may be model-dependent or brittle. Could the authors comment on the generality of this filtering component? Does this finding imply that the heuristic's effectiveness is strongly tied to the base model's initial capabilities or calibration?\n\n• Impact of Omitting Reasoning Traces: The ablation study in Table 3 shows that omitting explicit reasoning traces during training not only nearly halves training time but also improves downstream performance. This finding is significant and somewhat counter-intuitive. Could the authors provide further explanation or hypotheses for this phenomenon?\n\n• Analysis of the $k$ Parameter: The choice of $k$ in the K-step GUI Transition task is a central hyperparameter. Based on Tables 1 and 2, the optimal $k$ varies by model and task. For instance, $k=1$ (predicting the transition to the immediate next state) often performs very well, particularly for GUI grounding (best for 3/4 models). Conversely, Qwen2.5-VL-7B and InternVL3-8B achieved their best automation scores with $k=4$. Do the authors have any insights into this behavior? Is there an observable trade-off between the signal clarity of small $k$ values and the long-term goal-conditioning provided by larger $k$ values?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The primary strength is the \"K-step GUI Transition\" task. By formulating the training problem as a self-supervised inverse dynamics task , the authors provide a solution to bypass the need for expensive natural language annotations. This addresses the scalability bottleneck in GUI agent training data.\n- The paper provides a well-crafted design of a RL-framework. Novelly set the visual goal as the supervision signal. This design requires the VLMs to capture both the diverse GUI settings and the dynamics transformation between states, therefore enhancing the situational awareness of the model.\n- The experimental quality is a major strength. The authors validate their method across multiple VLM backbones and diverse benchmarks (automation and grounding). The extensive ablation studies (covering filtering , task formulation , training algorithm , and reasoning ) are thorough and leave little doubt about the efficacy of each component.\n- The paper is exceptionally clear. Figure 1  is a model for how to visually communicate a complex framework, perfectly contrasting the proposed method with prior work", "weaknesses": "See Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)