Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
Authors: Junlei Zhang
Zichen Ding
Chang Ma
Zijie Chen
Qiushi Sun
Zhenzhong Lan
Junxian He
Venue: COLM 2025 (2025)
Keywords: GUI agent
middle training
llm as agent
TL;DR: We comprehensively study how middle training on a series of tasks, except for GUI domain, can enhance specific capabilities such as GUI perception, visual reasoning, and knowledge.

Abstract: Graphical User Interface (GUI) agents offer cross-platform solutions for automating complex digital tasks, with significant potential to transform productivity workflows. However, their performance is often constrained by the scarcity of high-quality trajectory data. To address this limitation, we propose training Vision Language Models (VLMs) on data-rich, reasoning-intensive tasks during a dedicated mid-training stage, and then examine how incorporating these tasks in the mid-training phase facilitates generalization to GUI planning scenarios. Specifically, we explore a range of tasks with readily available instruction-tuning data, including GUI perception, multimodal reasoning, and textual reasoning. Through extensive experiments across 11 mid-training tasks, we demonstrate that: (1) Task generalization proves highly effective, yielding substantial improvements across most settings. For instance, multimodal mathematical reasoning enhances performance on AndroidWorld by an absolute 6.3\%. Remarkably, text-only mathematical data significantly boosts GUI web agent performance, achieving a 5.6\% improvement on WebArena and an 5.4\% improvement on AndroidWorld, underscoring notable cross-modal generalization from text-based to visual domains; (2) Contrary to prior assumptions, GUI perception data—previously considered closely aligned with GUI agent tasks and widely utilized for training—has a comparatively limited impact on final performance; (3) Building on these insights, we identify the most effective mid-training tasks and curate optimized mixture datasets, resulting in absolute performance gains of 8.0\% on WebArena and 12.2\% on AndroidWorld.

Published as a conference paper at COLM 2025
Breaking the Data Barrier - Building GUI Agents Through Task Generalization
Junlei Zhang* 1
Zichen Ding" Chang Ma Zijie Chen 2
Qiushi Sun
Zhenzhong Lan* Junxian He*
Zhejiang University & Westlake University
Shanghai AT Alborratrry
The University of Hong Kong
*KKUST
Abstract
Graphical User Interface (GUI) agents offer cross-platform solutions for automating complex digital tasks, with sigmificant potential to transform productivity workflows. However, their performance is often constrained by the scarcity of high-quality trajectory data. To address this limitation, we propose training Vision Language Models (VLMs) data-rich, reasoning- intensive tasks during a dedicated mid-training stage, and then examine how incorporating these tasks in the mid-training phase facilitates gener- alization to GUI planning scenarios. Specifically, we explore a range of tasks with readily available instruction-tuniing data, including GUI per- ception, multimodal reasoning, and textual reasoning. Through extensive experiments across 11 mid-training tasks, we demonstrate that: (1) Task generalization proves highly effective, yielding substantial improvements across most settings. For instance, multimodal mathematical reasoning enhances performance AndroidWorld by absolute 6.3%. Remarkably, text-only mathematical cata significantly boosts GUI web agent perfor- mance, achieving 5.6% improvement on WebArena and a 5.4% improve- ment on AndroidWorld, underscoring notable cross-mode mooal generalization from text-based to visual domains; (2) Contrary to prior assumptions, GUT perception drta-previously considered closely aligned with GUI agent tasks and widely utilized for training- has a comparatively limited im- pact on final performance; (3) Building on these insights, we identify the most effective mid-training tasks and curate optimized mixture datasets, resulting in absolute performance gains of 8.0% on WebArena and 12.2% on AndroidWorld Our work provides valuable insights into cross-domain knowledge transfer for GUT agents and offers practical approach to ad- dressing data scarcity challenges this emerging field. The code, data, and hwt..ooooo/hhttpppppp hkust-nlp/ Suantnnll/GGMid.... /UIMidd
models are available at https:/ /github.com,
1 Introduction
Interacting with graphical user interfaces (GUIs) has become a fundamental part of how humans engage with the world, from browsing the internet to using mobile apps. Develop- ing autonomous agents capable of seamlessly interacting with these interfaces as personal assistants has the potential to revolutionize daily life (Xie et al., 2024; OpenAI, 2025; Wu et al., 2024), making it more efficient and convenient.
Building GUT agents requires a combination of key capabilities: and interpreting GUT screenshots, grounding toouunding-translatinng human instructions into exe- cutable actions, and visual Maaning-ccaryying out tasks step by step to achieve the desired goal (Zheng et al., 2024b; Xu et al., 2024b; Ma et al., 2024). Among these, visual planning is the most challenging (Gur et al., 2023; Koh et al., 2024b; Yu et al., 2024). It demands breaking down complex instructions, like "check my GitHub repositories with the most stars," " into
CCo-fiistauthor Work done during JZ's visit to HKUST. Correspondance to Junlei Zhang (zhangjun- hh@@westlak.edu.cog and Junxian He (junxian@@cse.stthh)

---

Published as a conference paper at COLM 2025
Web2Cood: Pollwing gmnrate equiviient HIML and CSS
Multi modal Math provkied foxt Pltagibss
Non GUT Agent Traj plange in bothnion?
Charl Decument QA dobal. worth 20151
Tajectories
Nee-Ce Gut Perception
GUI Perceyttin
Analyne
Frone choose the
and Alently Bie Ies...
wrllin the and Mlnx
Mid- Training
Post- Training
>>
Base Model
Base Model Enhanced
GUI Agent Tasks
Focus
General Tasks
Reasoning...
Text Web Knowledge
Olympiad Math
CadeliO
Mathinsiruct
Mobile Trajectories
Gescraaz nost oclinn
Find the sum the posttove disteoms of
Liven a square mairit
How many mntees fenoe one needed?
giwn page
vedtr
sosolve Uha grobam. see stirt by beitng the dinksurs of We Ideettity give Starting
To praeiit the cutpur
The lnggh of the restangular Dowerbed x 4-1 meters Esc rectangolar
atep Hastng
for the ginas ispul.... [Mlomatei) Vour
statisann consulted the general policio....
cutppu camnet
apolagies
pesliction
Figure 1: Overview of the mid-training and fine-tuning process. Left: We first train the GUT agent on mid sod--traniing data, primarily from non-GUI domains, to investigate whether the enhanced capabilities can generalize to GUI agent tasks; Right: We perform post-training on GUI trajectory data.
precise, actionable steps. Vision-Language models (VLMs) naturally have the potential to serve as policy models for guiding GUI agent planning. With advanced prompting techniques, they can act as the foundational models for GUI agents (Zheng et al., 2024b; Song et al., 2024). However, most existing VLMs lack the reliability and stability needed to perform as effective GUI agents, often delivering subpar performance on benchmarks (Zhou et al., 2023; Deng et al., 2023; Koh et al., 2024a; Xie et al., 2024). For example, gpt4-o (Hurst et al, 2024) achieves a mere 15.6% on WebArena (Zhou et al., 2023), and Qwen2-VL-7B- Instruct (Wang et al., 2024a) could barely generate goal-aligned actions. Most errors stem from insufficient planning when tasks require multiple sequential steps. To address this limitation, researchers have been trying to improve the GUI policy VLMs by collecting or synthesizing GUI trajectory data (Xu et al., 2024b;a; Sun et al., 2024b; Ou et al., 2024). However, real-world GUI trajectory data is not readily available, making the acquisition of diverse and high-quality GUI datasets significant challenge. Additionally, synthesizing trajectories using large models often results in low-quality outputs, as even the most ad- vanced VLMs struggle to perform effectively on realistic GUI agent tasks. In light of this challenge, we focus our study on the critical question: How to improve the agentic abilities of VLMs for GUI tasks with more scalable data sources?
To this end, we introduce a mid-training stage to enhance the foundational agentic ca- pabilities of VLMs prior to fine-tuning them on limited subset of GUI trajectories for task-specific adaptation. Mid-training (illustrated in Figure 1) refers to an intermediate training stage between pre-training and fine-tuning, where models are enhanced with specialized capabilities for better adaptation.' While this approach has been successfully applied in previous studies across various scenarios involving LLMs and VLMs (Wang et al., 2024b; Sun et al., 2024a; Yang et al., 2025), it remains unclear which types of tasks can be effectively generalized to learning GUI agents, given the complexity and typically extensive context associated with such tasks. Previous research exploring alternative data sources has primarily focused on GUI-specific resources, such as web tutorials and GUI captions (Chen et al., 2024; Xu et al., 2024a; Ou et al., 2024), which falls short on representing
'hee definition of mid-training can differ, conceptually is similar to continual pretraining instruction tuning data in our context.
2

---

Published as a conference paper at COLM 2025
agentic planning abilities. In this work, investigate diverse set of mid mid-training data domains to evaluate their impact on learning GUI agents. These domains include general image perception, chart understanding, multimodal reasoning, and text-based tasks such as mathematics and programming.
We evaluate eleven datasets seven multimodal and four knowledge retrieval, and perception ($3.1). Samples are collected per domain, followed by separate mid-training on each, and fine-tuning on GUI trajectory dataset. By standardizing all datasets to generate high-level action thoughts before grounding to actions, we enhance planning transfer. continuous optimizer ensures smooth training transitions and mini- mizes forgetting. Analysis on mobile and web benchmarks validates the effectiveness of our mid-training approach. Pure text mathematical datasets show the largest gains, improving World by 5.4% and WebArena by 5.6%, demonstrating reasoning, abilities could transfer cross-domain. Coding datasets boost performance by around 3.0% on both tasks. Surprisingly, visual perception datasets yield modest gains, likely due to existing VLMs' strong visual capabilities. Based on these insights, we introduce GUIMid, a 300k dataset combining the four best-performing domains. GUIMid achieves SOTA AndroidWorld in pure-visual settings and improves Qwen2-VL to GPT4-o level performances on web browsing, with overall gains of 12.2% and 8.0% on WndroidWorld and WebArena.
focusing reasoning,
Vision-based GUI Agent Framework
Pure vision-based GUI agents. Previous work (Gur et al., 2023; Zheng et al., 2024b; Xie et al., 2024; Zhou al., 2023) has largely relied on structural text-based GÚI representations, such as HTML or accessibility trees. In contrast, we focus on the more challenging pure- vision setting, where vision-based agents take screenshots and task descriptions as input, generating coordinate-baset actions directly within pixel space. This pure- pure-vision approach provides key advantages: (1) it eliminates dependencies on backend structures, enabling cross-plattomn operation while avoiding the noise often present in accessibility trees, and (2) it aligns more closely with human interaction patterns, allowing for seamless integration into real-world workflows.
The inference process can be formalized as a specialized instance of Partially Observable Markov Decision Processes (POMDPs), represented by tuple S, O, T), where g de- notes the task goal, S the state space, A the action space, O the observation space (visual feedback from the screen), and T S A S the state transition function. At each time step f, the agent executes decisions according to policy π,, which integrates the task goal g, memory m, = {oj, 0, - 0 < < (capturing the history of actions and observations), and the current observation o,. The agent's trajectory, denoted as 7 = [sp, Ap, .,., st), emerges from the policy and environmental state transitions, as formulated by: (T = p (50) II
T π (a; g, m:) T (5141 a,)
(1)
1=0
We then introduce specific implementations of the observation and action space in our vision-based GUI agent framework.
The action space. Our GUI agent employs coordinate-based action space to ensure cross- platform compatibility and realistic human-like interactions. The policy model generating actions consists of two components: a planner model and a grounding model. The planner model first generates high-level action description following planning-rich thought, i.e. "Task instruction <thought> <high-level action>" Then the grounding model maps the high-level action into screenshot manipulations.
For instance, to search for "GUI agents" in search engine, the generated content is:
<thought>: To find unlabeled issues in the metaseq GitLab repository, click the "Issues" tab in the main navigation menu, then filter for issues without labels.
3

---

Published as a conference paper at COLM 2025
Domains
Ability
Datasets
Samples
Type
Vision-and- Lannn-aa--Lagguag Modality
Chart/Document QA
Perception
IntographicVQA. (Guo et al., 2024)
2,184
Instruction, Thought' Answer
Ureader QA (Guo
2024)
53,794
Instruction, Thought, Answer
MPDecVQA (Tito al, 2023)
471
Instruction, Thought, Answer
MathV360k (Liu et al,, 2024b)
93,591
Instruction, Thought, Answer
Perception
Perception
2023)
Instruction, Thought' Answer
DUE (Borchmann
2021)
143,854
Instruction, Answer
GUI Perception
Perception
MultiUI (Ciu et al, 2024a)
150,000
Instruction, Answer
Web Screemshot2oode
Perception
Web2Cade (Yun et al,, 2024)
150,000
Instruction, Answer
Math
Reasoning
(Zhang et al., 2024h)
150,000
Instruction, Thought, Answer
Multi-round Visual Conversation
Interaction
SVIT (Zhao al.,
150,000
Instruction, Thought, Answer
Non-GUI Agent Trajectories
Interaction
AllWorld (Guo et al,, 2024)
51,780
Instruction, Thought, Answer
Language Modality
MathInstruct
Reasoning
MathInstruct (Yue et al., 2023)
150,000
Instruction, Thought, Answer
Olympiad Math
Reasoning
NuminaMath el, 2024)
150,000
Instruction, Thought,
Codel/O Web Knowledge Base
Reasoning
Codel/O (Li et al., 2025)
150,000
Instruction, Thought, Answer
Knowledge
Synatra (Ou et 2024)
99,924
Instruction, Thought, Answer
AgenfTrek (Xu et al,, 2024a)
30,076
Instruction, Thought, Answer
Table Statistics of the domains and corresponding datasets used in mid mid-training, (*) indicates that some instructions in the dataset do not require "Thought" (e.g., "Answer concisely with one word or phrase.").
<high-level action>:
"Element Description": "Click the Issues tab in the main navigation menu", "Action": "click",
}
<grounded action>: Click [coordinate_x 0. 12] [coordinate_y 0.07]
We use JGround-V1-7B (Gou et al., 2025) for grounding, and our trained policy model for thought and high-level action generation. This separation also enables better transfer of planning ability through mid-training (as detailed in $3) in addition to flexible inclusion of new actions. The details of action spaces for mobile and web tasks be found in Table 9, 10.
The observation space and memory. The observation space is visual-rich in our frame- work, comprising of a screenshot of the current screen and simple meta-information (i.e., the current URL for web tasks). To augment agent memory, also provide the model with history of previous actions, using the concatenated output of planner generated high-level actions, e.g. "step 1: click 'the search results titled with wikipedia'; step type 'GUI Agent' into the search bar at the top of the page".
Breaking the Data Barrier via Mid-Training
Despite advancements in vision-language models, GUI agent training faces challenges due to limited high-quality trajectory data. We introduce a mid-training stage between general pre-training and task-specific post-training. This approach leverages abundant data from adjacent domains- -amainns-mmaee perception, chart understanding, multimodal reasoning, develop foundational capabilities before GUI-specific adaptation. Our two-step strategy includes mid-training on scalable data sources ($3.1) followed by fine-tuning on a small GUI trajectory dataset ($3.2). Our optimized training procedure introduced in $3.3 ensures consistent generalization to GUI agent skills. We briefly introduce these datasets and the training procedure in this section, and more details can be found in Appendix B, C.
and programming-

---

Published as a conference paper at COLM 2025
Domains
Datasets
Samples
Type
Web
OS-Genesis (Web) (Sun et al., 2024b)
3,789
Instruction, Thought, Action
MM-Mind2Web (Zheng al., 2024a)
21,542
Instruction, Thought, Action
VeeualWebArema (Koh et al., 2024a)
3,264
Instruction, Thought, Action
Mobile
OS-Genesis (Mobile) (Sun et al., 2024b)
4,941
Instruction, Thought, Action
Aguvis (Xu al., 2024b)
22,526
Instruction, Thought, Action
Table 2: Statistics of the reblne domains along with the corresponding GUT trajectory datasets used in post-training.
3.1 Mid- Mid-Training Data
We collect 150k diverse training samples for each domain to study cross-domain general- ization. For the non-GUT agent domain, we included 51k samples due to the scarcity of agent trajectories. The mid-training domains are listed in Table 1. For vision-language tasks, we include: Chart/ Shart/Documens QA (Guo et al., 2024; Tito et al., 2023) and Multi-modal Math (Zhang et al., 2024b) for fine-grained understanding and visual reasoning; Non-GUI Perception tasks (Ye et al., 2023; Borchmann et al., 2021) including Document OCR for fun- damental visual understanding; Web Screenshot2Code (Yun et al., 2024) for structured web screenshot interpretation, and multi-turn data from Visual Conversations (Zhao et al., 2023) and Non-GUT Agent Trajectories (Guo et al., 2024) to enhance VLM interactive capabilities.
Complementing these, we include pure-text data featuring more reasoning-intensive, and knowledge-rich tasks, including from medium-dificculy (MathInstruct (Yue et al., 2023)) to challenging (Olympiad Math (LI et al., 2024)) mathematical reasoning, Code I/O (Li et al., 2025) to develop procedural reasoning through code generation, and Web Knowledge Base (Ou et al., 2024; Xu et al., 2024a) to inject domain knowledge.
3.2 Post-Training GUI Trajectory Data
For post-training, we used high-quality GUT trajectories from state-of-the-art systems across platforms. We incorporated web and mobile data from OS-Genesis (Sun et al., 2024b), Aguvis (Xu al., 2024b), and MM-Mind2Web (Zheng al., 2024a), plus 3.2K manually annotated steps from VibualWebArena (Koh et al., 2024a). Our final dataset contains 28K web samples and 27K mobile samples, detailed in Table 2.
3.3 Training Method
We employ two-stage training approach consisting of mid-training followed by fine-tuning on GUT trajectory data. Both stages are integrated under single optimizer and learning rate schedule. During the mid-training stage, we mix the post-training GUT trajectory data into the mid-training domain data (e.g. CharlQA) to mitigate potential forgetting issues, which we will analyze empirically in $4.4. This mixing is particularly crucial when there exists substantial domain gap between the mid-training data and GUI trajectory data, especially when the former is primarily text-based (see Figure 4 for comparison). Additional training hyperparameters and details are detailed in Appendix D.
4 Experiments
4.1 Experimental Setup
To explore how non-GUI trajectory data can enhance VLMs' foundational agentic capa- bilities, we study 7 multi-modal domains and 4 language domains (Table 1). For the post-training data, we collect 56K high-quality data (Table 2). Following State-Of-The-Art (SOTA) works (Xu et al., 2024b; Sun et al., 2024b; Gou et al., 2025), we employ Qwen2-VL-7B- Instruct (Wang et al., 2024a) as our backbone. We first train our model on the mid-training datasets, then fine-tune it on GUT trajectories. To study whether these separate domains can
5

---

Published as a conference paper at COLM 2025
Task: Post my question, "is necessary NYC", in subreddit where I'm likely get
Model w/o Mid-Training
Thought In current situation,
entered
the search query "is car necessary in NYC" and clicked search. The screenshol shows relevant resalts, contirming I'm the right track. nest step to review results for the posis or discussions that answer my question. can now scroll through them for useful insights.
Scroll Down
Incorrect State
Action:
[Whong Actios
Scroll down the current page.
To Incorrec
Stuts)
Model Mid-Training
Thought The visible elements suggest I'm already in comment section with other typed my question, step submit it. However, the screenshot does not show clear submit button, so I'll look for button utilized to post comments,
Click 'Submil'
users,
Button
Correct State
likely labeled "Submit", "Tust".
Action:
Click Submit button for posting a comment in the subreddit.
nighh Action
To Correct Statc)
case illustrating the performance of the Model w/o Mid-Training and the Model w/ Mid- Midd-Taainnn under the same task. The middle text shows the model's thought process and the action taken, while the screenshots on the left and right represent the screen states before and after the action, respectively. The model with middle training (bottom) successfully reflects on errors and generates correct actions from error states, while the model without mid-training (top) fails to recover from such states.
Figure 2:
be combined to achieve superior performance, we randomly sample total of 300K exam- ples from high-performing domains (150K from MathInstruci, 20K from Codel/O, 50K from Olympiads Math, and 80K from Multi-modal Math) to create a consolidated mid-training dataset called GUIMid. During the mid-training stage, we mix the GUI trajectory samples into our mid-training dataset to make the training stable.
4.2 Evaluation Environment
We use Worro (Rawles et al., 2024) and WebArena (Zhou et al., 2023) our testbeds, as their dynamic nature makes them ideal environments for study the effects of different domains during the mid-training stage. For WebArena, we use the version processed in AgentBoard (Ma et al., 2024). We opt for interactive benchmarks over static bench- marks (Deng al., 2023; Li al., 2024) based on two considerations: (1) Static environments inherently contain annotation bias, wherein multiple valid paths exist to complete a task, yet annotator preferences result in only one path being labeled as correct for each step (e.g., finding a playlist through either search functionality or category navigation); (2) Our mid-training approach primarily enhances fundamental capabilities such as reasoning and perception rather than in-domaín knowledge for example, improving the model's 's ability to complete tasks in new websites through exploration and reflection. In contrast, step-by- step optimal action annotations rely heavily on the model's prior knowledge about specific websites or applications. We provide more evaluation details are provided in Appendix E.
4.3 Results on Separate Domain
We report the performances of different domains as mid-training on Webarena and Android- World (Table 3). Our analysis reveals several important findings:
Mathematical data generally improved the most: Both language-only and vision- language mathematical reasoning tasks demonstrate substantial performance improve- ments across benchmarks. In the language modality, models mid-trained with MathInstruct achieve the highest success rate on WebArena (10.9%) and strong performance on An- droidWorld (14.4%). Similarly, in the vision-language domain, Multi-modal Math shows

---

Published as a conference paper at COLM 2025
Domains
Observation
WebArena
AndroidWorid
PR
SR
SR
GUI Post-Training Only
Image
26.3
6.2
9.0
Public Baselines
GPT-1o-2024-11-20
Image
36.9
15.6
11.7
OS-Genesis-7B
Image Accessibility Tree
17.
AGUVIS-72B
Image
26.1
Claude3-Haiku
Accessibility Tree
26.8
12.7
Llama3-70b
Accessibility Tree
35.6
12.6
Geminil.5-Flash
Accessibility Tree
32.4
11.1
Vision-and-Languaee Modality
Chart/ Document QA
Image
24.6
6.2
15.3
Non-GUI Perception
Image
28.7
7.6
14.0
Perception
Image
27.4
Web Screenshoi2Code
Image
28.0
6.6
9.9
Non-GUI Agents
Image
30.8
8.5
13.5
Multi-modal Math
Image
30.4
8.5
15.3
Multi-round Visual Conversation
Image
30,0
9.0
12.6
Language Modality
MathInstruct v
Image
31.9
10.9
14.4
Olympiad Math
Image
31.5
8.5
13.1
Codel/O
Image
29.2
9.0
14.9
Web Knowledge Base
Image
31.3
Domain Combination (Sampled data from domains)
GUIMid
Image
34.3
9.5
21.2
Table 3: Progress Rate (PR) and Success Rate (SR) of Quen2-VL-7B-Instee across various domains using two-stage training strategy. Color-coded cells (green/ red) are employed to denote improvements or declines relative to the post-training only baseline, with deeper shades indicating larger score shifts.
impressive gains of 8.5% on WebArena and 15.3% on AndroidWorld. This consistent pattern suggests that mathematical reasoning capabilities, regardless of input modality, improve generalizable reasoning skills that transfer effectively to GUI agent tasks. A case study of a model mid-trained by MathInstruct (Yue et al., 2023) is shown in Figure 2. The task asks the agent to post the question "Is car necessary in NYC" on the Reddit website. Both agents initially navigated to incorrect pages. However, the MathInstruct-traine demonstrated better reasoning by thinking, "However, the screenshot does not show a clear submit button, so I'll look for button utilized to post comments," " and subsequently located the correct "submit" button. In contrast, the baseline model without mid-training became stuck the incorrect page and continued scrolling up and down fruitlessly.
agent
Strong cross-modal and cross-domain transfer: Language-only tasks demonstrate remark- able effectiveness for multi-modal GUI tasks. Olympiad Math achieves 31.5% progress and 8.5% success on WebArena, outperforming most vision-language tasks. Similarly, Codel/O reaches 14.9% success rate on AndroidWorld. Web Knowledge Base shows effectiveness primarily in the web domain, likely due to its focus on web-specific information rather than mobile. These results suggest that conduct mid-training on text-only mathematics, code, and knowledge data can enhance fundamental abilities for GUI agents, even for multi-modal tfferno valuable insights for addressing the challenge of limited GUT in-domain training data.
Non-GUI Agents data, despite its relatively small size 50K samples), demonstrates strong performance (30.8% progress, 8.5% success on WebArena; 13.5% success on IndroidWorld This suggests that agent interaction patterns can transfer to some extent. Multi-round Visual Conversation yields balanced improvements across benchmarks (9.0% WebArena, 12.6% on WodroidWorld)), Chart/ Document QA performs good on AndroidWorld(1) the web setting .suggesting the different requirement of digital platforms. However, the Web Screenshot2Code and GUI Perception data does not help a lot, this may due to Owen2--V--7--Instruue already be trained on GUI perception data.
but bad
Results on Domain Combination Our motivations to leverage larger quantities of non-GUT datasets to enhance VL.Ms' foundational capabilities, thereby enabling more
7

---

Published as a conference paper at COLM 2025
Performance on Worro of Varying Data Scales
Performance on WebArena of Varying Data Scales
Sucoess
Pragros
Success Rate (%)
Success Rate (36
Progresse (9R)
515
15-
10-
25-3
26
50k
100k
150%
200s
250k
300k
50k
103k
180k
200k
250k
300k
Mid--iaining Data
Mid-Training Data
Figure Performance of models trained on GUIMidwith different scales.
Training Strategy
Success Rate (%)
GUI Post-Training Only GUI Mid + Post-Training
6.1
10.2
Table 4: Performance Comparison on Mind2Web-Liive
Model
Allrecipes
Amazon
Apple
ArXiv
GUT Post- -ratnTTainnn Only
30.0
10.0
20.0
50.0
GUT Mid Post-Training
50.0
40.0
40.0
50.0
BBC News
Booking
Cambridge Dictionary 20.0
Coursera
GUI Post- Trainiraining Only
10.0
10.0
50.0
GUI Mid Post-Training
20.0
20.0
70.0
50.0
ESPN
GitHub
Google Flights 0.0
Google Map 10.0
GUI Post-Training Only
10.0
20.0
GUI Mid Post-Training
20.0
40.0
10.0
20.0
HuggingFace
Wolfram Alpha 20.0
Average
GUT Post- cess-Trannig Only
30.0
20.7
GUI Mid Post- fost-Training
50.0
50.0
37.9
Table 5: Performance Comparison on WebVoyager.
effective learning from limited GUI trajectory data. To validate this, we combine the top- performing domains to construct combined mid-training dataset: GUIMid, which consists of randomly sampled data (150K samples from MathInstruct, 20K from Codel/O, 50K from Olympiads Math, and 80K from Multi-modal Math). We explore the scaling law of GUTMid. Specifically, for the 300K sample mid-training dataset, we maintain the same ratio of non- GUI to GUI data as in our 150K sample experiments to ensure training stability. Instead of introducing new GUI data, we duplicate the existing GUI trajectory data. In Figure 3, the x-axis represents the effective mid-training data volume, calculated as the total training data volume multiplied by the fixed proportion allocated to mid-training samples. The model exhibits scaling laws with increasing mid-training data volume on both AndroidWorld and WebArena. While we observe slight decrease in WebArena Arena success rates around the 300K sample mark, the progress rate metric provides a more nuanced assessment of performance improvements. This metric, which captures fine- fine-grained capability development, shows consistent growth: from approximately 26.4% at 21.8K samples to 31.6% at 152K samples, and further to 34.3% at 300K samples. This steady upward trajectory indicates the effective of the scaling law of GUMMid.
To further validate WebVoyager (He et al., 2024) without additional fine-tuning. On Mind2Web-Live, we ran 51 accessible tasks, excluding those restricted by website access, and because ground-truth
method, we evaluated it Mind2Web-Live (Pan et al., 2024) and
8

---

Published as a conference paper at COLM 2025
12
1.2
0.5
0.8
1.0
1.0
0.5
0.7
0.8
0.7
0.6
0.5
0.4
0.2
0.3
8000
2000
Steps
Figure Comparison of training loss between two training strategies: (a) and (c) show the mixture of GUI trajectory data during mid-training, while (b) and (d) are not.
Domains
WebArena
AndroidWorld
PR
SR
SR
MathInstruct (no mixing)
24.0
9.0
9.0
MathInstruct (mixing)
33.6
8.5
14.4
Multi-modal Math (no mixing)
25.4
6.2
14.9
Multi-modal Math (mixing)
30.4
8.5
15.3
Table 6: Progress Rate (PR) and Success Rate (SR) with and without GUI trajectory data
integration during the mid-training stage. "mixing" indicates the mid-training data is mixed with GUI trajectory data, while "no mixing" indicates was not.
Domains
Difficulty
WebArena
AndroidWorld
PR
SR
SR
Orca-Math Randomly Sampled Data
Easy
31.9
10.0
9.9
Middle
30.6
9.5
10.8
Olympiad Math
Hard
31.5
8.5
13.1
Table 7: The impact of mathematical difficulty in mid-training data. We sample three
subsets from the NuminaMath dataset based on their difficulty levels.
labels can become obsolete as websites evolve-we report success rate as the primary metric. On WebVoyager we randomly sampled ten tasks from each of fourteen domains (140 tasks in total), omitting Google Search due to reCAPTCHA limitations, and followed the benchmark's standard human-evaluation protocol. Across both benchmarks our method delivers substantial gains: on Mind2Web-Live (10.2 vs 6.1), while on WebVoyager it yields an 83 % average relative improvement (37.5 99 vs 20.7), with particularly large gains in domains that demand complex reasoning, such as Cambridge Dictionary (+250 %) and Wolfram Alpha (+150 %).
achieves 67 % relative improvement
4.4 Ablation Study
The effects of adding GUI trajectory data to the mid-training stage. We compare the loss curves (Figure 4) and performances (Table when mixing/ no mixing GUI trajectory data into the mid-training stage. Due to the domain gap between mid-training data and GUI trajectory data, domain switching may cause sharp fluctuations in the loss curve, leading to instability (as shown in Figure 4, plots (b) and (d)), or even cause gradient overflow to NaN values. In contrast, plots (a) and (c) demonstrate significantly smoother training dynamics after merging the data. Table 6 reveals that after mixing GUI trajectory data, both MathInstruct and Multi-modal Math show significant performance improvements. In WebArena, MathInstruct's progress rate increased from 24.0 to 33.6 after mixing. Also, in AndroidWorld, Multi-modal Math's success rate improved from 14.9 to 15.3 after mixing.
Performance with Respect to Mid-training Data Difficulty We analyze performance across three NuminaMath subsets of varying difficulty: the relatively easier Orca-Math (Mitra et al., 2024), the highly challenging Olympiad Math (I.I et al., 2024), and medium-

---

Published as a conference paper at COLM 2025
Thiining Strolegy
Poet-training Data Ratio
Web-Arena (Progress Rate) WebArena (Success Rate)
WndroldWorld (Success Rate)
GUT Training Only
100%
26.3
6.2
9.0
GUT Mid Post training
20%
25.5
66
10.8
GUT Mid Post-traiming
60%
27.0
8.1
126
Table 8: Comparison of post-training data efficiency.
difficulty random subset. Table shows that performance in mobile environments generally improves with increased data difficulty, while WebArena exhibits no clear correlation. This difference likely stems from AndroidWorldh more diverse interaction patterns.
Post-training efficiency comparison with and without mid-training. We controlled the amount of in-domain trajectory data used in both the mid-training and post-training stages by randomly sampling different proportions of the original post-training dataset. Table 8 shows that our GUL Mid Post-training approach achieves performance comparable to the baseline while using only 20 % of the post-training data (WebArena: progress rate 25.5 vs 26.3, success rate 6.6 vs 6.2; AndroidWorld: 10.8 vs 9.0).
5 Related Work
Recent advancements in GUI agents have spurred the development of di- verse benchmarks. Early non-interactive benchmarks for web (Mind2web (Deng et al., 2023), WebLINX (Lu et al., 2024b), Web (He et al., 2024)) and mobile (AITW (Rawles et al., 2023), AITZ (Zhang et al., 2024a), AndroidControl (Li et al., 2024)) suffer from potential an- notator bias in tasks with multiple valid solutions. To mitigate this, interactive benchmarks were introduced for desktop and mobile (OSWorld (Xie et al., 2024), AndroidWorld (Rawles et al., 2024)) and web environments (WebArena (Zhou et al., 2023), VisualWebArem (Koh et al., 2024a), WorkArena (Drouin et al., 2024), WebCanvas (Pan et al., 2024)). However, these frameworks pose challenges for evaluating specific non capabilities like reason- ing. For instance, VisualWeb Arena's trajectory-level success metrics can mask reasoning improvements when perception is the primary bottleneck. While WebCanvas (Pan et al., 2024) aims to address perceptual challenges, many of its tasks are outdated due to live site updates. Therefore, to isolate and measure specific capabilities such as reasoning, we adopt the subgoal-annotated versions of WebArena and AndroidWorld from AgentBoard (Ma et al., 2024), which enables a more granular evaluation independent of perceptual limitations.
GUI Agents.
Mid-Training Mid-training refers to the stage between pre-training and post-training designed to enhance foundational capabilities (Abdin et al., 2024), whereas post-training optimizes models to adapt to specific downstream tasks. Recent works like Phi 3.5 (Abdin et al., 2024), Yi-Lightnnng (Wake et al., 2024), OLMe 2 (OLMo et al., 2024) and Codel/O (Li et al., 2025) have demonstrated the effectiveness of strategic mid-training interventions- use mid-training to enhance the foundational abilities like context length, multilingual knowledge and code understanding. For GUI agents, where high-quality trajectories are scarce, mid-training becomes particularly valuable. However, current research lacks systematic exploration of out-of-domain mid-training techniques specifically tailored for GUI agents-a critical gap that our work addresses.
Conclusion
In this paper, we propose using mid-training to enhance GUI agents' foundational capa- bilities, enabling more effective learning from limited trajectory data. While GUI-specific data remains scarce, there is much more non-GUI data such as mathematical reasoning, and coding. We explore 11 diverse non-GUT tasks, demonstrating for the first time the significant impact of mathematical reasoning data on GUT task performance, effectiveness of text-only mathematical reasoning in improving multimodal GUI agent capabilities. Our findings also reveal how non-GUl tasks such as text web knowledge and embodied agent trajectories can substantially enhance GUI agent performance. These results provide valuable insights for the research community in constructing more effective GUI i training pipelines. We will open-source all data, models, and training recipes to facilitate future research in the GUT agent domain.
well as the surprising
10

---

Published as a conference paper at COLM 2025
7 Acknowledgements
This project is supported by NSFC Grant 62306177 and the program "Development of Digital Therapeutic Systems Based on Large Language Models and Their Application to Major Brain Diseases" (Project No. 2025C01104).
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone, arxio preprint drXiv:2404.14219, 2024.
Lukasz Borchmann, Michat Pietruszka, Tomasz Stanislawek, Dawid Jurkiewicz, Michal Turski, Karolina Szyndler, and Filip Gralinski. Due: End-to-end document understanding benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, al. Guicourse: From general vision language models to versatile gui agents. Xiv preprint arXiv:2406.11311 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint ArXiv:2110.1 2021.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-28114 2023.
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable web agents at solving common knowledge work tasks? Xiv preprint arXiv:2403.07715, 2024.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations, 2025. URL httpttt://ppeeeeiiieeee www.w..eneooornnniioee.... id-kknogaisic..
Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. 2024. URL https://arxiv http:://aaxxvvv.ogaaaaaa 2412. 05237.
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint ar arXio:2307.1288. 128666..,1166, 2023.
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6864-6890, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, kila Welihinda, Alan Hayes, Alec Radford, et Gpt-4o system card. Xiv preprint ar Xiv::244002222 2024.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEÉ conference on computer vision and pattern recognition, pp.
2018.
11

---

Published as a conference paper at COLM 2025
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. Visualwebarena: Evalu- ating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeling of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 881-905, 2024a.
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arxio preprint ar Xio:2407.001776 2024b.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Nu- minamath. [https: nonecctnuminaa/aec aimo- http:///.....ooooopsssseee//////
co htt.://ae...otat.o/o///mmnnnnnna
(https://gi com/ dataaadddatttttddaaaaaa 2024.
Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codei/o: Condens- ing reasoning patterns via code input-output prediction. arr preprint ar arXiv:2502.07316, 2025.
Wei Li, William Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on ui control agents. Advances in Neural Information Processing Systems, 37:92130-92154, 2024.
Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Graham Neubig, and Xiang Yue. Harnessing webpage uis for text-rich visual understanding, ar Xiv preprint arXiv:2410.1382,, 2024a.
Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, and Junxian He. Diving into self-evolving training for multimodal reasoning. arXiv preprint arXiv:2412.17451, 2024b.
Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint ar arXiv:2406.08451 2024a.
Xing Han Lu, Zdeněk Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. In International Conference on Machine Learning, pp. 33007-33056. PMLR, 2024b.
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn Ilm agents. Advances in Neural Information Processing Systems, 37:74325-74362, 2024.
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1527-1536, 2020.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint ar 1724:22402.14830, 2024.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint XXie:2501000655 2024.
OpenAL. Computer-Using Agent. http:///www....ooooee/////t/tttr January 2025.
wtt.
Tianyue Ou, Frank F. Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indi- rect knowledge into direct demonstrations for digital agents at scale. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. wewperrrvvee..ettfouuriiititt
URL https:
12

---

Published as a conference paper at COLM 2025
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments, arXiv preprint ar arXiv:2406.12373, 2024.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36:59708-59725 2023.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, al. Android- world: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXio:2405.14573, 2024
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See Kiong Ng, Lidong Bing, and Roy Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024,
pp. 4663-4680, 2024.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Inter- active Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. URL https://arxivv wwtp:/sgg//////ca/ 207766
Yueqi Song, Frank F Xu, Shuyan Zhou, and Graham Neubig, Beyond browsing: Api-based web agents. 2024.
Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jian- ing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: P'aradigms, advances and beyond. arviv preprint ar arXiv:2403.14734 2024a.
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412.19723, 2024b.
Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Hierarchical multimodal transform- ers for multipage docvqa. Pattern Recognition, 144:109834, 2023.
Alan Wake, Bei Chen, CX Lv, Chao Li, Chengen Huang, Chenglin Cai, Chujie Zheng, Daniel Cooper, Fan Zhou, Feng Hu, et al. Yi-lightning technical report. arXiv preprint 2rXiv:2412.01253, 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qw en2-vl: Enhancing vision-language model's perception of the world at any resolution. Xrv preprint arXiv:2409.12191, 2024a.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. In Forty-first International Conference on Machine Learning, 2024b.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. ar Xiv preprint ar 4rXio:2402.07456, 2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040-52094, 2024.
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. aix preprint ar axXio:2412.09665, 2024a.
13

---

Published as a conference paper at COLM 2025
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2024b.
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents. ar preprint 1r1::22502.1310, 2025.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. Xrvi preprint ar orXt:2330005122. 2023.
Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. Exact: Teaching ai agents to explore with reflective-mets and exploratory learning. Xiv preprint ar afXt:2410000205, 2024.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arxiv preprint ar 2Xtx:2309.05653 2023.
Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, et al. Web2code: A large-scale webpage-to-con dataset and evaluation framework for multimodal llms. ar preprint arXi::2406.20098, 2024.
Jiwen Zhang, Jihao Wu, Teng Yihua, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thouge for gui agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, Pp. 12016-12031, 2024a.
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming
Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, et al. Mavis: Mathematical visual instruction tuning. aiv e-prints, pp- arXiv-2444 -207, 2024b.
Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arviv preprint ar Xrio:2307.00088) 2023.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. Xiv preprint ar Xrx::22401 .01644, 2024a.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-Av(ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, 2024b. URL /tttps::/opeereviiii wttp://ewww...//eoooriiiiieeieie
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXio:2307.13854, 2023.
14

---

Published as a conference paper at COLM 2025
Action
Description
click [[x] [y]]
Click at coordinates (x, y).
type [[x] [y]] [value]
Type content into field by coordinate.
scroll [[x] [y]] [value]
Scroll the page or specific element. Navigate the previous screen or page.
go_back
go_home
Navigate to the home screen.
long_press[[x] [y]]
Long press an coordinate.
enter
Press the Enter key.
open_app [app_name]
Open an app by [app_ name].
wait [value]
Wait for the screen to update for [value] seconds.
stop [answer]
Stop the task with goal status or answer.
Table 9: The action space for mobile tasks.
Action
Description
click [[x] [y]]
Click at coordinates (x, y).
type [[x] [y]] [value]
Type content into field by coordinate. Clear the content of an element.
clear [[x] [y]]
hover [[x] [y]]
Hover over element by coordinate.
press [keys]
Press key combination (e.g., Ctrl+v).
scroll [value]
Scroll the page.
new_tab
Open
tab.
page_focus [tab_index]
Switch to specific tab.
close_tab
Close the current tab.
goto [url]
Navigate URL
go_back
Go to the previous page.
go_ forfordddd
Go to the next page.
stop [answer]
Issue this action when the task is considered complete.
Table 10: The action space for web tasks.
A Details of the GUI Agent
Action Space. During the data processing stage, we standardized the action spaces sep- arately for mobile and web domains to achieve unified representations suitable for joint training. In the evaluation stage, we further aligned actions across different benchmarks by mapping them to the corresponding standardized action spaces. The actions involved in both the training and evaluation phases fall within the categories detailed in Table for the mobile domain and Table 10 for the web domain.
Observation space. To simulate real-world human interactions with GUI environments and to better investigate whether tiiddleg data enhances the reasoning ability of GUT agents, the observation space in this study consists solely of visual information. Specifically, during task execution, GUI agents receive only visual feedback from the screen and historical action information, without leveraging any additional textual information from the environment (e.g., DOM or accessibility trees). Such textual information could introduce extra information gain, potentially confounding the effect of middle-training data.
B Details of Mid-Training Data
Vision-language data generally comprises paired visual and textual information, such as image captions, annotated screenshots, and visually grounded instructions. Considering the inherently multi-modal nature of GUI Agents, leveraging vision-language data may facilitate better alignment between visual and textual modalities, potentially improving agents' comprehension of and interaction with graphical interfaces. Our primary focus for vision-language modalities includes:
(1) Chart/ Document Question Answering: Data in this category enhance the model's ability to perform reasoning-intensive tasks over visual representations of struc-
15

---

Published as a conference paper at COLM 2025
tured charts and documents. Our training data is constructed by randomly sampling approximately 56K samples from InfographicVOr and Ureader QA in MAmmoTH- VL (Guo et al., 2024), 500 samples from MPDocVQA (Tito et al., 2023), and 93.5K samples from the warm-up data of MathV360k (Liu et al., 2024b). In total, this process yields dataset of 150K samples for mid-training.
(2) Non-GUI Perception:
This category enhances the model's perception capabilities for non-GUI images, such posters, tables, and documents. Given the abundance of non-GUI perception datasets in online databases (e.g, documents (Kafle et al., 2018), posters (Ye et al., 2023), and plots (Methani et al., 2020)), investigate whether leveraging such data can improve performance on GUI-related tasks. Specifically, we construct the training data by integrating 6.1K samples from Ureader OCR (Ye et al., 2023) with 143.9K randomly selected samples from DUE (Borchmann al., 2021), yielding a total of 150k samples for the mid-training phase.
(3) GUI Perception: This category aims to enhance the model's perceptual capabilities for GUI images. To achieve this, the training dataset is constructed by randomly sampling 50K instances from each of the Action Prediction, Webpage Question An- swering, and Image Question Answering subsets within MultiUI (Liu et al., 2024a), resulting in a total of 150K samples for the mid-training stage.
(4) Multi-modal Math: Math data (Cobbe et al., 2021; Shi et al., 2024) has been widely used to enhance the reasoning capabilities of LLMs and VL..Ms. We explore whether incorporating multi-modal mathematical data can further enhance the planning capabilities of GUI-Agent. Specifically, we construct the training dataset by ran- domly sampling 150K multi-modal math problems from the Mavis dataset (Zhang et al., 2024b), which comprises high-quality mathematical questions accompanied by comprehensive reasoning processes.
(5) Multi-round Visual Conversation:
Agent trajectories often consist of multiple steps, requiring the model to understand and memorize previous steps to inform current decisions. Multi-round visual conversation data exhibits similar characteristics, as generating response in a given turn typically depends on the context of prior turns. We examine whether multi-turn multi-modal question-answent the performance of GUI-Agent tasks. Specifically, we construct the training dataset by randomly sampling 150K multi-turn dialogues from the SVIT dataset (Zhao et al., 2023), which comprises multi-turn question-answering interactions involving intricate, image-based reasoning.
data can enhance
(6) Web Screenshot2Code HTML code and corresponding web screenshots are readily available, providing rich information regarding the structure and interactivity of web elements (e.g., whether an icon is clickable or hoverable). We investigate whether leveraging such accessible data can enhance the performance of GUT- Agents on downstream tasks. Specifically, we construct the training dataset by randomly sampling 150K web screenshot-HTML. code pairs from the Web2Code dataset (Yun et al., 2024).
(7) Non-GUI Agent Trajectories: With the rapid development of Embodied AI, growing amount of non-GUT agent data is becoming available in domains beyond the web, such as household tasks (Shridhar et al., 2021). We investigate whether data from these domains can benefit GUI Agent tasks. Specifically, due to the limited availability of such data, we utilize all 51K samples from the AlfWorld subset of AA-mmoT-VV (Guo et al., 2024) as mid-training data for our experiments.
Text data is often more readily available and abundant compared to vision-language data, widely accessible through sources such as the internet. We investigate whether pure text data can enhance the capabilities of GUT Agents. For the text modality, our primary focus includes the following:
(1) MathInstruct: Text-based math datasets (Cobbe et al., 2021) are commonly used to enhance the reasoning capabilities of models. We randomly sample 150K examples from the CoT category of MathInstruct (Yue et al., 2023) as mid-training data.
16

---

Published as a conference paper at COLM 2025
Agenr
[ [[ng]
Actiona:
This loku cleman kontend) Use this
welpagges
fresh
25 746. 9.1 de accouna
id.
IT:00
16. 1i2 Hil Fabaken
detaut, "Evte" afhr
hyping
(sontend (t.
doer (dit Use this clerr the content a inpus bose
(S318:1) 18627. 218. 3E1. 3881 Jelect culeyaa.
hover jul Hover over enaat wits id.
press Dey pumbd Sindltes prosiing of key san binutioo
on the beyboard lng, Chieus
Cleendtion
scrol plown on scroll juoz seral the page up or down.
Teb Management Antira:
rab ttah Indess switch broceee's fooue to specific uding in indec.
newr sais Cyeen new, empty
What you looking for today?
dose sab (lose currenlyy active
Navigation
gets Nowigatee specifi
go, Navigate predinuely page go. back parformed).
go haddgae
page
Uninee
Complede plepeeinnn Action:
fansweri this action when you biel the task complete.
objection is fivd botthvaced arower, proside the boarint.
190.0
JAMRES
C&A
Typ:
Clur
Hosss
Scre
Tub
Tao Focus
Cose Tab
Gota
do Back
Stop
Plaae enter and subbiih an artion.
2008
Figure 5: The annotation UI for Vieual
(2) Codel/O: Codel/O (Li et al., 2025) novel approach that transforms code-based reasoning patterns into natural language formats to enhance the reasoning capa- bilities of Large Language Models. We randomly sample 150K examples from the Codel/O dataset.
(3) Web Knowledge Base: Text-based trajectories can be synthesized using methods like (Ou al., 2024; Xu al., 2024a), which leverage tutorial knowledge. However, generating multi-modal data at scale remains challenging. In this work, we explore whether text-based trajectory data can benefit vision-based GUI agents. Specifically, we utilize 100K trajectories from the Synatra dataset (Ou et al., 2024) and randomly sample 50K trajectories from the AgentTrek dataset (Xu et al., 2024a), both of which are web-domain datasets that may potentially enhance GUI-Agent performance web-based tasks.
(4) Olympiad Math: NuminaMath (LI et al., 2024) a comprehensive mathematical dataset comprising wide range of problems, including exercises from Chinese high school mathematics curricula as well as competition problems from US and international mathematics olympiads. We specifically select the most challenging subset problems categorized under Olympiads within NuminaMath, and randomly sample 150K examples from this subset to highlight the complexity inherent in advanced mathematical reasoning.
C
Details of GUI Trajectory Data
High-quality GUI trajectory data is critical for enabling GUI agents to advance digital automation. However, due to constraints in application scenarios and annotation costs, existing high-quality trajectory data remains limited. To better adapt mid- trained models to GUI-specific tasks, we collected the following trajectory data:
17

---

Published as a conference paper at COLM 2025
OS-Genesis: We leverage trajectory data from both mobile and web platforms provided by OS-Genesis (Sun et al., 2024b), which are automatically synthesized in reverse via interaction-driien approach. To further improve the inference quality of these trajectories, we enhance them with step-wise reasoning using gpt- 4o-mini (Hurst et al., 2024). Specifically, gpt-4o-mini generates five sets of Chain of Thought cata for each step and optimized them to ensure consistency with the corresponding action. If inconsistencies persist, the corresponding data will be discarded. Ultimately, approximately 4K high-quality single-step trajectory samples are collected for the web platform and 5K for the mobile platform.
(2) MM-Mind2Web: MM-Mind2Web (Zheng et al., 2024a) the multi-modal exten- sion of the Mind2Web dataset (Deng et al., 2023), designed for developing and evaluating generalist web agents capable of following natural language instruc- tions to complete complex tasks on arbitrary websites. This dataset aligns each HTML document with its corresponding webpage screenshot from the Mind2Web raw dump, enabling joint modeling of structural and visual web information. We also employed GPT-4o-mini to process the data, resulting in the CoT-enhanced MM-Mind2Web dataset with 21K annotated steps.
Vesual Arena: We randomly sample questions from different task template of the Wee (Koh et al., 2024a), and annotate 3,264 steps of data using our implemented annotation tools (Figure 5). To ensure the annotation is correct, each annotated trajectory will be replayed and is required to pass the task.
(4) Aguvis: Aguvis is synthetic dataset of high-qualite GUI agent trajectories that builds upon and enhances existing open-source datasets such as AitW (Rawles et al,, 2023), GUI Odyssey (Lu et al., 2024a) and GULAct (Chen et al., 2024). While these prior datasets typically include high-level goals, observations, and grounded actions, Aguvis enriches them by incorporating detailed intermediate reasoning and low-level action instructions. Leveraging a VLM, Aguvis generates step-by- step inner monologues comprising observation descriptions, agent thoughts, and fine-grained action instructions, enabling more advanced multi-modal reasoning and planning for GUI agents. We randomly sample 22K single-step reasoning data from Aguvis to support post-training.
D Training Details
Following sfate-of-the-art works (Xu et al., 2024b; Sun et al., 2024b; Gou et al., 2025), we use Qwen2-VL--B-Instruuc (Wang et al., 2024a) as our backbone model. During the mid- training experiments, the middle training stage and post-training stage are integrated under single optimizer and learning rate schedule, which we find important empirically to stabilize the training process. For the scaling law experiments presented in Figure 3, we save checkpoints of the models trained on GUIMid as shown in Table 3 based on the amount of mid-training data trained. To evaluate the performance at intermediate points in the scaling law experiments (e-g., after training corresponding checkpoints and continue training with a cosine learning rate schedule that gradually reduces the learning rate zero, starting from the learning rate value recorded at the checkpoint. Due to hardware constraints, we limit the batch size per device to 1 with gradient accumulation of 4. For the results of scaling to 300K samples, we directly report the performance in Table 3.
50K samples of mid-training cata),
the
E Evaluation Details
AndroidWorld. AndroidWorld evaluates autonomous agents in Android environments through 116 tasks across 20 real-world applications. Each task incorporates randomized parameters to generate diverse scenarios, enabling rigorous evaluation. Success rates (SR) are determined through system state inspection without modifying the original application source code. Due to application availability constraints, our final evaluation encompasses 111 tasks. We build GUT agents based on the M3A agent from Android (Rawles et al.,
18

---

Published as a conference paper at COLM 2025
Parameter
Value
Context Length
8192
Number of GPUs
Learning Rate
2 x 10-5
Training Epochs Batch Size Per Device 2
1.0
Gradient Accumulation
Learning Rate Scheduler
cosine
Warmup Ratio Precision
0,05
bf16
Table 11: The training hyperparameters used in Table 3.
2024), drawing inspiration from the implementation of See agents (Gou et al., 2025), which rely solely visual information by removing the SoM images and textual lists of elements from accessibility trees in the observations and converting dlement-based actions into pixel-level actions. Furthermore, to ensure the fairness and rigor of the experiment, we maintain consistency between the training and evaluation prompts by refraining from using deliberately crafted prompts targeting corner cases. This suggests that variations in agent performance can predominantly be attributed to the mid-training data.
Aeeare We utilize the agentboard (Ma al., 2024) version of WebArena (Zhou al., 2023), which contains 245 cases sampled from the original benchmark, enhanced with progress labels that provide both success rates and progress rates tites--ffering more granular insights into model capability development. To ensure evaluation accuracy and consistency, we deploy the Docker-based websites on our local infrastructure rather than using the public AWS setup, which can lead to cross-evaluation inaccuracies. Map-related tasks were excluded due to website configuration issues, leaving 211 tasks. For maximum reliability, restart the environment after each evaluation to eliminate potential cross-contamination between test sessions.
gpt-4o Baselines In the absence of 7B-parameter baselines with comparable data volume, we evaluate the strong model gpt-4o-2024-11-20 as a planning model in pure vision-based GUI scenarios, while employing UGround-V1-2 (Gou al., 2025) as the grounding model. To ensure fair comparison, all models generate outputs using an identical prompt as shown in Figure 10,11. Notably, SeeAct-V (Gou al., 2025) demonstrates exceptional performance under these same experimental conditions. This superior performance can be attributed to its meticulously crafted prompt specifically designed for Worlo, which addresses common edge cases and incorporates rules aligned with human intuition.
All the experiments are conducted with temperate as 0.0, top_p as 1.0, max context length 8192.
F Prompt
We list our prompt below:
1. The prompt for generating Chain-of-Thought on OS-Genesis (Mobile) trajectories.
2. The prompt for generating Chain-of-Thought on OS-Genesis (Web) trajectories.
3. The prompt for generating Chain-of-Thought on VisualWebArena trajectories.
4. The prompt for generating Chain-of-Thought on Mind2Web trajectories.
5. The prompt for evaluation on the AndroidWorld.
6. The prompt for evaluation on the WebArena.
19

---

Published as a conference paper at COLM 2025
Prompt for Generating CoT 0S-Genesis (Mobile) Trajectories
are a mobile agent. Please think step by step perform series actions on an Android device to complete the task. At each stage, you will receive the current screenshot, record of previous actions, and a hint for the next correct step. Based on this information, decide the next action to take without mentioning the provided correct answer hint.
#f Available actions:
### UI Operations:
"click [element]': Click on an element.
type [element] [value]'; Type content into a field by ID.
"scroll [element] [value]': Scroll the page or specific element. The direction can be 'up'. down 'left', 'right'. Leave the element blank for scrolling the entire page. 'go_back'' Navigate back.
'go_home Navigate to the home screen.
"longppeess [element]"; Long press on an element.
"enter': Press the Enter key.
oppn_app [app_name]'s Open app by name.
"wait [value]'; Wait for the screen to update [value] seconds.
### Task finishing:
'stop [value]': Stop the task with goal status answer. If you think the task is completed infeasible, set the status "success" or 'infeasible". If the task requires an answer, provide the answer here.
### Instruction for the thought process:
1. Describe the situation in detail, focusing the goal and the related visual in current screenshot. Ensure your reasoning aligns with the goal, predicting the most suitable action based on the screenshot and previous actions.
2. Aim to reason through the task as if solving it, rather than simply reflecting on the labeled outcome.
3. Conclude the action with the format below.
Finally, end your thinking process with the action below. In summary, the next action is:
{
"Element Description": "Describe the element you want to interact with, including its identity, type (e.g., button, input field, dropdown, tab), and any visible text. Keep the description concise and under 30 words. If there are similar elements, provide details to distinguish them.",
"Action": "Select an action from the following options: (click, type, scroll, gocback, go_ go_home, long_press enter, open_app, stop}. Choase one; do not leave it blank.",
"Value": "Provide additional input based on the chosen action:
For 'type': specify the input text.
For 'scroll': specify the direction ("up", "down", "left", "right"). For 'open_app': provide the app name in the format: app_name="thhe name of the app". For 'stop': provide "completed", "infeasible", or the required answer For 'wait': provide the waiting time in seconds, in the format: seconds="5s". For other actions: leave this field empty."
### Input:
Previous Actions:
(previous_actions)
Task: (intent)
Correct Action Hint: (correct_answer)
Figure 6: Prompt for generating Chain-of-Thought on OS-Genesis (Mobile) trajectories.
20

---

Published as a conference paper at COLM 2025
Proupt for Generating CoT on O5-Genesis (Meb) Trajectories
Inagine that you are initating hunans performing web navigation for task, stop step. At each stage, you can see the webpage us humans do through screenshat, and you know the previous actions based on recorded history, current screenshot, and information about the current website. You need decide the next action to take.
mention the hint answer your thought process and just reason through the task as if solving it yourself, but make sure your answer is the same with the hint answer.
you
hint answer.
## Available actions:
Operations:
click [element]': click on an elenent.
field by ID. cleear [elenent] Clear the content of an elenent.
type [elenent] [value]', Type content
hover [element]] Hover over element by
presss [value]'s Press key combination (e.g., ctrlev).
*scroll [doun] or scroll [up]" : Scroll the page.
#*# Tab Nanagement:
"nee_tab"" Open
"tru_focus [tab_index]'s Switch to specific tab.
"close_tabb" Close current tab.
#а URL Navigation:
goto [url]' Navigate URL.
go go_back": Go to the previous page.
jo_ferward',
next page.
*** Task finishing:
[answer]'s Issue this action when believe the task complete.
Instruction thought process:
visual cues current screenshot. your reasoning aligns with the goal, predicting the nost suiteble action based the screenshot and previous actions.
situation detail, focusing
goal
Aim through the task Canclude the action with the fornat below.
solving it, rather than simply reflecting the labeled outcone.
answer your thought process. Instead, reason answer independently. ensure your answer matches the hint answer.
Finally, end your thinking process with the action below. In summary, the next action is:
"Elenent Description": "Describe the elenent you want to interact with, including its identity, type (e.g., button, input field, dropdown, tab), any visible text. Keep description concise under 30 words. If there are similar elements, provide details to distinguish then.".
"Action": "Select an action from the following options: (stop, click, type, scroll, back, go forward, goto, fage_focus, close_tab). Choose action; do not leave this field blank.", "value": "Provide additional input based on the chasen action:
clear,
'click"; specify elenent click.
For 'type" specify the input text.
For 'scroll': specify the direction ("up", "down").
For 'goto": specify the URL to navigate to.
For 'clear'; leave this field expty.
For 'hover': specify the element hover
For "press"; specify the key conbination to press.
focusf specify the
to.
For "stop': provide one of the following: "completed", "infeasible", or the required answer. For "wait': provide waiting time in seconds, fornat: seconds="Ss".
For all other actions: leave this field empty."
}
Inout:
Previous Actions: (prewious_actions)
Task: (intent)
Correct Action Hint: (correct_answer)
Figure 7: Prompt for generating Chain-of-Thought on OS-Genesis (Web) trajectories.
21

---

Published as a conference paper at COLM 2025
Proupt for Generating CoT on VisualMebkrena Trajectories
Inagine that you are initating hunans performing web navigation for task, stop step. At each stage, you can see the webpage us humans through screenshat, and you know the previous actions based on recorded history, current screenshot, and information about the current website. You need decide the next action to take.
mention the hint answer your thought process and just reason through the task as if solving it yourself, but make sure your answer is the same with the hint answer.
you
hint answer.
## Available actions:
Operations:
click [element]': click on an elenent.
field by ID. cleaa [elenent] Clear the content of an elenent.
type [elenent] [value]', Type content
hover [element] Hover over element by
press [value]'s Press key combination (e.g., ctrlev).
'scroll [doun] or 'scroll [up]": Scrall the page.
#*# Tab Management:
"nee_tab"" Open
"page_focus [tab_ iindexddxx Switch to specific tab. "close_tab" Close the current tab.
#а URL Navigation:
goto [url]' Navigate URL.
go_back", Go to the previous page. next page.
jo_ferward',
**# Task finishing:
[answer]'s Issue this action when believe the task complete.
Instruction for thought process:
1. Describe the situation in detail, focusing on the goal and the related visual cues in the current screenshot. goal, predicting
Ensure your reasoning alig actions.
previous
Aim reason through the task as solving it, rather than simply reflecting on the labeled outcone. Conclude the action with the fornat below.
Do not mention the hint answer your thought process, but make sure your answer is the same with the hint answer.
Finally, end your thinking process with the action below. summary,
"Elenent Description": "Describe the elenent you want to interact with, including its identity, type (e-g., button, Keep description concise under words. there similar elenents, provide details to distinguish then." "Action": "Select action from
input field, drapdown, tab), any visible
following options: (stop, click, type, scroll, go_back, forward, goto, clear, hover, press, new rew,tab, page_focus, close_tab). Choose one action; do not leave this field blank.", "Value": "Provide additional input based the chosen action:
For 'click'; specify the element click.
For 'type" specify the input text.
'scroll': specify
("up", "down").
For 'goto': specify the URL to navigate to.
For "clear': leave this field empty.
For 'hover": specify the element
hover over
For 'press": specify the key conbination press.
For "page_ focus_focus:: specify the tab index swi tch to.
For 'stop': provide one of the following: "complated", "infeasible", or the required answer. seconds="5s".
in seconds, For all other actions: leave this field erpty."
provide waiting
#at
{url)
Previous Actions: [prewtous_actions)
Task: (intent)
Hint_Action:
(hint_action)
Figure Prompt for generating Chain-of-Thought on VisualWebArena trajectories.
22

---

Published as a conference paper at COLM 2025
Prompt for generating CoT Mind2Web trajectories
You are a smart and helpful visual assistant that well-trained to manipulate websites. Your task is to navigate and take action on the current screen step-by-step to complete the user request.
will provide you with the hint thought process and just reason through task as solving yourself, but make sure your answer is the same with the hint answer.
Please do not mention the hint answer in your
## Instructions:
You will be provided with screenshots and website information.
Review your previous actions to determine the next steps. Go back previous status if necessary.
Pay close attention to all specific requirements of the task.
## Analysis Guidelines
#*т Previous Actions Analysis:
You should analyze the previous actions and the current status of the task.
### Screenshot Description:
You should describe all the screenshot in detail, especially the interactive elements, such as buttons, search bars, and dropdown lists. #*# Sub-task Planning:
Analyze the task status based on the observation and past actions and detail a reasonable future action plan accomplish the user request.
You should carefully check **ALL SPECIFIC REQUIREMENT make the plan.
You MUST check whether the last action is conducted successfully by analyzing the current screenshot.
### Critical Analysis and Reflection:
Check thet the history actions have accomplished the request.
Critique the past actions make a decision on the next action, and decide whether to backtrack to the previous steps with actions like: go back, goto ürl; scroll úp:
Assess the feasibility of the current sub-task and the overall task, and decide whether to modify the plan.
Finally, end your thinking process with the action below. In summary, the next action is:
(
"Element Description": "Describe the element you want to interact with, including its identity, type (e.g., button, input field, dropdown, tab), and any visible text. Keep the description concise and under 30 words. If there are similar elements, provide details to distinguish them.".
"Action": "Select action from the following options: (click, type, scroll, gack go_home, long_press, enter, open_app, stop}- Choose one; do not leave it blank.",
"Value": "Provide additional input based on the chosen action:
For 'type': specify the input text.
For 'scroll': specify the direction ("up", "down", "left", "right"). For 'open_app': provide the app name in the format: app_name="the name of the app".
For 'stop': provide "completed", "infeasible", or the required answer For 'wait': provide the waiting time in seconds, in the format: seconds="5s". For other actions: leave this field empty."
}
##ttt Input:
Task: (task)
Previous Actions:
(previous actions)
Hint Answer: (hint_answer)
Figure 9: Prompt for generating Chain-of-Thought on Mind2Web trajectories.
23

---

Published as a conference paper at COLM 2025
Evaluation Prompt for Mobile Tasks
<image>
You are a mobile agent. You need to perform a series of actions to complete task on Android, step by step. At each step, you are provided with the current screenshot and previous actions you have taken. You need to decide the next action to take.
## Available actions:
### UI Operations:
"click [element]': Click element.
"type [element] [value]': Type content into a field by ID. 'scroll [element] [value]': Scroll the page or a specific element. The direction can be 'up', "down', 'left', or 'right', Leave the element blank for scrolling the entire page. go_back': Navigate back.
"go_home" Navigate the home screen.
"lesg_press [element]': Long press on an element.
"enter": Press the Enter key.
open_app [app_name]': Open an by name
'wait [value]': Wait for the screen update for [value] seconds.
Task finishing:
'stop [value]': Stop the task with goal status or answer. If you think the task is completed or infeasible, set the status to "successful" or 'infeasible'. If the task requires answer, provide the answer here.
Please provide your detailed thought process and specify the action you intend to perform. The action should include description of the element to be operated on, the type of action, and the corresponding value, formatted follows:
{
"Element Description" "Describe the element you want to interact with.", "Action": "Select action from the available options, do not leave it blank.", "Value": "Provide a value only action requires it." )..
### Input:
Previous Actions: (previous_actions)
Task: (intent)
Figure 10: Evaluation prompt for mobile tasks.
24

---

Published as a conference paper at COLM 2025
Evaluation Prompt for Tasks
<image>
Imagine that you imitating humans performing web navigation for task, step by step. each stage, you can see the webpage as humans through screenshot, and you know the previous actions based on recorded history, the current screenshot, and meta information about the current website. You need to decide on the next action to take.
## Available actions:
#*# Web Operations:
"click [element]'s Click on an element.
"ype [element] [value]': Type content into field by ID. "clear [element] Clear the content of an element.
'hover [element]] Hover over an element by ID.
'press [value]': Press key combination (e.g., Ctrltv).
'scroll [down]" or 'scroll [up]": Scroll the page.
Management:
'new_tab': Open new tab.
page__occus [tta__indx]': Switch to specific tab. close_tab': Close the current tab.
### URL Navigation:
'goto [url]': Navigate to a URL. go_backk Go to the previous page.
'o_foowwrd': Go to the next page.
#*# Task finishing:
'stop [answer]': Issue this action when you believe the task is complete (the value could be successful, infeasible, the required answer).
Please provide your detailed thought process and specify the action you intend to perform. The action should include description of the element to be operated on, the type of action, and the corresponding value, formatted follows:
(
"Element Description": "Describe the element you want to interact with.",
"Action": "Select an action from the available options. Choose one; do not leave it blank." "
"Value": "Provide a value only action requires it." }
##tt Input:
Current URL: (url)
Previous Actions: porevious_actions:
Task: (intent)
Figure 11: Evaluation prompt for web tasks.
25


Reviews:
[{"review_id": "kssEbELhvG", "reviewer": "Reviewer_qdBH", "summary": "This paper proposes enhancing GUI agent generalization by mid-training Vision Language Models (VLMs) on diverse, reasoning-intensive tasks with abundant instruction-tuning data. It finds that multimodal and even text-only reasoning tasks significantly outperform GUI perception data, leading to strong cross-modal gains. The authors curate optimized training mixtures, achieving up to 8-12%  improvement on WebArena and AndroidWorld.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- good improvement in results (8%-12% on WebArena and AndroidWorld)\n- its an interesting study to observe that more textual and multimodal reasoning training data can help improve downstream GUI tasks.", "reasons_to_reject": "- not much novelty other than using some standard datasets for training GUI models. But still it is an interesting and previously unknown (afaik) observation that these kinds of training can have a big impact on the GUI performance. However this is the main observation, doing it on more benchmarks is imp.", "questions_to_authors": "It is interesting to see with just this kind of mid-training data, Qwn2-VL-7B model is performing significantly better. But since the main contribution of this paper is the study that such mid-training data helps in GUI tasks, I think it is important to have some more comprehensive experiments done\n\nOther than Aguvis, I suggest the authors also consider comparison with other GUI models like UI_TARS, OS-ATLAS, InfiGUIAgent and older work like CogAgent, ShowUI etc. Or if there is any compelling reason not to compare with these, authors can provide their argument.\nCompare on other benchmarks, Multimodal Mind2Web and Mind2Web Live,OSWorld, AndroidControl, GUIOdyssey, AITW, AITZ\nOther than that, some more questions:\n\n- From the scaling law figs of GUIMid Mid-Training data it seems that the performance on both benchmarks can further improve beyond 300K. Any reason why it was stopped at 300K? \n- For the paper, it would be nice to do a case-study or analysis of what kinds of tasks are least or most benefited by these kinds of mid-training reasoning data.\n- How much GUI trajectory data is mixed in the mid-training phase ? Is this added to each of the individual settings (i.e. each row) in the Table 3? Also, if that is the case, why isn’t the ‘MathInstruct (mixing)’ number of Table 5 not matching with the corresponding result with MathInstruct data in Table 3?\n- By using the mid-training data, can the authors show that the model becomes more sample-efficient on the post-training data i.e. with lesser training data it achieves similar final performance. Also, if that’s the case would mid-trained models require lesser number of expensive post-training samples with planning or inner-monologue rich data like the 22k from Aguvis, to achieve similar performance?", "extra_scores": {}}, {"review_id": "7WIyLRw4cz", "reviewer": "Reviewer_GCJU", "summary": "The paper investigate the challenge of data scarcity for training agents that automate tasks on GUI. It proposes training these VLMs on data-rich, reasoning-focused tasks as a mid-training step to improve their ability to generalize to GUI planning scenarios. The research found that multimodal mathematical reasoning and text-only mathematical data significantly boosted performance on benchmarks like AndroidWorld and WebArena, demonstrating effective cross-modal generalization, while GUI perception data had limited impact.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents comprehensive experimental reports on GUI, demonstrating the effectiveness of the proposed mid-training approach and the GUIMid dataset through performance improvements.", "reasons_to_reject": "1. Lack of technical depth: The paper largely functions as an experimental report on combining existing open-source datasets rather than introducing significant technical novelty.\n\n2. Limited generalization: The conclusions regarding the optimal data mixture are strictly confined to performance on WebArena and AndroidWorld; the lack of evaluation on diverse GUI environments (e.g., OS World, Windows Agent Arena, WebVoyager) leaves the generalizability of the findings, particularly the proposed data composition, uncertain.\n\n3. Misleading title: The title is somewhat misleading, implying a solution to the fundamental challenge of collecting scarce GUI data, when the core contribution appears to be an empirical report and evaluation of already available open-source datasets.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hWWpiXj3fV", "reviewer": "Reviewer_yoKR", "summary": "The paper tackles the challenge of limited high-quality trajectory data for training GUI agents. Specifically, the authors propose to introduce a mid-training stage using VLMs trained on a diverse set of reasoning-intensive tasks—such as GUI perception, multimodal reasoning, and text-based reasoning—to improve generalization to GUI planning. Experiments across 11 tasks reveal that task generalization is highly effective, with notable cross-modal transfer benefits (e.g., text-only math data improving visual GUI tasks), while GUI perception data alone yields limited benefits. By combining the most effective mid-training tasks with optimized mixture, the authors achieve significant performance gains of 8.0% on WebArena and 12.2% on AndroidWorld.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel empirical contribution: This work is the first to systematically show that reasoning-intensive data from other domains can significantly enhance GUI planning, offering a practical solution for a field where collecting high-quality training data is particularly expensive.\n- Insightful findings: The work provides valuable observations for the community, notably that text-only math data yields the largest gains for visual GUI tasks, while GUI perception data—despite its apparent relevance—offers limited benefits.\n- Strong empirical results: Extensive experiments validate the proposed mid-training strategy, showing that optimized mixture tasks lead to substantial improvements on challenging benchmarks, with gains of 8.0% on WebArena and 12.2% on AndroidWorld.", "reasons_to_reject": "* Limited novelty in methodology: While the findings are insightful, the core approach—leveraging mid-training on a mixture of auxiliary tasks—is conceptually incremental and builds on existing transfer learning paradigms without introducing fundamentally new techniques.  Moreover, mid-training has become a standard stage in modern LLM training.", "questions_to_authors": "- How were the 11 mid-training tasks selected? Were there specific criteria for inclusion or exclusion?\n- Why do the authors think text-only math reasoning tasks generalize so effectively to visual GUI planning? Is there a shared reasoning structure that enables this?", "extra_scores": {}}, {"review_id": "ziADvFD6yl", "reviewer": "Reviewer_L7Ty", "summary": "Graphical User Interface (GUI) agents have recently become a hot topic, but high-quality GUI trajectory data remains scarce. This paper investigates the mid-training process of vision-language models (VLMs) to enhance GUI performance after specific fine-tuning. The study focuses on purely visual GUIs, which are easier to generalize. The authors collect 11 diverse non-GUI datasets — 7 multimodal and 4 textual — to conduct mid-training using Qwen2-VL-7B-Instruct. During mid-training, downstream GUI trajectory data is also incorporated. Experimental results on WebArena and AndroidWorld demonstrate that non-GUI mid-training significantly boosts final GUI performance, particularly with mathematical reasoning data. By identifying the most effective non-GUI samples, the authors construct the GUIMid dataset for combined mid-training.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The scarcity of high-quality GUI trajectory data is indeed a significant challenge. The use of 11 diverse non-GUI datasets for mid-training is interesting, and the experimental results show clear improvements on GUI benchmarks.", "reasons_to_reject": "The mid-training process appears to serve as a general form of data augmentation rather than being specifically tailored to GUI tasks. Moreover, the experiments are conducted solely with Qwen2-VL-7B-Instruct, raising concerns about the generalizability of the results, as different VLMs may respond differently to mid-training due to variations in pre-training data and data ratios.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work investigates how introducing a dedicated mid-training stage for Vision Language Models (VLMs) on diverse, reasoning-intensive non-GUI tasks can improve performance on GUI agent benchmarks. Reviewers acknowledged the significance of addressing data scarcity in GUI agent training by leveraging abundant external datasets spanning multimodal reasoning, textual reasoning, and GUI perception. The work provides novel empirical insights showing that cross-modal generalization occurs effectively—for example, text-only mathematical reasoning data significantly boosts visual GUI task performance. Extensive experiments across multiple benchmarks, including WebArena, AndroidWorld, Mind2Web-Live, and WebVoyager, demonstrated substantial performance gains of 8–12% absolute, with notable improvements in complex reasoning domains. However, some reviewers noted limited technical novelty, viewing the approach primarily as a data-centric empirical study rather than a methodological innovation. The authors responded with clarifications on their contribution as a data-centric approach, justification of benchmark choices consistent with prior work, additional experiments on interactive real-world web environments, and analysis of task-level improvements. Generally, reviewers appreciated the thorough empirical validation and actionable insights for future GUI agent training data design. I think this paper is a valuable contribution with strong empirical results and novel observations on cross-domain transfer, meriting acceptance despite some methodological conservatism.
Decision: Accept