VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model
Authors: Mengzhuo Chen
Jiani zheng
Lu Wang
Fangkai Yang
Chaoyun Zhang
Lingrui Mei
Wenjie Yin
Qingwei Lin
Dongmei Zhang
Saravan Rajmohan
Venue: Accepted by TMLR ()

Abstract: Training Vision-Language Models (VLMs) for Graphical User Interfaces (GUI) agents via Reinforcement Learning (RL) faces critical challenges: environment-based RL requires costly interactions, while environment-free methods struggle with distribution shift and reward generalization. We propose an environment-free RL framework that decouples action utility learning from policy optimization by leveraging a pretrained Value Environment Model (VEM), which requires no live environment interaction during policy optimization. VEM predicts value-aligned action utilities directly from offline data, distilling human-like priors about GUI interaction outcomes without requiring next-state prediction or environmental feedback. This avoids compounding errors and enhances resilience to UI changes by focusing on semantic reasoning (e.g., “Does this action advance the user’s goal?”). The framework operates in two stages: (1) pretraining VEM to learn action-level utility signals and (2) guiding policy exploration with frozen VEM signals, enabling layout-agnostic GUI automation. Evaluated across diverse benchmarks including Android-in-the-Wild for mobile apps and Multimodal-Mind2Web for web environments, VEM achieves state-of-the-art or highly competitive performance in both offline and online settings. It significantly outperforms environment-free baselines and matches or exceeds environment-based approaches, crucially without incurring interaction costs. Importantly, VEM demonstrates that robust, generalizable GUI agents can be trained efficiently using semantic-aware action utility prediction, proving effective across distinct interaction platforms like mobile and web. The code is available at https://github.com/microsoft/GUI-Agent-RL.

Published in Transactions on Machine Learning Research (02/2026)
VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model
Mengzhuo Chen!-i, Jiani Zheng?*i Lu W Fangkai Yang" Chaoyun Zhang', Lingrui Mei't, Wenjie Yin", Qingwei Lin', Dongmei Zhang", Saravan Rajmohan
UUnivesiity of Chinese Academy of Sciences, "Peking University, "Microsoft, 'KTH Royal Institute of Technology
cachbaaD00..gallliianotttarorrro [wlu, Eangkaiyyagg)mmicrosstf com
Reviewed on OpenReview:
Abstract
Training Vision-Language Models (VLMs) for Graphical User Interfaces (GUI) agents via Reinforcement Learning (RL) faces critical challenges: environment-based RL requires costly interactions, while environment-free methods struggle with distribution shift and reward generalization. We propose an environment-free RI. framework that decouples action utility learning from policy optimization by leveraging a pretrained Value Environment Model (VEM), which requires no live environment interaction during policy optimization. VEM predicts value-aligned action utilities directly from offline data, distilling human-like priors about GUI interaction outcomes without requiring next-state prediction or environmental leedback. This avoids compounding errors and enhances resilience to UI changes by focusing on semantic reasoning (e.g., "Does this action advance the user's goal?"). The framework operates in two stages: (1) pretraining VEM learn action-level utility signals and (2) guiding policy exploration with frozen VEM signals, enabling layout-agnostic GUI automa- tion. Evaluated across diverse benchmarks including android-in--he-Will for mobile apps and Multimodal-MMid2WWen for web environments, VEM achieves state-of-the-art highly competitive performance in both offline and online settings. It significantly outperforms environment--ree baselines and matches or exceeds environment-based approaches, crucially without incurring interaction costs. Importantly, VEM demonstrates that robust, general- izable GUI agents can trained efficiently using semanticcanwe aware action utility prediction, proving effective across distinct interaction platforms like mobile and web. The code is available at https:://ittuub.com// http://www..oommmmoooofofftoooe/ Aitiiiirroooo///AAAt----RRI RUT-Agent-RT.
1 Introduction
Vision-Languren Models (VLMs) have shown strong capabilities in common-sense reasoning, abstraction, and generalization, enabling applications across domains Zhou et al. (2022): Zhang et al. (2024d). including autonomous GUI agents Zhang et al. (2024a): Wang ct al. (202bb) ; Nguyen et al. (2024). These agents leverage VLMs to interprel visual and lextual inputs for aulomating GUI interactions, such as locating and clicking interface elements based on natural language commands. This allows automation of tasks ranging from web navigation complex software operations Yan al. (2023): Zhang Zhang (2023) ; Zhang et al.
2023
Rawles et al. 2024h: Bai et al. (2024): Hong et al. (2024). Despite the progress in general-purpose
* *Equal contribution.
Work intern Microsoft. 1 Coo-oorrepponding author.

---

Published in Transactions on Machine Learning Research (02/2026)
models like GPT-40 OpenAl
(2024) and Gemini 1.5 Pro Deep (2024), real-world GUI tasks remain
challenging due to the variability of interfaces. Minor layout changes (e.g., pop-ups or repositioned buttons)
can lead to misinterppetation Xie et al. (2024); Zhang et al. 2024b e
Bai et al. (2024): Zhang et al. (2024a)
This highlights the nced for specialized VLMs op-
200
0
timized for GUT lasks. Reinforcement Learning (RI) /Stton (2018) is an effective approach for align- ing models with target behaviors Zhai et al. (0024) ;
Google Play
10x
Sun et al. (2024). In encironmenn--boe bosed RL, models learn by interacting with environments and receiv-
Entree
Pry
ing feedback Toyama et al. (2021); Bai et al. (2024); Carla et al. 2023); Wang et. al. (2024c]; Lai et. al.
Youlaba Shorts
B8
(2024), but these methods suffer from high interac-
Cala
Sanec
Drane
YouTube
tion costs and sample inefficiency Xic ct al. (2021) ;
Googge
Nuu et al. (2022). Simulated environments Chae et al.
(2024): Gu et al.
(0244) can alleviate this but often
4.0%
100l
face fidelity issues and compounding prediction er-
rors Guan et al.
(2023): Zhang et al. (2024f): Ge et al.
Instal
(2024). Alternatively, environment- free RT. leverages
See Flay Siore
offline RL Snell et al. (2022): Hong et al.] (2023) Bai
iet al. (2024bg Wang et al. (2024a) or reward model
training Stiennon et al. (2020): Ouyang et al. (2022).
avoiding the need for online interaction. However,
offline RL is challenged by distribution shift Levine
(a) Task: What's the latest video from GameSpot Reviews?
(b) Task: Open the calendar and show me this week's events?
ct al. 2020 and limited exploration Prudencio et al. (2023). while reward model-based methods may fail to genoralize in dynamic GUI scenarios where in- terface changes render static reward signals inelfec- tive Stiennon et al. (022)).
Figure 1: Two GUT lasks with the action marked as a red dot (O). In (a), clicking the 'share' button is unlikely to revcal reviews. In (b), the action may open the calendar app to display weekly events.
As shown in Figure humans excel at estimating the long-term utility of actions (i.e., slate-action values Q(s. a)) without explicitly seeing the next state. VLMs, trained on vast corpora of human- GUI interactions, inherently encode similar priors about action outcomes Hao et al. (2023); Bai et al. (2023); Chen et al. (0023). To operationalize this capability and address aforementioned challenges, we propose an environment-frg free RL framework that decouples value estimation from policy optimization. Unlike prior approaches that rely on reward models or require direct. interaction with the environment, our method leverages a pretrained value environment model (VEM) to directly approximate state-action values from offline data. A key distinction here lies in feedback and value estimation: traditional reward models typically provide sparse, trajectory-level feedback (i.e., evaluating an entire sequence of actions rather than individual steps), whercas the VEM learns dense, step-wise value function one that estimates the long-term utility of each individual action in the sequence. This dense, step-specific value signal delivers more granular and actionable guidance, which is particularly critical for complex multi-step tasks where assigning credit to specific actions is otherwise difficult. By distilling human-like priors into a frozen VEM, our policy model bypasses the need for explicit reward engincering or error-prone next-state simulations. Additionally, the VEM's resilience to superficial UT changes stems from its focus on semantic reasoning: it evaluates an action based on its contribution to the overall task goal (e.g., 'Does this action advance the user's goal?"), rather than attempting to predict brittle, pixel-level next states.
Concretely, our framework operates in two stages:
1. Value Environment Model Pretraining: The VEM is trained offline to predict state-action values Q(s, a), capturing the long-term utility of actions in diverse GUI contexts. This avoids the compounding errors of next-state prediction by focusing on value estimation, which aligns better with VLMSs inherent. reasoning strengths.
2. Policy Exploration with Frozen VEM:
During policy training, the VEM provides value-guided signals to iteratively refine the policy's 's action selection. By directly exploring for high-value actions that are
2

---

Published in Transactions on Machine Learning Research (02/2026)
grounded in the VEM's understanding of GUI semantics, the policy learns to generalize across unseen layouts and functionalities without online interaction.
To evaluate our method, we conduct rigorous experiments across diverse GUI automation benchmarks, specifically Android-in-the-Winn Wild (AITW) Rawles et al. (2024) for mobile applications and Multimodal- Mind2Web (MM--Mind2Web) Deng et al. (2023) for web-based tasks, using dual offline/onlinn protocols. Across both platforms, our approach achieves strong performance. On the ATTW benchmark, with only 500 training trajectories (equivalent to one-third the scale of the DigiRL Bai et al. (2024 2025a) dataset), our approach achieves 34%/35% offline task success rates General/ Ceneral/Webshopppil Webshopping domains, outperforming environment-free counterparts by 5-26% and exceoding previous environment-basee methods by 8%. In AITW online deployment, we attain 49.15% 19% general task success, surpassing environment-free methods by 23-49% while remaining comparable to environment- based policies (38.98%) in procedural efficiency (7.59 vs. 7.25 average steps). On the MM-Mind2Web dataset, our method achieves success rates of 55.8%/51. 101.22%5505559 on the cross-task/crosss wrrrosswebbste/eeoo cecoccrooooooe test sets, outperforming other methods trained on the same MM- Mind2Web training set by 4-35%. Notably, we only use 7.7k training samples, yet our performance surpasses models trained on large-scale pretraining data by 2-39%. Crucially, our method eliminates catastrophic failures scen in generic models (c.g., GPT-40) and achieves substantial relative improvements over the strongest baselines across both mobile and web environments, all without environmental interaction costs. These results demonstrate that offline policy optimization guided by VEM can rival online-trained systems while significantly advancing environment-free paradigms for broader GUI automation tasks.
2
Related Works
2.1
Environment- Based Methods
Environment--ased RL methods train VLMs through direct interaction with GUI environments, where rewards explicitly provided by the environment. Several frameworks like AndroidEnv Toyama et al. (2021) and DistRL [Wan et al. (2024ch enable agents to learn through trial-and-crror interactions with real-world digital interfaces. Recent works such as DigiRL [ai et al. (2024) and AutoWebGLM Lai et al. (2024) demonstrate that environment-basee RL can effectively align VLMs with complex GUI navigation tasks through autonomous exploration. However, these mothods face significant limitations in sample efficiency due to the high cost of environment interactions Xie al. (2021); Niu et al. (2022), particularly in real-world applications where collecting online feedback is expensive and suffers from high latency. To address this, some approaches like WebRL Qi et al. (2024) and WorldGPT Ge et al. (2024) attempt to simulate GUI environments, but they struggle with state prediction accuracy and compounding errors in long interaction sequences Guan el al. (2023): Zhang el al. (2024f). The fundamental challenge lies in the dynamic nature of real-world GUTs, where interface elements and layouts frequently change, making environment-dependeent reward signals inherently unstable Zhang et al. 2024bbe
2.2 Enviromment- Methods
Traditional planning-based based methods, such as Agent Q Putta et al. (2024), ReST-MCTS Zhang et al. (002cc), and QLASS Lin et al. (2025), rely on Monte Carlo Tree Search or similar algorithms. These approaches require extensive environment interaction and on-policy data, limiting their practicality in real-world GUI scenarios due to privacy, latency, and cost issues. Environment-free methods address these limitations by using offline datasets or learned reward models. Offline RL techniques [Snell et al. 2022); Hong et al. (2023) fine-tune vision-languas models (VLMs) with pre-collected trajectories, seen in large-scale GUI agent training Wang et al. (2024a) Bai et al. (2024). Reward modeling Stiennon al. (2020); Ouyang al. 20222 predicts task success from static datascts, enabling behavior alignment without real-time feedback. General-purpose VLMs like GPT-4o OpenAl (2024) leverage pre-trained capabilities for zero-shot GUI underslanding Yan et al. (023): Zhang et al. (2023), offering out-of box solutions without RL training. However, environment free mothods face challenges. Offline RL suffers from distribution shift novel GUI configurations Levine et al. (2020), and reward models are sensitive interface changes |Prudencio et al. (2023). Advanced VLMs like
GPT-4o struggle with GUT complexity due to insufficient task-specific fine-tuning ie et. al. (024 Zhang
3

---

Published in Transactions on Machine Learning Research (02/2026)
et al. 2024bb. Recent hybrid approaches aim to bridge these gaps. SeeClick Cheng et al. (024 combines environment-free pretraining with targeted fine-tuning, while Digi-Q Bai et al. [2025a] learns a Q-value function for action generation without full environment simulation. Related work on process-level reward modcling shows that providing step-wise supervision aligned with goal progress can significantly improve long-horizon decision making SStlur et al. (2024). Our Value Environment Model (VEM) follows similar step-level supervision philosophy, while extending it to multimodal, perception-heavy GUT environments with frozen value model for fully offline policy optimization.
3
Method
This section presents our method for training a GUI agent with offline data, followed by an extended thcoretical analysis. As shown in Figure 2 we first describe how to learn VEM from GPT-4o labeled data, then show that the resulting policy can achieve near-oplimal performance under certain coverage and accuracy conditions. We further incorporate distribution-shit arguments to highlight the relationship between dataset quality and final policy performance.
3.1 Preliminary
We formalize GUI navigation Markov Decision Process (MDP) = P,r,., which serves convenient abstraction for sequential decision making in GUT enviromments, without implying access to environment dynamics or return-based optimization during training. A state s s & represents the agent's 's current interaction context and consists of the task description, the interaction history, and the current GUI screenshot. An action a E AA corresponds to a concrete GUI operation (e.g., click, scroll, type), with the action space being compatible with each benchmark specification (Appendix B
The transition function P(s' 8, a) describes how the GUI state would evolve after executing an action, but is unknown and inaccessible in our environment-free setting, and is not modeled explicitly. The reward function is not observed from the environment. Instead, we assume access to task-aligned, action-level supervision derived from offline annotations, which indicates whether an action is beneficial or detrimental to task completion. The discount factor 7 E [0, 1) is included for notational completeness but is not used to estimate discounted returns.
Under this formulation, we consider fully offline learning setting in which no further interaction with the environment available and all learning signals derived from static data.
Given offline dataset D = {ss, d:)) collected by unknown suboptimal behavior policies 3, our goal is to learn a policy (a s) that selects actions aligned with task progress as indicated by the offline supervision signal, rather than explicitly maximizing expected discounted return.
3.2 Value Environment Model Training
A core challenge in GUI automation is the scarcity of explicit reward signals that indicate whether a chosen action advances or hinders task completion. While the benchmark datasets include trajectory-level success labels, they lack granular, step-level supervision that directly reflects task progress. To address this, we leverage GPT-4o to generate dense, action-level supervision based on full trajectory context. Rather than estimating cumulative return or learning classical Q-function, our goal is to obtain coarse but task-aligned signal that indicates whether action is likely to advance the overall task.
Specifically, we assign each state-action pair binary label capturing whether the action is beneficial or detrimental to the target task. This annotated supervision guides the learning of a Value Environment Model (VEM), which predicts an action-level utility signal and enables fully offline policy learning without explicit environment interactions.
LLM-Guided Annotation. For each state-action pair (s, a) in our offline dataset D, we leverage GPT-4o with chain-of-thought reasoning Wei et. al. (2022) to generate binary labels ((s,a) {1,2), simulating
4

---

Published in Transactions on Machine Learning Research (02/2026)
human-like judgments of task progress. This design simplifies annotation, and the absolute label values do not affect training as they are normalized before optimization (see Appendix D
By providing the LLM with the full task specification and trajectory context, it assesses an action's semantic contribution to the final goal. Importantly, this assessment does not rely on predicting future states or explicit rewards; instead, it captures whether an action directionally aligned with task completion given the current. context. As a result, the learned VEM focuses on task-level semantics rather than surface-level GUI layout details.
The resulting annotations provide coarse but informative supervision signals, where é = 2 denotes actions expected to advance task completion and L = 1 denotes potentially counterproductive steps. These labels do not constitute estimates of discounted return, but serve as Lask-aligned utility signals that are sufficient. to guide stable, offline policy optimization.
Value annotation
State
Policy Exploration
Click
Action
Click
Click
ScrolIDown
Value
x
GUI Agent
PressHome
VEM Training
State
SFT
Policy Learning
Annotated
VEM
veluee data
Step
Environment
Step Policy Learning Exploration
Figure 2: VEM Architecture: (1) Offline dataset annotation using GPT-4o's task understanding, and VEM training via supervised regression. (2) Policy optimization through frozen VEM maximization, encouraging the policy model to explore high-value actions.
Supervised Value Learning. Using the annotated subset D {(s,, a, 4)), we fine-tune a Qwen2.5VV Bai al. 2025b) model to predict. label values through squared error minimization:
min E (..0,))~5 [(uo(s,)) e)a]
The learned model uo predicts an action-level utility signal distilled from GPTT-4o's annotations, reflecting whether action advances task completion. Rather than estimating cumulative return transition dynamics, it captures task-aligned action desirability using purely offline supervision, reducing cost and latency during subsequent training.
Stable Policy Guidance. After convergence, we freeze ug as a fixed VEM to provide consistent action evaluations. While the binary labels represent simplified supervision, they effectively encode task progression patterns that guide subsequent policy learning. This approach maintains stability by decoupling utility learning from policy optimization, while remaining fully offline-tainnale.
3.3 Policy Learning with the Frozen VEM
Having established a VEM that can evaluate actions in any given GUI state, we now derive policy that selects actions maximizing the predicted utility. By freezing uy as a static action utility predictor, policy learning becomes a stable optimization problem that leverages consistent task-aligned supervision without requiring environment interaction.
5

---

Published in Transactions on Machine Learning Research (02/2026)
3.4 Value Maximization with Frozen Utility Model
We formulate policy learning utility maximization problem guided by frozen action utility model ua(s, a), referred to as the Value Environment Model (VEM). This model is pre-trained offiine using GPT-4o-provided action labels and remains unchanged during policy optimization.
The policy [a 8) is optimized to maximize the expected utility estimated by the fixed VEM over the offline dataset D:
max E am
uo(s,a) a)
frosen
The frozen utility model serves as a surrogate supervision signal, enabling policy optimization without environment interaction or return estimation. The decoupling of utility learning and policy learning ensures training stability and computational efficiency.
Coverage Regularization via SFT Initialization. To reduce the risk of distribution shift and ensure that the policy remains within the support of the offline dataset, we initialize ve using supervised fine-tuning (SFT) on behavior trajectories. This anchors the policy close the behavior policy 3, implicitly encouraging good coverage of D without requiring explicit regularization during optimization.
Policy Optimization. model:
We apply standard policy gradient updates to optimize ah against the fixed utility
V VoJ(FT) = E,-D, amay (1)) V log Tz(a s) uo(s, a)],
where us remains fixed throughout. This update increases the likelihood of actions assigned higher task-aligned utility under the frozen VEM.
Distinction from Traditional RL. Unlike actor critic or Q-learning frameworks which jointly update value and policy networks our method explicitly separates these stages. The utility model is trained once offline using distilled supervision and then frozen during policy optimization. This separation offers the following advantages: (1) No environment interactions are required after utility learning,
(2) A fixed supervision signal avoids instability from bootstrapping or non-stationary targets. (3) Policy learning reduces to optimizing against static, pretrained utility signal.
Interpretation of the Frozen Value Model. The VEM uo encodes task-conditioned action utility rather than expected return, and does not depend on transition dynamics temporal credit assigument. It replaces traditional reward feedback with static, value-aligned supervision signal, transforming policy learning into a form of offline, utility-guided behavior refinement. Training over static data with a frozen utility model yields stable updates and avoids the variance associated with on-policy rollouts or dynamic targets. This is especially suitable for GUI domains, where environment interaction is expensive or unavailable.
3.5 Theoretical Analysis
We present an explanatory analysis of policy optimization under static, task-aligned utility signal learned from offline data. Rather than providing guarantees with respect to optimal return, this analysis aims to clarify why optimizing against frozen, imperfect but value-aligned critic can lead to policy improvement in practice, particularly in high-branching GUI domains.
Coverage and Utility Alignment Assumptions Let (s, a) denote the state-action visitation dis- tribution of policy We introduce two conditions that characterize the regime in which our method operates:
(1) Dataset Coverage.
Cov m) Bee.e)dd [1{(s,)) a) E supp(D)))

---

Published in Transactions on Machine Learning Research (02/2026)
We assume that the learned policy satisfies Cov(D, 2 1 - -,, meaning that most actions selected by the policy lie within the support of the offline dataset.
(2) Utility Alignment. We assume that the learned utility model uæ(s, provides a task-aligned signal on the dataset support, in the sense that higher values correspond to actions that are more likely to advance task completion as judged by a capable teacher model. Formally, uo need not estimate cumulative discounted return, nor satisfy Bellman consistency; it is only required to be directionally aligned with task success on D.
Policy Improvement with Frozen Utility Model We consider the policy obtained by maximizing the expected frozen utility over offline states:
arg max EAWD, (la)) [uo(s,a)]
Under the coverage and utility alignment assumptions above, optimizing against a frozen utility model encourages the policy to concentrate probability mass on actions that are semantically aligned with task progress, while avoiding unsupported or out-of-distribution actions. This setting does not admit guarantees with respect to optimal return, but it allows us to reason about stability and monotonic improvement relative to the provided utility signal.
This analysis differs fundamentally from classical Q-funetion-based performance bounds. We do not assume access to, or approximation of, an optimal state-action value function, nor do we model temporal credit assignment or discounted returns. Instead, the analysis characterizes optimization behavior under a static, noisy, but lask-aligned supervision signal.
Consequently, the theoretical discussion should be interpreted as explanatory rather than prescriptive: it clarifies why freezing the utility model and maintaining high dataset coverage can yield stable and effective policy optimization in oflline settings, but it does not imply Bellman optimality or bounded suboptimality with respect to an optimal policy.
Practical Considcrations In our implementation, the utility model ug trainod using GPT-4o-genoratee binary action labels reflecting whether an action advances task completion. Human validation on 50 random samples from each dataset shows 90% agreement rate (see Appendix [ supporting the claim that the learned signal is well aligned with task-level objectives, though not necessarily with long-horizon return.
To limit distribution shift, we initialize the policy via supervised fine-tuning (SFT) on behavior trajectories, anchoring optimization within the dataset support. Empirically, we observe Cov(D, z 81% where coverage is measured as the fraction of gelicy-generated actions that follow valid action scherna defined by the benchmark and whose (action type, target UI element / coordinates) can be found in the offline dataset. This contributes to stable optimization when using frozen utility model. When the policy is not initialized with SFT or drifts away from the dataset support, observe significantly degraded training stability and final performance, highlighting the importance of staying in-distribution for GUI agents with irreversible actions.
This analysis highlights three design choices that are critical in practice: (1) learning a utility signal that is semantically aligned with lask success; (2) maintaining high dataset coverage through behavior-cloned initialization; and (3) freezing the utility model during policy optimization to avoid non-stationary targets. Empirically, improving the quality of the utility model-c.g. via larger vision-language backbones or refined prompting consistently improves downstream task success. For example, scaling the utility model from 7B to 32B parameters yields more reliable action evaluation and stronger policies. Policies trained without SFT initialization exhibit degraded performance, underscoring the importance of remaining within high-coverage regions of D.
This data is calculated by Auto-GUl-Baa in the offline AITW benchmark
7

---

Published in Transactions on Machine Learning Research (02/2026)
4
Experiments
In this scction, we evaluate the effectivencss of our proposed Value Environment Model (VEM) framework. We conduct experiments on two GUI automation benchmarks: AITW[Rawe et al. 2024 representing mobile
application environments, and MM-Mind2Web Deng et al. 2023 representing web-based environments.
4.1 Data Collection
Training Data for Critic Model. To align Qwen2.5VL with the critic model's schemna and evaluation framework, we used GPT-4o for multimodal data annotation and assessed agent actions. As shown in Figure our evaluation system defines two action quality levels based on effectiveness.
Level (Suboptimal Actions). Manifest deviations from optimal task execution, specifically including: (1) erroneous text. inputs compromising workflow integrity, (2) interface interactions triggering ad adversarial outcomes such as advertisement redirections, and (3) premature declarations of task completion prior to objective fulfillment.
Level (Optimal Actions). Demonstrate maximally effective task-solving behaviors, characterized by three critical patterns: (1) verifiable task completion through interface state validation, (2) robust recovery strategies, and (3) context-aware selection of optimal entry points for subtask resolution.
We further report the monetary cost of this annotation process and contrast it with environment-basel training costs in Appendix F The complete evaluation prompts and annotation protocols are formally specified in Appendix J
Table 1: AITW dataset: training/ /testing tasks and interaction steps with action quality levels.
Table 2: MM-Mind2Web dataset composition: num- ber of tasks and interaction steps.
Category
Split
Tasks
Steps
L1
L,2
Split
Tasks
Steps
Train
436
3340
1187
2153
Train
1009
7775
General
Test
100
777
214
563
Cross- Cross-Task Test
252
2094
Cross- Website Test
177
1373
Train
560
6240
1939
4301
Webshopping
Test
912
5911
Test
91
772
273
499
Both benchmarks follow SeeClick Cheng et al.
(2024) standards for data selection and evaluation. Tables
and 2
detail ArTW and MM-Mind2Weeb task/ step distributions and action quality levels (Level 1/2).
4.2 Implementation Details
Critic model We used GPT-4o to annotale and score benchmark data with task descriptions, action sequences, evaluated actions, and annotated scrcenshots, enhancing labeling accuracy with visualized annota- tions. Suboptimal data was generated using GPT-4o to address data imbalances. Prompts are in Appendix J More details provided in Appendix D
Policy model The AITW dataset policy is built on the SFT Auto-GUI base model, while the MM- Mind2Web policy uses Qwen2.5VI-3BB its base. In both cases, the critic model parameters kept frozen during training. Our Q-value curve in training is very stable, detailed in Appendix D
4.3 VEM Performance
We evaluate the performance of our VEM models on both the General and WebShopping datasets, as shown in Table 3 Our trained VEM achieves accuracy scores 77% and 85% on the AITW and MM-Mind2Well benchmarks, demonstrating high performance reliability. This level of accuracy is sufficient to drive policy optimization via exploration. As shown in Figure 3 the model assigns high value to the correct action ('click cart') while penalizing suboptimal actions, guiding the policy's 's exploration.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Teat Nezureeeentc shooping on newegg 00m7.
petion_type::
2011
Step "atthnubpe" "CUCK" "akoupooe
fape": "eclonusppes "xped_)ee"" "hraeggammm
Explonation:
Current Screenshet:
"stton, typ "CUCK" "dick point: joantt:j... Urit Dutpot: Explanation:
(0O le pureeerreeee
<iaagoo
ALEAAA L-A-BRATE
Google
"actio, hype "SEROI, DOWK
Sillcc Duspul:3 Explanation:
www.eailggggggo you borkgggiin
CONTRNTE STOWIIN
STATUS_CCMPLLTE Ecola talln
"action_yype "PRESS HOME"
Citti O Pl..pat
Deplonation: not benelictel to task
Step O:Click
Step 1:Click
Step 2:Type
Current Screenshot
Action Spaces
Next Screenshot
Figure 3: VEM scoring different actions at single timestep.
Our (Success, Steps)
DigiRL Offline (Failed, Steps)
DigiRL Online (Success, 10 Steps)
Click
Complete
Click
Complete
Serell Gowwn
Click
Complete
Figure 4: A case study of task execution trajectory comparison with DigiRL.
4.4
Main Results
We ovaluate our method on two benchmarks: AITW (General and Webshopping tasks) and MM-MindWWch. For AITW, by comparing predicted actions with human annotations; (2) Online Evaluation deploys the agent in Android environments (aligned with DigiRL), using GPT-4o as judge, with 10-step limits and duplicate task removal. To ensure the reliability of this metric, we performed manual validation study which revealed a 97.5% agreement rate between the LLM judge and human expert evaluations, supporting its use as an accurate proxy for task success (see Appendix H for details). For MM-Mind2Web,, we conduct Offline Evaluation, calculating element accuracy, operation F1, and step success rate. Baseslines Details can be found in Appendix D
two evaluation schemes: (1) Offline Evaluation computes step/Lask success rates (SRs)
4.4.1 AITW Benchmark Results
Table Performance on the VEM.
As shown in Table 1 and Table 5 our approach achieves SOTA performance AITW bench- marks. Offline, our method with Auto-GUI policy model attains 30.0% Task SR in Gen- eral and Webshopping domains, outperforming baselines. Scaled to SeeClick (9.6B), it reaches 34.0% and 35.0%, showing value model guidance efficacy without extra data. Online, our method generalizes well under real-world noise, achieving 49.15% and 20.00% success General and Webshopping. Interaction efficiency maintained with step lengths comparable supervised methods (Auto-GUI. 7.83/7.86 vs. 7.92/9.34 SeeClick: 7.59/7.68 vs. 8.80/9.80), avoiding DigiRT-online's trade-off of shorter steps (7.25/7.37) for lower success.
Precision
Recall
F1
Acc
AITW (Goneral+Webahopphhnn
0.83
0.84
0.83
0.77
MM-Mind2Wsb
0.85
0.84
0.84
0.85

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Offline results on ATTW benchmark.
Table Online results on ArTW benchmark.
General
Webshopping
General
Webshoppling
Slep SR Task SR Step SR Task SR
Task SR Step Leu Taek SR Step Len
Closed-soureu GPT-40 OpenAl
Closed-source
2024
68.4
63.9
GPT-4a OpenAl 2024
0.00
9.00
0.00
Small-scale Open Models (5200M)
Small-scale Open Models ($200M)
Auto-GUI (200M)
T8.0
Auto-GUI (200M)
Dight-offline (200M)
83.2
23.0
84.2
24.0
Digiri-oftlne (200M)
38.98
7.61
14.29
8.26
Digirl-online (200M)
83.3
26.0
85.1
26.0
Digirk-online (200M)
38.98
7.25
11.43
7.37
DigiQ (200M)
DigiQ (200M)
OurSAuto-CU1 (200M)
84.7
30.0
85.6
30.0
Ours Auto-GUU (200M)
42.37
7.83
14.29
7.86
Large-scale Open Models (298)
Large-scale Open Models (2:9)
CogAgeut (9B)
73.7
16.0
72.2
9.0
CogAgent (9B)
35.96
7.23
14.29
7,80
Secclick (9.6B)
$3.3
26.0
79.1
18.0
Scccllck (9.6B)
25.42
8.80
11.43
9.80
Ourk (9.0H)
34.0
86.7
35.0
Ourägwellick (9.68)
49.15
7.59
20.00
7.68
4.4.2 Mind2Web Benchmark Results
As shown in Table 6 our 3B model attains the highest Step Success Rate (Step SR) of 55.8%, 51.2%, and 50.5% under Cross- Task, Cross-Wees Website, and Cross-Domain settings. Compared with other models trained only on the MM-Mind2Web dataset, our model achieves state-of-the-art (SOTA) performance across all metrics. Even when compared with models that have been extensively and diversely trained on larger datasets, our model shows only slight underperformance operation step success rate. This fully demonstrates the effectiveness of VEM, which can effectively simulate interaction with the environment
on two lest sels, while still surpassing them
limited training set, achieving excellent results even 3B policy model.
Table 6: Performance comparison on MM-Mind2Web across different settings. We report element accuracy AEle,Acc), operation F1 (Op.F1), and step success rate (Step SR).
Method
Cross-Tase Op.F1
Cross Website
Cross-Domaie Op.Fl
Ele.Acc
Step
Ele.Acc
Op.F1
Step SR
Ele.Ace
Step SR
Agent Framework
GPT-4o
SeeClick
Cheng et al.
2024
32.1
33.1
33.5
GPT-4o
GPT-4V SeeAet Zheng 6t al
2074B
46.4
73.4
40.2
38.0
67.8
32.4
42.4
69.3
36.8
GPT-4V
Wan al.
36.5
42.0
Agent
GPT-1o OpenAl
2024
5.7
77.2
4.3
5.7
79.0
3.9
5.5
86.1
1.5
GPT-4 (S0%) Open AT et
2024
29.6
20.3
20.1
13.9
27.0
23.7
ShowUU1-2 Lin
88.6
37.2
41.6
83.5
35.1
39.4
86.8
35.2
MindAct-X Jony,
2023
55.1
75.7
52.0
42.0
65.2
38.9
422.1
66.5
39.6
WebGUM-XX Furuta es nl.
2024
57.2
80.3
53.7
45.3
70.9
41.8
43.9
72.2
41.4
Falcon-UI-7B Shen
2021
31.7
25.8
25.2
Magma-8B Yang el 2025
57.2
76.9
45.4
54.8
79.7
43.4
56.7
80.6
47.3
MiniCPM-Ve V-C-MM-------- (-1L Gilma
2024
20.3
81.7
17.3
23.8
86.8
20.8
17.9
74.5
17.6
SeeClick-9.6i Cbeng el al. 2024
28.3
87.0
25.5
21.4
80.6
16.4
23.2
84.8
20.8
Ours-3B
55.8
58.9
51.2
58.2
50.5
These models, like Ours, were trained only on the training set of MM- W2M-M,ndWWeb, while other models were trained on more data. Text-only input.
These results come from Qin et al. (2025).
These results come from Gou et al. (2025)
4.4.3 Ablation Study
We performed extensive ablation studies (detailed in Appendix E which confirmed the benefits of a larger critic model and more training data. Crucially, the studies showed that. SFT initialization is vital for performance, our binary labeling scheme was more effective than a finer-grained 3-class approach. It also includes an efficiency comparison against environment-basee RL and analyses of different LLM annotators.
4.5 Case Study
Real-world deployment of GUI agents faces challenges from dynamic environments, often causing navigation errors like entering ads. Models trained via SFT struggle to recover, while value-guided approach enables
10

---

Published in Transactions on Machine Learning Research (02/2026)
robust adaptation. As shown in Figure 1 our method completes the task-'Show the shopping cart newegg.com via precise state valuation, avoiding common failures seen in DigiRT, baselines. This is enabled by our value model's continuous feedback interpretation. Further comparisons are in Appendix K
5 Conclusion
We presents an environment-free RL framework for GUT automation that decouples value estimation from policy optimization through Value Environment Model (VEM). Our approach replaces arror-prone next-state simulations with semantic reasoning over GUI elements, enabled by offline learning from human demonstra- tion data. The two-stage training paradigm achieves structured credit assignment without environmental interaction, while maintaining procedural efficiency comparable to environment- based methods. Experimental results on Android-in -eeiedi-iithe---te Wild and Settimoddl-Minnd22W demonstrate superior task success rates over existing environment-ffe free approaches and significant improvements in generalization capability compared to vision-language models. The framework establishes semantic-driven value estimation as an effective pathway for layout-agnostic GUI automation with sample efficiency. In the future, we plan to explore self-supervised approaches for training the value model, aiming to reduce labeling overhead and further improve scalability.
References
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. orXiv preprint 2rviv:2406.1180 11896, 2024.
Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, and Aviral Kumar. Digi-q: Learning q- vavalue functions for training device-control agents, 2025a. URL http://aaxxiv..orrgaa/////0/0/ /ttp//bb/2225215566
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vasion-language model with versatile abilities. ar preprint ar 2rv: 12966, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijic Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXic preprint XrXiv:2502. 13923, 2025b.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning, pp. 3676 3713. PMLR, 2023.
Hyungjo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yoo. Web agents with world models: Learning and leveraging environment dynamics in web navigation. arXiv preprint ar arXim:2410 19.222,, 2024.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengehuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arxi preprint arXiv:2310.09) 09/70, 2023.
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Guicourse: From general vision language models to versatile gui agents, 2024. URL https://arxiv: atg/abb//20..113333
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Secelick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10993,, 2024.
Google DeepMind. Gemini 1.5 pro. https://eoolllllgooooorcroooaggg 2024. Ac- cessed: 2025-01-14.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Alcksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetuned foundation models, 2024. URL
2105
Zhiqi Ge, Hongzhe Huang, Mingze Zhou, Juncheng Li, Guoming Wang, Siliang Tang, and Yueting Zhuang. Worldgpt: Empowering Ilm as multimodal world model. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7346-7355, 2024.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents, 2025. URL
https://arxiv. http://axxivv..g/bbbbb// a21// 24213.52433.333
Yu Gu, Boyuan Zheng, Boyu Gou, Kai Zhang, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, Huan Sun, and Yu Su. Is your IIm secretly a world model of the internet? model-based planning for web agents. arXiv preprint (xXf=:241.06659 2024.
Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model-basee task planning. Advances in Neural Information Processing Systems, 36:79081-79094, 2023.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 2023.
Joey Hong, Sergey Levine, and Anca Dragan. Zero-shot goal-directed dialogue via rl on imagined conversations. arXiv preprint xrXiv:3111.0588, 2023.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290, 2024.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the ninetcenth international conference machine learning, pp. 267-274, 2002.
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent. arXiv preprint arXiv:2404.03445. 2024.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016: 016643,,. 2020.
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One lision-language-action model for gui visual agent, 2024. URL https://arxiv. org/ abs/ ////22411.111 17465
Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, and Kai-Wei Chang. Qlass: Boosting language agent inference via q-guided stepwise search, 2025. URL https://aaras org/
Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, al. Gui agents: A survey. arXiv preprint ArXiv:2412.138 2024.
Haoyi Niu, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, Xianyuan Zhan, et al. When to trust your simulator: Dynamics-aware bybrid ofline-and-online reinforcement, learning. Advances in Neural Information Processing Systems, 35:36599-3665 36612, 2022.
OpenAL. Gpt-4o. http::///ppaaa.ccc////eee/llllap 2024. Accessed: 2025-01-14.
12

---

Published in Transactions on Machine Learning Research (02/2026)
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapirg Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishuman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Kontiso-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufci Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Hcaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantamu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitschcider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Fukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Memick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyconwoo Noh, Long Ouyang, Cullen O'Keefe Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambautista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Saymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalic Staudacher, Felipe Petroski Such, Natalic Summers, Πya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wicthoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 tochnical report, 2024. URL https://arrai www.ziivorrg//baa http:/ba//2a0 2303.08777
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feodback. Advances in neural information processing systems, 15:27730-277444 2022.
Marek Petrik and Bruno Scherrer. Biasing approximate dynamic programming with a lower discount factor. Advances in neurol information processing systems, 21, 2008
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafacl Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents, 2024. URL https
//arxiv.oo www.iv.or///aaa orgooog/aao. httg/abs/2408.077
Zehan Qi, Xiao Liu, lat Long long, Hanyu Lai, Xueqiao Sun, Xinyue Yang, Jiadai Sun, Yu Yang, Shuntian Yao, Tianjie Zhang, et al. Webrl: Training Ilm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.02337 2024.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL
https://arxiv org/ 2//////b/221.... 12326
Christopher Rawles, Alice Li, Damiel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36, 2024.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. Xie preprint orXiv:2410.08146 10.08146, 2024.
Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, and Xiangyang Ji. Falcon-ui: Understanding gui before following user instructions, 2024. URL htttts///aaxt:: //t:x//aaxvvvvv/ org/ /bs//412..... 09362
Charlie Snell, Tlya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offiie rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.1187/ 2022.
Nisan Stiemnon, Long Ouyang, Jelfrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Newral Information Processing Systems, 33:3008-3022 2020.
Chuanneng Sun, Songjun Huang, and Dario Pompili. Llmebased multi-agent reinforcement learning: Current and future directions. arXiv preprint arXiv:2,05111106 2024.
Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018.
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv preprint orXiv:2105.122. 13231, 2021.
Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang. Omniparser: A unified framework for text spotting, key information extraction and table recognition, 2024. URL https:///aaat httpssvworrg/bb//403...... 19128
Lu Wang, Fangkai Yang, Chaoyun Zhang, Junting Lu, Jiaxu Qian, Shilin He, Pu Zhao, Bo Qiao, Ray Huang, Si Qin, et al. Large action models: From inception to implementation. arXiv preprint 110002112.10047, 2024a.
Shuai Wang, Weiwen Liu, Jingxuan Chen, Weinan Gan, Xingshan Zeng, Shuai Yu, Xinlong Hao, Kun Shao, Yasheng Wang, and Ruiming Tang. Gui agents with foundation models: A comprehensive survey. arXie preprint Xrxim:24110 10022411.000 1.08990 2024b.
Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents. orXiv preprint arXiv:24 1480s, 2024c.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Aduances in neural in information processing systems, 35:24824 24837, 2022.
Tengyang Xic, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finctuning: Bridging sample- efficient ofline and online reinforcement learning. Adoances in neural informalion processing systems, 34: 27395-211 27407, 2021.
Tianbao Xic, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Stheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint orXiv:2404.0700 2024.
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07502 2023.
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, and Jianfeng Gao. Magma: A foundation model for multimodal ai agents, 2025. URL https: https://arxivy wtwgab////////////aa 250. 13130.
Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. fine-tuning large lision-language models as decision prcision-making agents via reinforcement learning. arXiv preprint 2rX 10292, 2024.
Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, et al. Large language modell-bainee gui agents: A survey. arXiv preprint arXiv:2411.18977 2024a.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction. or preprint or orXiv:2402.07000 07939999 2024b.
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint or 0rXtw:2812.13711 2023.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Lim telf-training via process reward guided tree search, 2024c. URL http:::://aaxiioor//ab///////
03816
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024d.
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-toon -tought for gui agents. arXiv preprint arXiu:2403.02718, 2024e.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfci Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, el al. Mme-realworld: Could your multimodal 11m challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.188 13257, 2024f.
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv preprint 2309. 11436, 2023.
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents, 2024. URL /tapp://arxivv.
2309. 11436
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, 2024a. URL /ttpps://pperrevier hetr/forumiidrppicc..001111
15

---

Published in Transactions on Machine Learning Research (02/2026)
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yonggiang Ma. Ilamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024b. Association for Computational Linguistics. URL http:://axxi..orgabbss.. 13372
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Lin. Learning to prompt for vision-language models. International Journal of Compuler Vision, 130(9):2337 2348, 2022.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix
A Impact Statement
This work presents an environment-free framework for training GUI agents via a pretrained Value Environment
Model (VEM), which offers several positive societal implications.
First, in the domain of GUI agents, the proposed method significantly lowers the barrier to developing and deploying intelligent agents across diverse user interfaces, including mobile and web environments. By eliminating the need for interactive environments during training, the framework enhances sample efficiency and robustness to layout variability, enabling broader adoption of automation technologies in areas such as accessibility support, software testing, and digital task assistance.
Second, from the perspective of reward modeling, VEM shifts the focus from enviromment-depeeddett rewards semantically grounded value estimation. This approach provides stable and generalizable supervision signals, reducing reliance brittle hand-crafted rewards or noisy environment feedback. It offers scalable and interpretable alternative for aligning agent behavior with human intent in scenarios where explicit reward signals are unavailable or costly to obtain.
Third, with regard to world modeling, the VEM introduces a valuo-centric abstraction that bypasses the need for explicit state transition modeling. By learning long-term action utility directly from offline data, the method avoids compounding prediction errors common in traditional model-based approaches. This weigt form of world modeling demonstrates strong generalization across tasks and environments, offering promising direction for leveraging pretrained multimodal models as implicit world models.
In summary, this work contributes to the development of scalable, robust, and semantically aligned agents, advancing the broader goal of building efficient and general-purpose intelligent systems for real-world graphical user interfaces.
B Actions
B.1 AITW Actions
The available actions include CLICK, TYPE, PRESS BACK, PRESS HOME, SCROLL DOWN, SCROLL UP, SCROLL LEFT, SCROLL RIGHT, PRESS ENTER, STATUS TASK COMPLETE, and STATUS TASK IMPOSSIBLE.
Please note that the action space here is different from the origin action space in AITW. We have split the DUAL POINT in AITW into two parts: click and scroll, specifically as follows: CLICK, SCROLL DOWN, SCROLL_ UP, SCROLL LEFT, SCROLL_RRIHLS
B.2
MM-Mind2Web Actions
The available actions include CLICK, TYPE, SELECT, this is consistant with the action space in MM- Mind2Web.
C Proof of Extended Performance Bound
Proof. We bound the suboptimality gap J(x') J(F) in two stages: (i) relating value-function approxination error to policy return difference, and (ii) accounting for distribution shift between ă and the behavior policy 3.
1. Relating Q-function Error to Return Difference. By the performanc-diiferencce lemma Kakade & Langford (2002), for any two policies æ and n',
1 E Bande Ba (s, V~ (s))
- = 1
17

---

Published in Transactions on Machine Learning Research (02/2026)
Taking n² and n' कt, and noting V (s) max; (s) get
J(x)) (
B Bond
max Q" -F amm 1
Under our approximale-Q assumption,
Qo(s,a) Qs) ((s,a) ED.
Since î maximizes EgwD (Qo(s, a)), for each s E D.
max (o( - E 20(s,a) (o s 0.
Hence for all D,
max (8, - E a (s)
S mmax Qo a) E [ + + S 28.
Thus, restricting the expectation in the performance lemma to E D,
2e
J(x²) - ((R) <
Under coverage condition, â does not visit unseen (s, a), so D covers the support of da,, and the above probability is (approximately) 1.
2. Accounting for Distribution Shift. In practice, मे may induce state distribution d differing from dx.. Standard distribution-shift bounds (see, e.g., Levine et al. (2200) yield
28 Rmax dx- dzlb 1-7
|J(=') - [(+) 1
where Rmax = maxs,a [r(s, a) and || is divergence measure. Since r(s, a) is bounded by Q" a) S Qmax: we absorb constants into c. Moreover, [|d,- dzلl can be bounded by | Blll under mixing conditions (cf. Petrik & Scherrer (2008)
Conclusion. Combining the two steps, there exists a constant e > 0 (depending on 7, Qmax and mixing properties) such that
J(=') s (e + | 31)
Finally, bocause Ω is trained purely offline with fixed Qs, its gradient estimates rely deterministic VEM queries rather than noisy environment samples, yielding lower variance compared to on-policy RL.
This completes the proof.
Implementation Details
Data Format The reinforcement learning paradigm requires standardized data transformations across input modalities. For our experiments on the AITW datasct, we trained the Auto-GUI-Base model, which is pre-trained to outpul normalized coordinates in the range of [0,1]. Therefore, we utilize a device-agnostic [0,1] screen-space coordinate system. For the MM-MMin2WWeb dataset, we trained and experimented with the Qwen2.5-VVL-33 model, which tends to produce absolute coordinates rather than normalized [0,1] coordinates, so used the original coordinate space achieve better performance. Detailed action space configurations are provided in Appendix
18

---

Published in Transactions on Machine Learning Research (02/2026)
Critic model We employ GPT-4o to annotate and score data within the benchmark training set. The imput provided to GPT-4o comprised the task description, the complete sequence of actions, the specific action that requires evaluation, and the corresponding annotated screenshot. The inclusion of the global action sequence, along with visualized annotations on the screenshot (indicating click coordinates), enhanced the accuracy of GPT-4o's aulomated labeling. Furthermore, acknowledging inherent data quality issues within the benchmark dataset, which includes a small proportion of suboptimal steps, we proactively generated additional suboptimal data using GPT-4o. This augmentation strategy aimed to mitigate potential biases in the Critic Model's training due to skewed data score distributions. The prompts utilized for data annotation and suboptimal sample generation by GPT-4o are detailed in Appendix J
To evaluate GPT-4o's annotation quality on the benchmarks, we randomly sampled 50 instances from each dataset and had them independently annotated by three human experts. Using majority voting as the human ground truth, we found GPT-4o annotations to match with 90% accuracy. GPT-4o annotations on both benchmarks achieve 90% human consistency with 3-hour processing efficiency. Evaluation dotails can be found in
We fine-tuned the Qwen2.5VL-7B model using the LLaMA-Pactory Zheng et al. (024bb framework to develop state classification capabilities. The imput part of our Critic Model includes: the textual task description, the history of actions, the current screen image, and the action currently pending execution. The output of our Critic Model is the score given by GPT-4o. The formal specification of input composition and prompting strategy appears in Appendix []]
Training leveraged distributed data parallelism an 4-GPU NVIDIA A100 cluster, configured with 3 training epochs and global batch size of 16. The optimization process employed AdamW with an initial learning rale of 1 x 10-5, achieving convergence within 12 hours while maintaining computational efficiency through gradient accumulation strategies.
Policy model In our experiments, we found that training with reinforcement learning without prior supervised fine-tuning leads to poor performance. We attribute this to two main reasons: (1) prompts alone are insufficient to ensure correct output formatting, and (2) the model's initial capability is weak, resulting in extremely sparse positive samples during training, which makes effective learning difficult. In training the critic model, we enhance the quality of annolated data by providing GPT-4o with the global action sequence and the annotated current screenshot. Additionally, we construct suboptimal negative samples using GPT-4o to mitigate the issue of label bias. These strategics collectively contribute to improving the quality of Qo
For the AITW dataset, the policy architecture builds upon the Supervised Fine-Tuned (SFT) Auto-GUT base model. For the MM-Mind2Web dataset, used Qwen2.5VL-33 as the base model. During the training of our policy model, we kept the paramcters of the critic model frozen.
For the MM-Mind2Web dataset, we first used Swen2.55VL-331 as the base model and performed supervised fine-tuning using the LLaMA-Factory Zheng et al. (202bb) framework with a learning rate of 1e-5 for one epoch, enabling the model to learn the output format. This was followed by reinforcement learning training.
Our implementation executed full fall-peremeter optimization on an 4-GPU NVIDIA A100 cluster, configuring the training process with batch size of 16 samples and 1 10-5 across 10-20 training epochs. This configuration achieved stable convergence through progressive reward signal aligmment, demoustrating paraaneder-effiiient adaptation characteristics.
Figure 5 shows the Q-value curves during training of the policy model based on Auto- for the AITW dataset and the policy model based on Qwen2.5VL-3B for the MM-Mind2Web dataset. As observed, the Q-values initially increase and then begin to converge. From the figure, it's evident that our training is very stable.
Please note that the Q-value discussed herein has undergone a normalization procedure. Specifically, the Critic Model yields Q-values that are either 1 or 2. Subscquently, scaling transformation is applied according to the formula:
Q
Q @scated =
-0.75
2
19

---

Published in Transactions on Machine Learning Research (02/2026)
AITW: Auto-GUI
MM-Mind2Webb Qwen2.5VL-38
orginal smoothed
0.18
0.16
0.20
0.14
0.15
Urain valuee
0.12
0.34
0.10
0.08
0.10
0.00
0.08
0.04
150
200
250
300
Figure 5: Q-value progression during policy model training.
This transformation effectively maps the original Q-values into a bounded range of 0.0.2,...25)
Notably, the gradient update strategy incorporated differential learning rale scheduling bet ween the frozen Critic components and tunable policy layers, effectively balancing knowledge retention with operational flexibility. The training of the Auto-GUI model required under 12 hours, while the training of the Qwen2.5VL- 3B model required under 36 hours, reflecting optimized memory utilization patterns.
Datasets The Android in the Wild (AITW) dataset is a large-scale benchmark designed for research in Android device control via natural language instructions. comprises 715k episodes across 30k unique instructions, collected from over 350 Android applications and websites. The dataset includes interactions recorded cight device types (from Pixel 2 XL to Pixel 6) running Android versions 10 through 13, encompassing various screen resolutions.
Following the setup adopted in SeeClick Cheng et al. (024]) we selected a subset of data from the General and WebShopping categories of the AITW dataset to serve as our training and testing sets. This selection was made to align with the instruction wise wise split strategy proposed by SeeClick, which aims to mitigate overfitting and better assess generalization capabilities across diverse tasks and applications.
The MM-Mind2Web benchmark comprises over 2k open-ended lasks collected from 137 real-world websites across 31 domains. MM-Mind2Web features training set and three test sets: (1)Cross Task, which includes tasks from websites scen during training; (2)Cross Website, which contains tasks from unseen websites; (3) Cross Domain, which comprises lasks from entirely unseen domains designed to evaluate different aspects of model generalization.
Baselines For AITW benchmark, to comprehensively evaluate our proposed method, we compare it against several baselines on the Android-in-the-Wil -iil (AITW) benchmark. These baselines encompass a diverse range of approaches in multimodal reasoning, visual-language modeling, and reinforcement learning for GU1-based agents.
GPT-4o AT (2024) is OpenAT's flagship multimodal model that integrates text, vision, and audio modalities. Despite its general-purpose capabilities, GPT-4o exhibits limited performance on GUI navigation tasks, highlighting the challenges of applying generalist modols to specialized domains.
Auto-GUI Zhang & Zhang (2024 introduces multimodal chain-of-action framework that leverages visual context and historical action sequences to predict subsequent actions. Trained the AITW datasct, Auto-GUI achieves competitive performance in both offline and online scttings.
CogAgent Hong al. (2024 a visual-language model designed for GUI understanding and navigation. By employing high-resolution image encoders, CogAgent effectively captures ano-grained UT elements, leading to improved performance benchmarks like AITW and Mind2Web.
20

---

Published in Transactions on Machine Learning Research (02/2026)
SeeClick Cheng al. (2024) focuses on GUI grounding by pretraining the ScreenSpot dataset, which includes diverse screenshots and instructions. As purely vision-based agent, SeeClick demonstrates strong performance on GUI tasks with minimal training data, emphasizing the efficacy of GUI grounding pretraining.
DigiRL Bai et al. 2024) employs an offline reinforcement learning approach to train device-control agents. By utilizing srrantage-weightee RL and an automatic curriculum, DigiRL significantly improves success rates on the AITW dataset compared supervised fino-tuning methods.
DigiQ Bai (2025a) builds upon DigiRL by introducing Q-value function learning for visual- language models. This approach enhances data efficiency and convergence performance, enabling better policy extraction and improved success rates GUL navigation tasks.
For MM-Mind2Web benchmark, evaluate approach against diverse set of baselines, encompassing both agent frameworks and agent models. These baselines are selected to represent the current state-of-the-ar. in multimodal web agents, covering range of modalities and grounding strategies.
Agent Frameworks
GPT-40 + SeeClick Cheng et al. (2024): This framework utilizes GPT-4o for planning and SeeClick for visual grounding. SeeClick is a vision based grounding method that identifies clickable elements on the UI using object delection lechniques.
GPT-4o + UGround Gou et al. (2025): UGround is a universal grounding framework that combines textual and visual cues map instructions to UI elements, When paired with GPT-4o, it enhances the agent's ability to interpret and act upon complex web interfaces.
GPT-4V + SceAct Zheng al. (2024a): SeeAct integrates GPT-4V with grounding strategy that leverages both HTML structure and visual information. It demonstrates strong performance in exccuting tasks on live websites by effectively grounding textual plans into actionable steps.
GPT-4V + OmniParser Wan et al. 2024): OmniParscr is vision-based UI parser that converts screenshots into structured representations. Combined with GPT-4V, it enables the agent to understand and interact with web interfaces using visual inputs alone.
Agent Models
GPT-40 Ap 2024):
GPT-4o is a cutting-edge multimodal large language model engineered to process and generate responses across text, vision, and audio modalities. Its robust architecture enables to serve as a powerful baseline for evaluating sophisticated multimodal understanding and the generation of contextually relevant actions in diverse scenarios.
GPT-4 (SOM) OpenAl al. (2024): This approach leverages the capabilities of the GPT-4 large language model, enhanced by the Sct-of-Marks (SOM) prompting technique. SOM aims lo significantly improve the model's grounding abilities, particularly in lasks that require precise identification and interaction with specific user interface (UI) elements within visual inputs.
Mind Deng et al. (2023): A 3B-parameter model for web automation, excelling at generating executable actions (clicks, typing, navigation) from webpage visuals and natural language. Trained on MM-Mind2Web, it shows strong performance on the Mind2Web benchmark.
WebGUM-XL-33 Furuta et al. [004 3B-parameter multimodal agent processing webpage screenshots and HTML for web navigation (clicking, typing). Jointly fino-tuned on instruction- finetuned LM and vision encoder, achieving SOTA on MiniWoB and WebShop, with strong general- ization to Mind2Web.
21

---

Published in Transactions on Machine Learning Research (02/2026)
SceClick-9.6E Cheng al. (2024): A visual GUI agent automating tasks using only screenshots, focusing on GUT grounding. It employs GUT grounding pre-training and automated data curation, introducing ScreenSpot, a GUI grounding benchmark. Results show a correlation between GUI grounding and downstream task performance.
Show Lin et al. (2024): :: 2B-parameter vision-languagg-acliinn model for GUT automalion. Key features include Ul-guided visual token selection and interleaved vision-languagg-actiin streaming. Trained on a 256K muslruction-following dataset.
Falcon-UI-78 Shen et al. 2024 :: GUT agent model enhancing GUI context understanding. It
introduces Insight-UI UT Dataset for instruction-free pre-training of GUT comprehension, followed by fine-tuning on instruction-basee datasets.
Magma-8B Yang et al. (2025) A foundation model for multimodal AI agentic tasks, integrating verbal and spatial- intelligence. Pre-trained on diverse datasets with Set-of fet-of-Mark Mark (SoM) for action grounding and Traco-of-Mark (ToM) for action planning, excelling in UI navigation and robotic manipulation.
MiniCPM-V-GUU-SB Chen al. 2024h: A visual-based GUI agent model trained on the GUTCourse dataset suite. Based on MiniCPM-V. it enhances VLM's interaction with GUTs, improving performance in OCR, grounding, and understanding GUI components and interactions.
E
Ablation Study
To comprehensively evaluate our VEM framework and validate its key design choices, we conducted an extensive series of ablation studies and analyses. These experiments investigate the impact data scale, model size, the supervision signal itself, and the overall algorithmic contribution and efficiency of our approach.
E.1 Impact of Core Components and Training Strategy
We first analyze the impact of fundamental components: the scale of training data, the size of the critic and policy models, and the necessity of Supervised Fine-Tuning (SFT) initialization.
Data and Model Scaling. As shown in Table 7 performance consistently improves with more VEM training data and larger critic/policy models. For instance, increasing the critic model from 7B to 32B parameters boosts the General Task SR. Notably, even with only 30% of the training data, our method remains effective. This demonstrates the framework's robustness and scalability.
Importance of SFT Initialization. To quantify the role of SFT, we trained a policy using only VEM-guided RL without the SFT warm-start. Table S shows a substantial drop in performance on the MM-Mind2Weel benchmark. This confirms that SFT is a crucial step for guiding the policy into a reasonable region of the state-action space and mitigating distribution shift, a common practice the field.
Table 7: Ablation study data and model scaling (AITW oflline benchmark). The asterisk (*) marks the baseline configuration for each study.
Ablation Var
Configuration
General
Webshopping
Step SR (%)
Task SR (%)
Step SR (%)
Task SR (%)
Full Dataset (100%)*
84.7
30.0
85.6
30.0
VEM Training Data
Reduced (50%)
84.1
28.0
85.2
28.0
Minimal (30%)
83.4
26.0
84.0
27.0
Owen2.5VL-7BB- Owen2.5VL--33
84.7
30.0
85.6
30.0
Critic Model Size
85.7
32.0
86.2
32.0
Policy Model Size
Ours (200M)*
84.7
30.0
85.6
30.0
Ours (9.6B)
86.3
34.0
86.7
35.0
22

---

Published in Transactions on Machine Learning Research (02/2026)
Table 8: Ablation on SFT initialization WMee-Mind2Web benchmark). Performance degrades significantly
without the SFT warm-start.
Method
Cross- Cross-Taske (%)
Cross- Website(%)
Cross-Domand
SFT + VEM RL (Ours)
55.8
51.2
50.5
VEM RL only (no SFT)
37.5
41.7
39.1
E.2
Analysis of the VEM Supervision Signal
The quality of the VEM is contingent the supervision signal used to train it. We analyzed several factors, including the granularity of the value labels and the methodology for generating them.
Value Label Granularity. We experimented with finer-grained 3-class labeling scheme (Low, Medium, High). As shown in Table 9 the simpler binary labeling approach outperformed the 3-class scheme on the AITW benchmark. This suggests that the binary distinction provides a more stable and effective supervision signal, likely bocause most actions the datasct are cither clearly beneficial or not, making the intermediate calegory sparse and difficult to learn from. We believe this is because the intermediate class introduces semantic overlap with both positive and negative actions, which blurs the value boundary and increases annotation noise in LLM-based labeling. Since the Value Environment Model (VEM) is frozen during policy optimization, such ambiguity in value supervision directly propagates to policy gradients, whereas binary labels yield sharper and more robust signal under distribution shift.
Annotator and Prompt Design. The quality of LLM-based annotation is critical. We evaluated the impact of the LLM annotator and the prompt design on annotation accuracy for MM-Mind2Web. using the dataset's ground truth reference. As detailed in Table 10 using providing the full trajectory context in the prompt both lead to higher-quality supervision. This validates our chosen methodology for generating reliable value labels.
powerful L.M (GPT-40) and
Table 9: Comparison of binary vs. 3-class value labels on the AITW offline benchmark.
Label Type
Step SR (%)
Task SR (%)
Binary (Ours)
83.6
29.0
Class
82.9
25.0
Table 10: Analysis of annotation accuracy on MM-Mind2WWbb based on LLM annotator and prompt design.
Configuration
Annotation Accuracy (%)
Annotator Choice
GPT--o (Ours) Qwen2.5VIL-33
96.39
93.45 Step-only Context
Prompt Design
Full Trajectory (Ours)
96.39
62.57
E.3 Algorithmic Contribution and Efficiency
Finally, we conducted experiments to isolate the contribution of our algorithm and quantify its efficiency gains.
Isolating Algorithmic Contribution. To verify that our performance gains stem from the VEM framework itself, not just the base model, compared method against Digi-Q using identical LLaVA-1.5-7B
23

---

Published in Transactions on Machine Learning Research (02/2026)
backbone. As shown in Table I1 even level playing field, our method significantly outperforms Digi-Q, confirming the algorithmic advantage of our value-estimation approach.
Efficiency Analysis. To quantify the practical benefits of our environment-free approach, we compared its computational and sample costs against a representative environment- based method, DigiRL (Table 12 Our VEM framework achieves superior or comparable performance while being significantly more efficient, entirely avoiding the time, cost, and complexity of live environment rollouts.
Table 11: Comparison with Digi-Q Q on the AITW online benchmark using an identical LLaVA-155-B backbone.
Method
General
Webshopping
Task SR (%)
Step Len
Task SR (%)
Step Len
Digi-Q (LLaVA-7B)
33.90
6.76
5.71
8.43
Ours (LLaVA-7B)
40.68
7.91
14.29
8.12
Table 12: Efficiency comparison between oflline VEM framework and environment-based RL method.
Metric
VEM (Ours)
DigiRL (Env-based)
Training Time
12 hours (offline)
3-5 days (with env rollouts)
API Calls for Supervision
-15k (for annotation)
N/A (uses env rewards)
Total Samples Used
-58 (ofline)
> 150k (online + offline)
F Cost Analysis of LLM Annotation
Our euvironment-fiee framework relies on a one-time LLM-based annotation process to obtain step-level value supervision. Under experimental sctup, annotating the training data requires approximately 15,000 API calls to GPT-4o.
Each annotation call includes a long textual prompt describing the task and trajectory, together with multiple GUI scrcenshots. Using the GPT-4o pricing at the time of experimentation ($2.5 per 1M input tokens and $10 per 1M output tokens), and conservatively assuming around 12k input tokens and 200 output tokens per call, the total annotation cost can be estimated as:
12,,000 106
200
15,000
2.5
x 10) $500.
106
Iî GPT-4o were queried directly reward model during policy optimization, a reward evaluation would be required for every sampled (s, a) pair. Under our training configuration (batch size 16, 15 epochs, and 9,580 training steps for AITW), this would result in
Nealls 3,340 + 6,240
X 15 x 16 ~0 143,760
16
GPT-4o calls.
Using the same token statistics (12k input, 200 output tokens), each call costs approximately $0.032, leading total cost of
Cairect 143,760 0.032 v $4,600,
which is over 9x higher than our one-time distillation cost. The environment interaction cost can be estimated as:
Ceny Nom Thour P Poms
24

---

Published in Transactions on Machine Learning Research (02/2026)
where Nvm denotes the number of cloud instances, Thours is the total training time in hours, and Pvm is the hourly price per instance.
Concretely, following the DigiRL setup with up to 64 parallel Android emulators running on clusters with over 100 CPU cores, this corresponds to approximately Nym = 8 instances (each with 16 CPU cores and one T4 GPU), a training duration of T hourss = 72-120 hours (3-5 days), and an hourly price of Pvm $1.2 $1.3. Thus,
Cenv 8 (72 120) 1.25 $720 $1,200,
which is on the order of $960 per training run, excluding additional engineering, debugging, and system maintenance overhead.
This comparison highlights key trade-off between environment-basod and environment-free approaches: while environment-free methods incur an upfront annotation cost, they avoid repeated environment interaction expenses and offer improved scalability and stability.
G Limitations
Our approach requires training a Value Environment Model, which in turn necessitates additional high-quality data annotation. In this work, we leverage GPT-4o to perform the annotation task. By providing GPT-4o with detailed task description, the complete action history, the current action, and the annotated screenshot corresponding to the current action, we are able to generate high-quality annotations that achieve up to 90% agreement with human experts.
Moreover, due to the limited number of negative samples in the dataset, there risk of label bias affecting the training of the Critic Model. To mitigate this issue, also utilize GPT-4o to construct high-quality negative samples. These designs significantly improve the annotation accuracy of the Critic Model, albeit at the cost of requiring additional annotation resources. Exploring how to achieve comparable performance with lewer additional resources, or even in an unsupervised manner, remains an important direction for fulure work.
Validation of LLM-based Online Evaluation
While oracle-based evaluation functions represent the gold standard for benchmarks, online evaluation for AITW relies on an LLM-based judge (GPT-4o). This approach was chosen specifically for fair and direct comparability with recent stale-of-the-arl methods such as DigiRL and DigiQ, which are evaluated in the same manner.
To directly address and verify the reliability of this LLM-based ovaluation, we conducted an additional manual validation study. We randomly sampled 40 task trajectories from our online experiments and had expert human annotators independently review the final outcomes determined by the LLM judge.
The results, summarized in Table 13 show 97.5% agreement rate between the LLM's judgments and human experts. The LLM was highly accurate, particularly in identifying failures. This high degree of agreement strongly supports the reliability using GPT-4o as a practical and accurate proxy for task success in the ATTW-online setting.
Table 13: Human Validation of the LLM Judge for ATTW-online Evaluation.
Task Outcome (LLM Judge)
Number of Samples
Human Expert Confirmation
Failure
29
29 Confirmed Failures
Success
11
10 Confirmed Successes
Total
40
39 Agreements (97.5%)
25

---

Published in Transactions on Machine Learning Research (02/2026)
Research on the Automated Annotation Quality of GPT-4o
We conducted systematic study on the annotation quality of GPT-4o, aiming to evaluate its consistency and accuracy on standard benchmark tasks. Specifically, we randomly sampled 50 instances from each of two public datasets ATTW and MM-Mind2Web. Three professional annotators, who are proficient both Android and Web platforms and have experience with GUI Agents, were invited to independently annotate each instance. During the annotation process, all annotators completed their work independently without knowledge of each ofh'r's labels. The annotation platform provided unified interface and detailed instruction document, which included example data, annotation criteria, and strategies for resolving common ambiguities.
We adopted a majority voting strategy to generate the human reference labels (the "gold standard") and compared them against the automated annotations produced by GPT-4o. The results show that GPT-4o achieved 90% agreement with the human annotations across both tasks. Moreover, it completed all annotations in approximately hours, demonstrating strong efficiency and practical potential.
Regarding cthics, this study approved by the Institutional Review Board (IRB) of our institution. There were no foreseeable physiological or psychological risks to participants, and all annotators provided informed consent before participation, fully understanding the nature and objectives of the study. Annotators were compensated on an hourly basis at a rate not lower than the local minimum wage, and in accordance with the average pay standards for professional data annotators. This complics with the NeurIPS Code of Ethics regarding fair compensation for labor involved data collection and curation.
J Prompt
Prompt of AITW GPT-4o input
As an expert in the field GUI and reinforcement learning, you will receive complete screenshots and textual descriptions interactions for given task. You need to evaluate specific step in terms of its value within the task chain, similar to what value function does in reinforcement learning. Detailed criteria and standards are given below.
## Explanation of the input content: Task: Brief description
the current GUI task, such as implementing the "Get Hong Kong hotel prices" task in Android GUI.
2. Complete operation description and corresponding screenshot sequence for the task
(1) Text description of operations: Contains 11 types of GUI operations. Specific fields and their meanings are as follows: [i] CLICK: Click on specific position on the screen. If link software, it will enter; if be selected. The "click_point" is represented by two- dimensional array indicating the position of the click, relative to the top-left corner of the screenshot and within a range from 0.0 to 1.0.
text, it will
example: "action_type"; "CLICK", "click_point"; [0.5, 0.5] [2] TYPE: An action type that sends text. Note that this simply sends text and does not perform any clicks element focus enter presses for submitting text. example: "action_type"; "TYPE", "typed_text"; "capital of England"
26

---

Published in Transactions on Machine Learning Research (02/2026)
[3] PRESS_BACK: Return to the previous page. Usually the previous webpage.
example: "action_type"; "PRESS_BACK"
[4] PRESS : Return to the system home page. Use this action to return to the home screen when the current screen is not the desired
you can reselect the program you need
to enter.
example: paction_type" "PRESS HOME"
[5] PRESS ENTER: Press the enter key to execute a step. Generally, after confirming the input text, use this action start the search.
example: "action_type": "PRESS_ENTER
[6] STATUS TASK_ MAASK_CMMPLEEE An action used to indicate that the desired task has been completed and resets the environment. This action should also be used if the task is already completed and there is nothing more to do. For example, the task
turn on the Wi-Fi when TAST_AAASKKSTAA example: "action_ type:: "STATUS _ASK COMPLETE"
already on.
[7] STATUS TASK IMPOSSIBLE: An action used to indicate that the desired task is impossible to complete and resets the environment. This can result from various reasons including UI changes, Android version differences, etc.
example: "action_typer "STATUS TASK IMPOSSIBLE" [8] SCROLL_DOWN! Scroll down.
oxample: pection_typ"" "SCROLL_DOWN"
[9] SCROLL_UP. UP: Scroll up.
example: "action_type".. "SCROLL_UP"
[10] SCROLL.LEFT: Scroll left.
example: " pactiin_ttpp""e
"SCROLL_LEFT"
[11] SCROLL_RICHT: Scroll right.
example: "action_type"; "SCROLL_RIGHT"
(2) Corresponding screenshot before each operation. If the operation is of the "CLICK" type, the click position is marked with red dot in the image.
3. The current action to be evaluated and the corresponding screenshot.
## Evaluation Criteria:
Here are the detailed descriptions the two levels. Attention needs to be paid whether the action taken based on the current screenshot promotes efficient task execution, rather than the relevance of the content shown in the current screenshot to the task:
Level 1: The action is not the optimal choice for completing the
task at this moment, which may lead to deviations from the task flow. For example:
(1) Incorrect text input.
(2) Clicking button that might lead to advertisement.
(3) Announcing the task's success when it has not actually been achieved.
Level 2: The action is the optimal and correct choice for completing the task at this moment. For example:
27

---

Published in Transactions on Machine Learning Research (02/2026)
(1) When showing task completion, the displayed content can fully achieve it.
(2) When entering an unrelated interface, you can return the main screen by executing TRRES_HOOES
(3) Selecting the most correct entry point to complete the current task.
## Output requirements:
Format: {"rating": int, "explanation", str). Do not include any additional characters beyond this format The "rating" field should represented by the number or indicating the evaluation level. The "explanation" field should explain the evaluation process that led to this rating, without including descriptions of operations after the current step ( future operations are considered unknown).
Example Input:
Task Requirements: What the capital of England?
Action and ScreenShot:
" 524, 0. 0.06] text": "capital of England" step 2: saction_type". "PRESS ENTER"
step 0: faction_type" "CLICK",
step 1: "action_type". "TYPE",
step 3: "action_type". "STATUS TASK COMPLETE"
Current Action: step "action_type"; "PRESS ENTER"
"PRESS_ENTER"
## Example Output:
("rating": 2, "explanation"; "The action of pressing enter after typing cappttl of England' is an appropriate step to get the answer to the task requirement of finding out the capital of England, which is an optimal action towards achieving the task goal.")
Task Requirements: 0 Action and ScreenShot: 0 Current Action: 0
Prompt of AITW critic input
As an expert in the field of GUI and reinforcement learning, you will receive textual descriptions of history interactions for given task. You need to evaluate the current action, similar to what a value function does in reinforcement learning. Detailed criteria and standards are given below.
Explanation of the input content: Task: Brief description of the current GUI task, such implementing the "Get Hong Kong hotel prices" task in Android GUI. 2. Description of History operation
28

---

Published in Transactions on Machine Learning Research (02/2026)
Contains 11 types of GUI operations. Specific fields and their meanings are as follows:
[1] CLICK: Click on specific position on the screen.
link software, it will enter; if it is text, it will be selected. The "click_point" is represented by tro-dimensional array indicating the position of the click, relative to the top-left corner of the screenshot and within range from 0.0 to 1.0.
example: "action, type": "CLICK", "click_point", [0.5, 0.5] [2] TYPE: An action type that sends text. Note that this simply sends text and does not perform any clicks for element focus or enter presses for submitting text.
example: "action_type"" "TYPE", "typed_ text": "capital of England"
[3] CRESS_BACK, Return to the previous page. Usually the previous webpage.
example: "action_type"" "PRESS_BACK"
[4] #RESS_HOMEL Return the system home page. Use this action return the home screen when the current screen is not the desired one, you can reselect the program you need to enter. example: "oction_type". "PRESS_HOME"
[5]
Press the enter key to execute a step. Generally, after confirming the input text, use this action to start the search.
example: section_type". "PRESS_ENTER"
[6] STATUS TASK COMPLETE: An action used to indicate that the desired task has been completed and resets the environment This action should also be used if the task is already completed and there is nothing more to do. For example, the task is to turn on the Wi-Fi when it is already on. example: faction_type"" [7] STATUS TTATUS_TASK_IET ITAAAASISIISIILLE....
"STATUS_ BAt___TASSSASSSS_A COMPLETE"
action used to indicate that desired task is impossible to complete and resets the environment This can result from various reasons including UI changes, Android version differences, etc.
example: cetion__tpe"" "STATUS_TASE TASK_ IMPOSSIBPSS
[8] SCROLL_DOWN, Scroll down.
example: Daction_type". "SCROLL_DOWN"
[9] SCROLL_UP UP: Scroll up.
example: "action_ "; "SCROLL_UP"
[10] SCROLL_LEFT: Scroll left.
example: paction_type". "SCROLL_LEFT"
[11] SIGHTL Scroll right.
example: "action_type". "SCROLL_RIGHT"
3. The current action to be evaluated and the corresponding screenshot (the screenshot before each operation. the operation is of the "CLICK" type, the click position is marked with red dot in the image.)
## Evaluation Criteria:
Here are the detailed descriptions of the two levels. Attention needs to be paid to whether the action taken based on the current
29

---

Published in Transactions on Machine Learning Research (02/2026)
screenshot promotes efficient task execution, rather than the relevance of the content shown in the current screenshot to the task:
Level 1: The action is not the optimal choice for completing the task at this moment, which may lead to deviations from the task flow. For example:
(1) Incorrect text input.
(2) Clicking a button that might lead to an advertisement.
(3) Announcing the task's s success when it has not actually been achieved.
Level 2: The action the optimal and correct choice for completing the task at this moment. For example:
(1) When showing task completion, the displayed content can fully achieve it.
(2) When entering an unrelated interface, you can return to the main screen by executing "PRESS_HOME."
(3) Selecting the most correct entry point to complete the current task.
## Output requirements: 1 or 2 (INT)
## Example Input:
Task Requirements: What is the capital of England?
Previous Action:
step "action_type". "CLICK", "click_point' "[0.524, 0.06]
step 1: "action_type": "TYPE", "typed_text": "capital of England"
Current Action and Screenshot:
step 2: "action_type": PPRESS_ENTER
## Example Output:
Task Requirements: 0
Previous Action:
0
Current Action and Screenshot:
<image> 0
Prompt of MM-Mind2Web GPT-40 input
As an expert in web interaction and reinforcement learning, you will receive complete sequence of web interaction steps and corresponding descriptions for a given task. You need to evaluate a specific step in terms of its value within the task chain, similar to value function in reinforcement learning. Detailed criteria and standards are given below.
## Explanation of the input content: Task: Brief description of the current web task, such "Search for product e-commerce website".
30

---

Published in Transactions on Machine Learning Research (02/2026)
2. Complete operation description and corresponding sequence for the
task:
(1) Text description operations: Contains 3 types web follous:
actions. Specific fields and their meanings
[1] CLICK: Click
web element at specific position. The is represented by tro-dimensional array indicating the absolute position of the click in pixels. example: "action_type"
"click", "click_point", [100, 150]
[2] TYPE:
Click and input text into a field at a specific position. The "click_point" is represented by two
dimensional array indicating the absolute position of the click in pixels.
- example: "action_type"; "type", rclick_point?" [200, 300],
"value": "search term"
[3] SELECT: Click at a specific position to open a dropdown
menu, then select an option. Note: The dropdown options may
not visible before clicking, and the "value" field represents the option that will appear and be selected only after the dropdown is opened. The "click_point" is represented by two-dimensional array indicating the absolute position of the click in pixels. example: "action seseiion_tpe";: "select", "click_point"; [150, 200], "value": "Qween"
(2) A corresponding screenshot each operation on the current page. The "click_point" position of current action is marked with semi-transparent red dot in the image.
3. The current action to be evaluated and the corresponding screenshot. Please note that you only need to evaluate the current Action (just one step within the complete operation sequence).
## Evaluation Criteria: Focus on whether the action taken the current step efficiently promotes task completion, not just its relevance to the current page:
Level 1: The action is not the optimal choice for completing the
task at this moment, which may lead to deviations from the task flov. For example:
(1) Clicking the wrong element.
(2) Typing incorrect irrelevant text.
(3) Selecting an incorrect dropdown option.
Level 2: The action is the optimal and correct choice for
completing the task at this moment. For example:
(1) Clicking the correct button or link to proceed.
(2) Typing the correct text into the appropriate field.
(3) Selecting the correct dropdown option.
## Output requirements:
Format: ("rating": int, "explanation": str). Do not include any additional characters beyond this format
The "rating" field should be or 2, indicating the evaluation level. The "explanation" field should explain the reasoning for
31

---

Published in Transactions on Machine Learning Research (02/2026)
this rating, without referencing any operations after the current step (future actions are unknown).
## Example Input:
Task Requirements: Search for "laptop" e-commerce website. Action and Screenshot:
step 0: poction_type"" "click", "click_point" [120, 40]
step 1: faction_type"" "type", "click_point". [300, 400], "value":
laptop"
step "action, " "click", "click_point"; [350, 400]
Current Action(to be evaluated);
step 1: faction_typer "type", "click_point"; [300, 400], "value": laptop"
## Example Output:
("rating": 2, "explanation": "The action of typing 'laptop' into the search field the correct and optimal choice for completing the task of searching for laptop on an e-commerce website. This action directly contributes to achieving the task goal." "}
Task Requirements: 0
Action and ScreenShot: 0
Current Action:
Prompt of MM-b critic input
As expert in web interaction and reinforcement learning, you will receive textual descriptions of history interactions for given web task. You need to evaluate the current action, similar to what a value function does in reinforcement learning. Detailed criteria and standards are given below.
Explanation of the input content: Task: Brief description of the current web task, such "Search for product
e-commerce vebsite".
2. Description of History operation
Contains 3 types of web actions. Specific fields and their meanings are as follows:
[1] CLICK: Click on web element at a specific position.
click_point" is represented by a two-dimensional array
indicating the absolute position of the click in pixels, such [100, 150].
example: faction_type": "click", Click and input text into a field at a specific position The "click_point" is represented by a cro-dimensional array indicating the absolute position of the click in pixels. example: "action_type": "type", "click_point"; [200, 300], value": "search term"
[100, 150]
[2] TYPE:
[3] SELECT: Click at specific position to open dropdown menu, then select option. Note: The dropdown options may not be
32

---

Published in Transactions on Machine Learning Research (02/2026)
visible before clicking, and the "value" field represents the option that will appear and be selected only after the dropdown opened. The "click_point" is represented by two- dimensional array indicating the absolute position of the click in pixels.
example: "action_type": "select", "click_point"; [150, 200], "value": "Qween"
3. A corresponding screenshot of each operation on the current page. The Selick_pointt position of current action marked with a semi -transparent dot in the image.
## Evaluation Criteria:
Here are the detailed descriptions of the two levels. Attention needs to be paid to whether the action taken based on the current screenshot promotes efficient task execution, rather than the relevance of the content shown in the current screenshot to the task:
Level 1: The action not the optimal choice for completing the task at this moment, which may lead to deviations from the task flow. For example:
(1) Clicking the wrong element.
(2) Typing incorrect or irrelevant text.
(3) Selecting an incorrect dropdown option.
Level 2: The action
optimal and correct choice for completing the task at this moment. For example:
(1) Clicking the correct button link to proceed.
(2) Typing the correct text into the appropriate field.
(3) Selecting the correct dropdown option.
## Output requirements: or (INT)
## Example Input:
Task Requirements: Search for "laptop" e-commerce website.
Previous Action:
step 0: paction_type"; "click", "click_poine [120, 40]
step 1: "ection_type"; "type", "click_point": [300, 400], "value":
laptop"
Current Action and Screenshot:
step "action_type": "click", "click_point": [350, 400]
## Example Output:
2
Task Requirements: 0
Previous Action:
0
Current Action and Screenshot:
<image> 0
Prompt of AITW GPT-40 get negative samples input
33

---

Published in Transactions on Machine Learning Research (02/2026)
As an expert in the field of GUI and negative sample data constructor you need to generate a new negative sample the current action based on historical screenshots and corresponding action descriptions, task description, and the original current action. Detailed criteria and standards are given below.
## Explanation of the input content:
1. Task: Brief description of the current GUI task, such as implementing the "Get Hong Kong hotel prices" task in Android GUI. History operation description and corresponding screenshot sequence for the task
(1) Text description of operations: Contains 11 types of GUI operations. Specific fields and their meanings are as follows: [1] CLICK: Click on specific position on the screen. If it is a link or software, it will enter; if it is text, it will selected. The "click_point"
represented by two- dimensional array indicating the position the click, relative the top-left corner of the screenshot and within range from 0.0 to 1.0.
example: paction_type"" "CLICK", "click_point"; [0.5, 0.5] [2] TYPE: An action type that sends text. Note that this simply sends text and does not perform any clicks for element focus or enter presses for submitting text. example: "action_type", "TYPE", "typed_text", "capital England"
[3] [RESS_BACK, Return to the previous page. Usually the previous webpage.
example: "action_type": "PRESS_BACK"
[4] PRESS_HOME: HOME: Return to the system home page. Use this action to return to the home screen when the current screen is not the desired one, so you can reselect the program you need enter.
example: faction_type". "PRESS "PRESS_HOME" HOME"
[5] PRESS ENTER: Press the enter key to execute step. Generally, after confirming the input text, use this action to start the search.
example: "action_type" action used to indicate that [6] STATUS TASK__TSS____ TASK_AAASSSTTTTTE COMPLETE:
"PRESS_ENTER:
desired task has been completed and resets the environment. This action should also be used if the task is already completed and there is nothing more to do. For example, the task is to turn on the Wi-Fi when it is already on.
example: "action_type" An action used to indicate that the [7] TATTUS_TASK_IMMOSSST desired task is impossible to complete and resets the environment. This can result from various reasons including UI changes, Android version differences, example: "action_typed
"STATUS_ TASK_ COMPLETE"
STATUS_TASE IMPOSSIBLE"
[8] SCROLL_DOWN! Scroll down.
example: "action_typp type:: "SCROLL_DOWN"
[9] SCROLL UP: Scroll up.
example: "action_type"; "SCROLL_UP
34

---

Published in Transactions on Machine Learning Research (02/2026)
[10] SCROLL_LEFT: Scroll left.
example suction_type"" "SCROLL_LEFT"
[11] SCROLL_RIGHT: Scroll right.
example: paction_type"" "SCROLL_RIGHT
(2) Corresponding screenshot before each operation. If the operation is of the "CLICK" type, the click position is marked with red dot in the image.
3. The positive current action and the corresponding screenshot.
## Criteria for generating negative samples:
The given input is positive current action that meets the Level standard below. To conduct data augmentation, need to generate its corresponding negative current action, i.e., the action described below as level 1.
Level 1: The action is not the optimal choice for completing the
task at this moment, which may lead to deviations from the task flow. For example:
(1) Incorrect text input.
(2) Clicking button that might lead to an advertisement.
(3) Announcing the task's success when it has not actually been achieved.
Level 2: The action is the optimal and correct choice for
completing the task at this moment. For example:
(1) When showing task completion, the displayed content can fully achieve
(2) When entering unrelated interface, you return the main screen by executing "PRESS_HOME HOMS.
(3) Selecting the most correct entry point to complete the current task.
## Output requirements:
Format: ("action_desc"; dict, "explanation" str). Do not include any additional characters beyond this format The "action_desc" field needs to provide the fields involved in the newly generated negative sample action according to the text description given above. The "explanation" field needs to explain the logic for giving this new negative sample.
## Example Input:
Task Requirements: What the capital of England? Previous Action and ScreenShot:
step 0: Paction_type". "CLICK", "click_poine 524, 0 06]
step 1: "action_type": "TYPE", "typed_text": "capital of England"
Origin Action:
step 2: "action_type": "PRESS_ENTER"
## Example Output 1:
{
"action_desc;; "aationnttpp"": "STATUS_ TASKS_TASK_AOG ' "explanation": "Since text about the capital of England has already been entered in the search box, pressing enter directly at this step should give the answer. However, if generate an
35

---

Published in Transactions on Machine Learning Research (02/2026)
action indicating task completion, it will seriously deviate from the current task."
## Example Output 2:
{
"action_ded" desc": (raction_type", "CLICK", "click_point"; "[0.87, 0. 0.52]")
"explanation: "Since text about the capital of England has already been entered in the search box, pressing enter directly this step should give the answer However, click the adjacent advertising area, it will deviate from the task."
generate a
Task Requirements: 0 Previous Action and ScreenShot: Origin Action: {
Prompt of MM-Mind2Web GPT-4o get negative samples input
As an expert in web interaction and negative sample data constructor, you need to generate a new negative sample of the current action based on historical screenshots and corresponding action descriptions, task description, and the original current action. Detailed criteria and standards are given below.
## Explanation of the input content:
1. Task: Brief description of the current web task, such as "Search for a product on an e-commerce vebsite".
2. History operation description and corresponding screenshot sequence for the task:
(1) Text description of operations: Contains 3 types of web actions. Specific fields and their meanings are follows:
[1] CLICK: Click
web element at specific position. The is represented by two-dimensional array indicating the absolute position of the click in pixels.
example: "action, type": "click", "click_point"; [100, 150]
[2] TYPE:
Click and input text into a field at a specific position. The "click_point" is represented by two-
dimensional array indicating the absolute position of the click in pixels.
example: "action_type": "type", "click_point"; [200, 300],
"value": "search term"
[3] SELECT: Click at specific position to open a dropdown
menu, then select an option. Note: The dropdown options may not be visible before clicking, and the "value" field represents the option that will appear and selected only after the dropdown is opened. The "'lick_points is represented by -ro-dimensional array indicating the absolute position of the click in pixels.
36

---

Published in Transactions on Machine Learning Research (02/2026)
example: cettion_tppe", "select", "click_point"; [150, 200], "value": "Qween"
(2) A corresponding screenshot each operation on the current page. The "click_poini" position of current action is marked with semi- semi-transparent red dot in the image.
3. The positive current action and the corresponding screenshot
## Criteria for generating negative samples:
The given input is a positive current action that meets the Level standard below. conduct data augmentation, we need to generate its corresponding negative current action, i.e., the action described below level 1.
Level 1: The action is not the optimal choice for completing the
task at this moment, which may lead to deviations from the task flow. For example:
(1) Clicking the wrong element.
(2) Typing incorrect or irrelevant text.
(3) Selecting an incorrect dropdown option.
Level 2: The action is the optimal and correct choice for
completing the task at this moment. For example:
(1) Clicking the correct button or link to proceed.
(2) Typing the correct text into the appropriate field.
(3) Selecting the correct dropdown option.
Output requirements:
Format: {"action, ("action_deac": desc":
dict, "explanation"; str). Do not include any additional characters beyond this format. The "action_desc" field needs to provide the fields involved in the newly generated negative sample action according to the text description given above. The "explanation' field needs to explain the logic for giving this new negative sample.
## Example Input:
Task Requirements: Search for "laptop" an c-comeecce website. Previous Action and Screenshot:
step 0: paction_tppe" "click", velick_point' [120, 40]
step 1: "action_type"" "type", "click_point"; [300, 400], "value":
laptop"
Origin Action: step faction_type": "click", "click_point": [350, 400]
## Example Output 1:
{
"action_dees desc": ("action_type" "click", "click_point"; [900, 100]},
"explanation": "Instead clicking the search button submit
the query, clicking random area on the page will help complete the search task and may deviate from the task flow."
## Example Output 2:
{
37

---

Published in Transactions on Machine Learning Research (02/2026)
"action_ " "ractiionttyp".. "type", "click_point"; [300, 400], "value": "asdfgh"),
"explanation"; "Typing irrelevant text into the search field instead of the correct query will not help achieve the task goal." "
Task Requirements: 0 Previous Action and Screenshot: (
Origin Action: {
K
Case Study
Here we randomly sample cases (Figure G our method and baselines.
from our test experiments to show the difference between
38

---

Published in Transactions on Machine Learning Research (02/2026)
Task:haa'' the latest video from GameSpot Reviews?
REVIEW
seenting
Type
Google
Google
Google
DigiRL offiin
Click
Click
Click
Click
Click
Google
online online
Click
Click
Click
Click
Done
Figure 6: Case study Ours, DigiRL offline and DigiRL online on the task: what's the latest video from GameSpot reviews?
39

---

Published in Transactions on Machine Learning Research (02/2026)
Taslc Search top rated burger restaurants on Google Maps
Our
Clic
Click
Type
Done
DigRL OffR
Scroll Down
Click
Press Home
Scroll Down
Clock
Press Back
Click
Clock
Pres Home
M
DigRR online
Screl Down
Clok
Press Back
Back
Press Back
Figure Case study of Ours, DigiRL offline and DigiRL, online on the task: scarch for top rated burger restaurants on Google Maps.
40

---

Published in Transactions on Machine Learning Research (02/2026)
Task What's on the menu at Red Lobster?
Our
Clrck
Clcck
Cick
Type
Clkc
Clrk
Clock
Clkc
Difi offine
Cleck
Ok
Clcc
Cl
Clkck
Clock
Prest Back
Clock
Preas Back
Dijil
Type
Figure 8: Case study of Ours, DigiRL offline and DigiRL online on the task: what's on the menu at Red Lobster?
41

---

Published in Transactions on Machine Learning Research (02/2026)
Task:What the speed of a rocket?
Google
Type
DigRL offiie
Click
Press Back
Clic
Click
Type
Press Enter
Google
DigiRL online
Down
Type
Figure 9: Case study of Ours, DigiRL offline and DigiRL online the task: what is the speed of rocket?
42


Reviews:
[{"review_id": "l5p08ABt1J", "reviewer": "Reviewer_kaKt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary of Contributions\n\nThe paper describes a training approach for GUI agents. The key idea is to build a dataset with action labels from a capable multimodal system (GPT-4o), then train a smaller model to predict these labels. Actions are labeled in two categories: clearly beneficial for the task, or not. This learned predictor serves as a static critic during policy optimization on offline data, eliminating the need for environment interaction during training.\n\nThey evaluate on AITW (mobile) and MM-Mind2Web (web). On AITW, their method achieves 30-35% offline task success rate with the SeeClick backbone, compared to 23-29% for DigiRL/DigiQ. Online, they report 49% task success vs ~39% for environment-based methods. On MM-Mind2Web, they achieve 50-56% step success rate, outperforming methods trained on the same data by 4-15%.\n\n**Strengths:**\n- Practical recipe that eliminates environment interaction during policy training\n- Evaluation across two platforms (mobile, web) with both offline and online protocols\n- Ablations confirm key design choices matter (SFT initialization, binary vs 3-class labels)\n- Prompts and methodology are documented in sufficient detail\n\n**Weaknesses:**\n- The \"Value Environment Model\" framing overstates what's happening—this is a distilled dense reward model, not a learned Q-function\n- The theory (Section 3.5) assumes Qθ approximates Q*, but nothing in training ensures this relationship holds\n- Distribution shift is handled only via SFT initialization with no explicit constraints during optimization\n- Key baselines are missing: using GPT-4o rewards directly, DPO, process reward models\n- Some comparisons mix model sizes (their 9.6B vs 200M baselines)", "claims_and_evidence_yn": "No", "claims_explanation": "I think the claims in this paper are partially supported by evidence.\n\n## Supported claims\n\nThe paper makes several empirical claims about performance that are supported by experiments with comparisons to baselines and ablations. Although the baselines could be stronger, it seems like the claims about the recipe for agent training, and its overall performance in comparison to alternatives is reasonably made.\n\n## Unsupported claims\n\nThe paper claims to learn a Q-function and presents theoretical results supporting the performance of the training approach based on this assumption. However, nothing in the paper actually justifies the use of the name Q-function as no part of the system actually estimates the sum of discounted rewards. Instead, the learning system is really a distillation of GPT-4o labels that can be trained against efficiently.", "audience_yn": "Yes", "audience_explanation": "While I have some concerns about the overall framing of the paper, I suspect that a fraction of the TMLR audience would be interested in knowing the results presented. The paper describes a useful training recipe, which I suspect is particularly important for GUI training where action can be clearly identified as valuable vs not and a central challenge is the high branching factor of potential actions that can be taken.", "requested_changes": "## Critical (required for acceptance)\n\n1. **Revise Q-function framing.** The paper claims to learn a \"state-action value function\" but what is actually learned is a classifier over GPT-4o's binary action labels. This is a dense reward model, not a Q-function—there is no estimation of cumulative discounted return, no Bellman structure, no temporal credit assignment. The method section and abstract should be revised to accurately describe what is being learned.\n\n2. **Remove or substantially revise the theoretical analysis (Section 3.5).** Theorem 3.1 assumes Qθ approximates Q* with bounded error, but nothing in training justifies this assumption. Validating that GPT-4o labels agree with humans (90%) does not establish that the labels track long-term value. Either (a) remove the theory, (b) provide evidence that the assumption holds, or (c) reframe the theory as analyzing a different quantity (e.g., a reward model) with appropriate caveats.\n\n## Suggested improvements\n\n3. **Clarify baseline comparisons.** Several comparisons mix model sizes (e.g., 9.6B SeeClick vs 200M baselines). Please clearly separate results by model scale and discuss how baselines were selected.\n\n4. **Report cost comparisons.** The paper emphasizes cost-effectiveness relative to environment-based methods but does not compare against using GPT-4o labels directly during training. What would it cost to use GPT-4o as the reward signal without distillation? This would help readers understand the value of the distillation step beyond API cost savings.\n\n5. **Discuss distribution shift handling more thoroughly.** The method relies on SFT initialization to keep the policy in-distribution, with no explicit constraint during optimization. The 81% coverage number (footnote, page 6) is unexplained. How was this measured? How sensitive is performance to policy drift?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qy7frYKiXW", "reviewer": "Reviewer_n77i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary \n\nThe paper proposes \"VEM,\" an environment-free Reinforcement Learning (RL) framework for training GUI agents without requiring costly online interactions during policy optimization. The authors introduce a two-stage approach: first, pretrain a Value Environment Model (VEM) on offline data to estimate long-term action utility using semantic supervision distilled from a teacher model. And second, guide policy exploration by maximizing signals from this frozen VEM. This method effectively bypasses the compounding errors associated with traditional next-state prediction models.\n\n# Strengths\n\nS1: On the AITW benchmark, VEM achieves 49.15% online task success, surpassing environment-free baselines by 23-49% and matching environment-based methods. On MM-Mind2Web, it outperforms baselines trained on the same data by 4-35%.\n\nS2: The framework eliminates interaction latency, reducing training time to approximately 12-36 hours compared to the 3-5 days required for environment-based counterparts on Qwen2.5VL-3B model.\n\nS3: The authors provide a theoretical performance bound (Theorem 3.1) quantifying policy suboptimality based on dataset coverage and value approximation error.\n\n# Weaknesses\n\nW1: **Limited scope (no desktop benchmarks)**: The evaluation is restricted to mobile (Android) and web browser environments. It lacks evaluation on general desktop OS tasks (e.g., Windows/macOS/Linux platforms), limiting claims of broad GUI generalization.\n\nW2: **Dependency on annotator quality**: The framework relies entirely on the quality of the \"teacher\" model (GPT-4o) for data annotation, thereby capping performance to the teacher's capabilities.\n\nW3: **Coarse reward signal**: The reliance on binary labels (optimal/suboptimal) is somewhat simplistic and may lack the granularity needed for nuanced decision-making.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors support their claims with comprehensive empirical evidence, demonstrating that VEM outperforms or matches baselines like DigiRL and SeeClick across both mobile (AITW) and web (MM-Mind2Web) benchmarks. To address concerns regarding LLM-based metrics, they provide human validation studies showing a 97.5% agreement rate for the online judge and 90% for data annotation. Furthermore, the claims of efficiency are substantiated by comparisons showing significantly reduced training times compared to environment-based methods , and the framework is grounded in theoretical performance bounds linking suboptimality to dataset coverage.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a critical bottleneck in training autonomous GUI agents by proposing an efficient \"environment-free\" offline RL framework that bypasses costly live interactions. This approach is highly relevant to researchers focusing on GUI agent/Computer use agent.", "requested_changes": "No critical change is required from the reviewer’s perspective. The authors can consider two points to consolidate the work further.\n\n\n1. Desktop benchmarks: To substantiate the claim of training \"generalizable GUI agents\", please expand the evaluation beyond mobile and web platforms to desktop environments. Specifically, the reviewer recommends including online evaluation on OSWorld[1] and offline evaluation on datasets like ScreenSpotPro[2] and UI-Vision[3] to demonstrate robustness across different OS layouts.\n\n\n2. Labeling intuition: Please provide a brief explanation for the ablation result in Table 9, where binary labeling outperforms the 3-class scheme. Clarifying why the intermediate class degrades performance would strengthen the justification for the chosen supervision signal.\n\n\n[1] Xie et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.\n\n[2] Li et al. ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use.\n\n[3] Nayak et al. UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OvJ5BpBMSh", "reviewer": "Reviewer_jojP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces VEM, an environment-free method to train GUI agents without the need for trajectory-level rewards. It does so by leveraging a teacher LLM's ability to assess whether the latest action taken by the GUI agent led to an improvement (i.e. it moved the agent toward its goal) or not (worsening the situation/clicking on ads/declaring the task is done while it is not complete). This has the advantage of providing a denser supervision, as the reward is perceived at every action. Using the teacher LLM's annotation, they earns a state-action value function which is then used to guide the training of the policy. The method outperforms other environment-based and environment free baselines presented in the paper on 2 benchmarks, while being more sample-efficient and faster to train. Moreover, the ablation studies confirm the efficiency of the method, showing it is effective with smaller datasets and small models (200M parameters).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims regarding performance and efficiency are well-supported by the following evidence:\n\n- The method of distilling human-like priors from a teacher LLM (GPT-4o) into a \"Value Environment Model\" (VEM) to provide step-wise supervision is technically sound. The authors address the critical challenge of sparse rewards in GUI tasks by generating step-level labels (optimal vs. suboptimal), which creates a dense signal that guides the policy.\n- The paper provides evaluation on two benchmarks: Android-in-the-Wild (AITW) and Multimodal-Mind2Web, which shows that the method  is not benchmark-specific\n- In Offline settings, VEM significantly outperforms environment-free baselines (e.g., surpassing DigiRL-offline by ~7% in Task Success Rate on AITW General).\n- In Online settings, the method achieves state-of-the-art results (49.15% success on AITW General), outperforming environment-based methods like DigiRL-online.\n- The authors convincingly demonstrate the efficiency of their approach. They report training times of ~12 hours versus 3-5 days for environment-based counterparts and achieve these results with significantly fewer samples. The paper also includes ablation studies, which confirm the necessity of SFT initialization to mitigate distribution shift and validate that binary value labeling is more robust than finer-grained rewqrds.\n\n- Finally, the authors address the reliability of their LLM-based evaluation metrics (GPT-4o judge) by conducting a manual validation study (Table 13), showing a 97.5% agreement rate with human experts. This strengthens the validity of the reported online success rates.", "audience_yn": "Yes", "audience_explanation": "The findings are likely to be of interest to a broad segment of the TMLR audience, for the following reasons:\n- The paper addresses the bottleneck of the cost and instability of environment-based training while having good performance.\n- The method demonstrates a successful application of offline RL with dense value shaping to handle GUI interactions while showing how to initialize the policy with SFT.\n- The strategy of using a powerful teacher (GPT-4o) to annotate offline trajectories for student policy training is a valuable case study in model distillation and synthetic data utilization.\n- It is an approach complementary to the practice of training on simulated environments [1]\n\n[1] https://arxiv.org/abs/2504.11543", "requested_changes": "### Cost of sampling\nThe method is presented as being more efficient than environment-based ones. While this is demonstrated by having good performance with less training data, I think that understanding the monetary cost of collecting the dataset (i.e. the API calls) would make readers understand the trade-off better (env-based vs env-free).\n\n### Related works\n\nI think that the method presents an interesting connection to [2], which shows that learning a process-reward model that rewards moving towards a goal leads to improved performance in the math setting. I feel like it could be interesting to mention it, if the authors find it relevant. However, I don't think it is necessary for acceptance.\n\n### Writing/clarification\nI think section 3.1 could use a bit of a re-write to be more clear and explicit about all the functions and variables:\n- The text could flow better; it feels like an enumeration written in a paragraph\n- For the 2nd paragraph (\"Given a ...\", it should be connected to the first in a more clear way.\n\n[2] https://arxiv.org/pdf/2410.08146", "broader_impact_concerns": "I don't have any concern on the ethical implications of this work.", "additional_comments": "I recommend acceptance. This paper addresses a bottleneck in GUI agent training with a sound, efficient environment-free approach. The experimental results support the claims of improved efficiency and performance compared to environment-based baselines. Addressing the minor clarifications regarding sampling costs and writing flow will further strengthen the quality of the submission.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)