Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation
Authors: Yuyang Wanyan
Xi Zhang
Haiyang Xu
Haowei Liu
Junyang Wang
Jiabo Ye
Yutong Kou
Ming Yan
Fei Huang
Xiaoshan Yang
Weiming Dong
Changsheng Xu
Venue: NeurIPS 2025 poster ()
Keywords: GUI agent
critic models
RL
GRPO
Primary area: applications
TL;DR: A GUI-Critic for Pre-Operative Error Diagnosis method with Data Collection Pipeline with Reasoning Bootstrapping and Reinforcement Learning

Abstract: In recent years, Multimodal Large Language Models (MLLMs) have been extensively utilized for multimodal reasoning tasks, including Graphical User Interface (GUI) automation. Unlike general offline multimodal tasks, GUI automation is executed in online interactive environments, necessitating step-by-step decision-making based on the real-time status of the environment. This task has a lower tolerance for decision-making errors at each step, as any mistakes may cumulatively disrupt the process and potentially lead to irreversible outcomes like deletions or payments. To address these issues, we introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution, by reasoning about the potential outcome and correctness of actions. Specifically, we propose a Suggestion-aware Group Relative Policy Optimization (S-GRPO) strategy to construct our pre-operative critic model GUI-Critic-R1, incorporating a novel suggestion reward to enhance the reliability of the model's feedback. Furthermore, we develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test, filling existing gaps in GUI critic data. Static experiments on the GUI-Critic-Test across both mobile and web domains reveal that our GUI-Critic-R1 offers significant advantages in critic accuracy compared to current MLLMs. Dynamic evaluation on GUI automation benchmark further highlights the effectiveness and superiority of our model, as evidenced by improved success rates and operational efficiency. The code is available at https://github.com/X-PLUG/MobileAgent/tree/main/GUI-Critic-R1.

Look Before You Leap: A GUI-Critic-RI Model for Pre-Operative Error Diagnosis in GUI Automation
Yuyang Wanyan 1,2,3t Xi Zhang"; Haiyang Xu Haowei Liu', Junyang Wang", Jiabo Ye Yutong Kou', Ming Yan 1 Fei Huang", Xiaoshan Yang! ,21, Weiming Dong 1.2 Changsheng Xu MAIS. Institute of Automation, Chinese Academy of Sciences, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, China Alibaba Group
wanyanyuyang2021 @ @an.ac,cn, xiaoshan. yang @ nipr.ia.ac.cn, (shuofeng.xi) xhy,, yml 19608 @ @alibaba-inc.comm
Abstract
In recent years, Multimodal Large Language Models (MLLMs) have been exten- sively utilized for multimodal reasoning tasks, including Graphical User Interface (GUI) automation. Unlike general offline multimodal tasks, GUI automation is executed in online interactive environments, necessitating step-by-step decision- making based on the real-time status of the environment. This task has a lower tolerance for decision-making errors at each step, as any mistakes may cumulatively disrupt the process and potentially lead to irreversible outcomes like deletions or payments. To address these issues, we introduce a pre-operative critic mecha- nism that provides effective feedback prior to the actual execution, by reasoning about the potential outcome and correctness of actions. Specifically, propose a Suggestion- awweee Group Relative Policy Optimization (S-GRPÓ) strategy to construct pre-operative critic model GUI-Critic-R/ incorporating novel sug- gestion reward to enhance the reliability the model's feedback. Furthermore, develop reasoning-booosttrapiing bootstrapping based data collection pipeline create GUI-Critic-Tree -rriic and a GUI-Critie-Teed Test, filling existing gaps in GUI critic data. across both mobile and web domains
Static experiments the GUI-Critic- reveal that our GUI-Critic-RI offers significant advantages in critic accuracy com- pared current MI.LMs. Dynamic evaluation on GUI automation benchmark further highlights the effectiveness and superiority of our model, as evidenced by improved success rates and operational efficiency. The code is available https: //github. com CoomXXPLUUS CoolUG/MMobllee httee///ooolleget//eeeeeeeeeeeee tteee/eee///ee/e/e wwwree/mai//aee /twam/mannu/////a///ee//ea/aa/aa CriticcRi..
Introduction
Recently, Multi-mon Large Language Models (MLLMs), leveraging their remarkable perception and reasoning capabilities, have demonstrated outstanding capabilities in various domains [2, 41]. Among these, GUI automation, as practical multi-modal application scenario, is emerging as significant technological revolution in artificial intelligence interactions [38, 16, 50, 15, 44, 26, 27]. To be specific, given an online GUI device and natural language instruction, requires the GUI agent driven by MLLMs generate series of precise operations similar to how humans do [48, 32], such as click, type, and scroll.
Unlike traditional offline multimodal tasks such as visual question answering [19] and optical character recognition [14], GUI automation task operates within an online interactive environment
'Equal contribution. Work done during internship at Tongyi Lab, Alibaha Group. *Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(8)
Tøsk Instruction: the Bluetooth.
Roalise
Qaxe-7hh
ProCeit Owan-18
Coac GPT-4o
Success Rate AndroidWorld
27.6
Succco Rucchb)
23.2
22A
Svipe
ADP)
devceo
poferenc
(Busturt
Fan
(b)
Task Instruction: Rename the second audio to mauiiii
(c)
Task Instruction: Turn the Bluetooth.
Clet Fourth tatto
driver.
Dela
Geitegg aptirdll
streen and fury Battsth
GLI
CICC The First burtord
Down
Centrol Center
Figure 1: (a) shows an example of GUI automation. Case studies in (b-c) demonstrate how pre- critic prevents erroneous and redundant actions in GUI automation. (d) illustrates the quantitative performance comparison between pre-critic methods and baseline on AndroidWorld [27].
and has some inherent key challenges [32]. To be specific, the agents are required to generate coherent and sequential operations step by step. In this way, an error in one step can have cumulative effects on subsequent operations (e.g., delete a file shown in Figure I(b)), thereby disrupting the entire interactive process. However, constrained by limited reflection capabilities, current MILMs usually struggle to detect errors independently [34]. Therefore, to ensure the single-step accuracy, it is necessary to provide MLLM- based agents with additional feedbacks that incorporates critic analysis, such as assessments of the action correctness, potential outcomes, and action suggestions. Nevertheless, in the dynamic environment, erroneous operations usually require additional steps to correct (e.g., refunding after an incorrect payment), and some dangerous errors may be irreparable (e.g., deleting files shown in Figure 1(b)). Therefore, to prevent these issues from happening, we believe that the critical feedback should be provided to the agent before the action actually executed. Additionally, completing user instruction
GUI device typically entails multiple pathways. From an application perspective, the GUT agent is expected to complete instructions with an optimal path that contains the least number of steps. In this way, the prior critical feedback may also prevent the model from selecting a sub-optimal path with more steps as shown in Figure (a)) thereby enhancing the efficiency of completing instructions.
Considering the above issues, in this paper, we propose pre-operative critic (pre-critic) mechanism for GUI automation. Specifically, before performing operation in online environments, the pre- critic mechanism first evaluates whether the operation generated by the agent is beneficial to the instruction completion, through comprehending the sereenshot and analyzing potential results of the operation. Then, it provides real-time critical feedback to the agent, including the underlying causes of errors and corrective suggestions, to assist in refining its decision-making process. For example, as shown in Figure 1 (b), given the instruction Rename the current audio to 'my may_adiio', the agent initially predicts the action of click (delete button) in the 5-th step. Such operation may cause the audio file to be deleted and the task to fail. Fortunately, with the pre-critic mechanism, we can catch the dangerous error before executing, and provide objective feedback the agent. Besides, Figure 1(c) illustrates case where the pre-critic identifies a more efficient method to turn on Bluetooth (i.e., enabling Bluetooth via the control center), thereby reducing the operation steps. We also exhibit some statistic results in Figure 1(d), where the pre-critic helps increase the success rate of baseline from 22.4% to 27. 7.,% in the dynamic GUI automation [27] * In conclusion, these figures demonstrate that the introduction of pre-critic can effectively alleviate the aforementioned issues of error revision and operational inefficiency in online environments.
An intuitive approach to implementing a pre-critic model is leveraging existing MILMSs. However, closed-source models [13] incur heavy efficiency and cost issues, making them unsuitable for real-time GUI automation. Besides, open-source models [5, 2, 41] struggle as pre-critic models due to inherent
Mooe detailed experimental results about efficiency can be found in Table 2

---

limitations in comprehending GUI interface and forecasting interaction outcomes [16, 36, 51]. To this end. we propose a specialized 7B model GUI-Critie-R1 that harmonizes performance and efficiency for GUI pre-critic. In particular, to enhance the model's GUI reasoning and generalization capabilities, we introduce a Suggestion-aware Group Relative Policy Optimization (S-GRPO). In S-GRPO, a suggestion reward is innovatively designed to refine the model's 's critic reasoning process and ensure it provides reliable suggestions for fixing the error operation. Moreover, since it lacks both training and test datasets for pre-critic in GUI automation, we develop and GUI-Critie- Test dataset. Specifically, GUI-Critic- Train contains 6K high-quality chain-of of-thought data about mobile devices for robust training. GUI-Critic- aims to explicitly evaluate the pre-critic model's diagnostic capabilities when exposed to novel instructions or applications. In experiments, we first compare our GUI-Critic-RI with advanced MILLMs on the GUI-Critic-Test, and find that our model achieves satisfactory results in the critic accuracy. Then, we also apply the GUI-Critic-RR to a real-time GUI automation benchmark (i.e. AndroidWorld [27]), the improvements on the success rate further demonstrate the effectiveness of the proposed method.
crasoningg-ootttrappiin based data
collection pipeline to construct a GUI-Critic-Trane
includes 1k samples encompassing both mobile and web scenarios, and
Our main contributions are summarized as follows:
1. To the best of our knowledge, we the first to investigate a pre-operative critic mechanism for diagnosing GUI operations. To achieve it, we propose Suggestion-awarr Group Relative Policy Optimization strategy integrating a novel suggestion reward, and develop our pre-critic model GUI-Critic-RI This model is capable of delivering constructive and insightful feedback to refine the GUI reasoning process.
2. We present reasoning bootsrrapigiggni based data collection pipeline to construct GUI- Critic-Train dataset, comprising 6k high-quality chain- of of-thought annotations. Additionally, a GUI-Critic-Test dataset is developed to comprehensively evaluate the critic model's performance both mobile and web domains.
3. Extensive experiments on both GUI-Critic-Ted Test dataset and dynamic GUI automation bench- mark validate the efficacy of our GUI-Critic-R model in producing reliable judgments and feedback for GUI operations.
2 Related Work
LLM-based GUI Agent. Recently, significant attention has been directed toward Graphical User Interface (GUI) agents for task automation on smart devices designed for the mobile and web environments [50, 15, 36, 49, 56, 45, 10, 52, 20, 30, 39, 40]. Despite these advancements, GUI agents frequently make erroneous decisions. Some research has incorporated reflection modules in GUT automation frameworks to verify operation correctness based on the both current screenshot and the screenshot after execution [? 16, 1]. For example, Mobile-Agent- v2 frameworks [36] employ multi-agent architectures that separate planning, Decision-making and reflection to optimize task tracking, memory, and eтror correction. But these methods require additional steps to undo actions and pose the risk of irreversible operations, resulting in lower efficiency and accuracy. In this paper, we introduces pre-critic mechanism for diagnosing potential errors.
Critic Model for LLMs. Large language models (LLMs) do not always generate the best reasoning output when performing challenging inference tasks [34]. To address this limitation, several studies propose 121f-refinemen1 31, 24] and self-reflect techniques [29, 9, 35, 47, 55, 6, 7] to reifine the output themselves. However, their effectiveness is largely restricted by their reliance on the inherent capabilities of LLMs, which may hinder the broader application and scalability of these methods [11]. In contrast, several studies [22, 43, 42] explore employing independent critic model to produce natural language feedback for the evaluation of outputs generated by Large Language Models (LLMs). Critic-V [51] extends this inspiration to the area of VLMs to train critic vision-language model to locate imperfections in visual content perception and errors in reasoning steps. However, they focus on general offline tasks. Differently, we explore the application of critic model in more complex scenarios of GUI automation, which presents increased challenges due to its operation in an online environment. In this paper, we perform pre-operative critic to resolve the challenges.
Reinforcement Learning for Reasoning. Recent research has increasingly focused on enhancing the reasoning capabilities of LLMs through Reinforcement Learning (RL), drawing inspiration from
3

---

Collection Pipeline Reasoning Rootstrapping
GUI-CritieRI
GUI Operations Collection
GUI Critiques Generation
RFT Cold-Start
Opanctoo Strembment GUI-CIR
Pragrasive CoT Paradign
Labuinn The cungg
Tra-Torig
Tesk Insmectior Search for Jonert
Hood
de Eocgle 1e.
Operahan Hafery Tetter tun louth Giskkn
Pronesi
Suggestion- AAarr Group Relative Policy Optimization (S-GRPO)
Correct Actios Oee sheee bultae fast
Resoning Bauistrapping
Negotive Operations Sampling
Query
Text Te Operchor
unmumnat shows the we Latter
fore of
fellee ballon
Concition Oin shene be
latte' bette morted
Oulpel
Filtering
Reword
Accurscy Iverect chaaea
Porrett Reword
Adnanlage
Figure 2: The left shows reasoning- hoaprsgapooootttpppnnngg based data collection pipeline, including the GUT operations collection and GUT critiques generation. Specifically, progressive CoT paradigm and reasoning- bootstrapping strategy are employed to ensure the quality of critiques. The right illustrates the training strategy for our GUI-Critic-R RI model. The process begins with a RFT cold-start on the GUI-Critic- Train dataset, and followed by the implementation of our proposed S-GRPO. Besides, a novel suggestion reward is employed to constrain the correctness of suggestions.
models such as DeepSeek-RI [8] and Kimi-1.5 [33]. These models employ rule-based reward mecha- nisms to enhance reasoning performance. Several studies attempt adapt the idea of reinforcement learning with verifiable rewards in multimodal scenarios. For instance, R1-V [53] investigates the application of rule-based RL in geometry problems and object-counting tasks, while Visual-RFT [17] extends this approach to open vocabulary and few-shot detection, reasoning grounding, and fine- grained few-shot classification. Recent studies [12, 25, 23, 53, 46] further generalize the algorithm to address more general tasks such as multimodal mathematical reasoning, decision- making, and planning. In this paper, we extend the reinforcement learning algorithm to train critic model that diagnoses operations in GUI automation, and propose S- S-GRPO strategy with a novel suggestion reward to enhance the model's s reasoning capabilities.
3 Method
3.1 Problem Definition of Pre-Operative Critic
The GUI automation task can be formally characterized as Markov Decision Process: M = (E, Tagent)) The state of the environment e E & consists of user instruction, historical interactions, and the current screenshot of the device. The action space A encompasses all available operations (actions) including click, long press, type, scroll, home, back, and done. At each step, the MLLM-based agent Tagent observes the environment and selects an action a. Upon executing it receives a novel observation é', with the probability given by the state transition function P(e' (e, a). The agent iterates this process until accomplishing the desired instruction or encountering terminal state.
In this paper. we propose a pre-operative critic model (eratic(e, a) to critique the decision-makinn process of Tegents before the action is actually executed. Specifically, taking state and action a as inputs, the model produces a correctness score E [0, 1] that reflects the rightness of a. Besides, it generates critique e and corrective suggestion s in natural language, with the former explaining the rationale behind the correctness judgment and the latter suggesting a better action if a is incorrect.
3.2 Data Collection Pipeline with Reasoning Bootstrapping
In this section, we detail the data collection pipeline of our proposed GUI-Critic-Train and GUI- Critic- datasets. Each sample the dataset includes the environmental state of the step, a operation candidate a, well annotations for correctness score / and suggestion This section

---

commences by introducing the collection of operations in Section 3.2.1, followed by the presentation of our reasoning bootstrapping method for generating critique in Section 3.2.2.
3.2.1 GUI Operations Collection
To begin with, we collect successful automation trajectories from publicly accessible datasets, encompassing correct step-level operations across various GUI scenarios. The action in a specific state e is considered as the operation candidate whose correctness score is positive (1 1), and the corresponding action description considered the suggestion 8.
Negative Operations Sampling. Subsequently, we collect samples with negative score (1 0) based on the states of the above correct operations. To be specific, given these states, open-source MLLM-based agents are first employed to predict a set of operations. Then, these operations are evaluated according to rule-based criteria, and we retain those deemed incorrect. In this way, the obtained negative operations are aligned as closely as possible with the real error distribution in GUI environments, thus ensuring the quality of the dataset. The corrective suggestions for these samples are the same as the corresponding positive samples.
Data Filtering. The data collected using the aforementioned methods may not be entirely reliable. This is because that public datasets could contain erroneous annotations, and the rule-based criteria are not entirely reliable, they may also erroneously penalize some correct operations. Consequently, further data cleaning is necessary. Specifically, we adopt GPT-4o [13] as a pre-critic model to judge the rightness of the operations in the collected samples, and we retain the samples for which the annotated scores are consistent with the judgment by GPT-4o. Finally, we denote the collected samples as De Decoction {Ee, an lns 8n))
3.2.2 GUI Critiques Generation
The Decaction collected in the above section still lacks the critique e that elucidates why an action is considered correct incorrect. In this section, resort to Chain-of-Thought (CoT) technique enabling MLLMs to handle the challenging critique generation.
Progressive CoT Paradigm. We design progressive CoT paradigm to helps MLLMs to perform deliberate, structured, and analytical thinking about the GUI operations. Specifi- cally, our paradigm contains three parts <thinking>... ... </thinnkig>> <score>... .ccorr>..<<sccor>>>> <suggestion>... </suggestion> which respectively corresponding to intermediate logical thought t, correctness score I, and suggestion The reasoming contains the content of critique In particular, the logical reasoning in the <thinking> field is composed of several essential components:
1. Observation: Analyze the screenshot's state, ignoring user instructions.
2. Possible Result: Speculate the most possible result of the action.
3. Critique: Assess correctness of the action with reasoning.
Observation enhances the model's 's perceptual acuity by demanding comprehensive understanding of spatial and contextual elements present in GUI. Possible Result boosts foresight by prompting the model to predict outcomes from current observations. Finally, Critique involves critical analysis of actions to verify their correctness.
Reasoning Bootstrapping. Following the above format, leverage current MLLMs generate CoT reasoning progress. However, performing reverse annotation of CoT conditioned on ground- score / and suggestion is harmful. This is because that the MLLMs may overly depend pre-known annotations, potentially biasing the thought process toward these outcomes rather than reflecting the actual critique reasoning sequence. Therefore, we propose a reasoning bootstrapping strategy to generate high-quality thoughts without prior knowledge of the ground-truth I and s. Specifically, merely provided with environmental states and action candidates a, forces the model simulate genuine critic reasoning process for mar times as follows:
B(caa..e) (F,, 1, ) max T(c,a), Select (Z,, T, 8:) (1 = 1) A =s,
(1)
where ने is excellent MILM and / denotes the generated thought. We select reasoning outputs containing the correctness score and suggestion that match with annotations to compile D Da = {EEm, amy bms Sm, tm)) m==1
M
5

---

Finally, the GUI-Critic- dataset is constructed by aggregating the collected datasets, which can be represented as D. D acctiiion Ue_ennt For the annotation of reasoning process, we construct it with the pipeline introduced in Section 3.2.1.
dataset that does not require the
3.3 GUI-Critic-RI
Constructing a pre-critic model for GUI automation is not easy since it requires thorough under- standing of GUI knowledge, multimodal processing, and logical reasoning abilities. In this section, cultivate the deliberative analysis ability of the pre-critic model for complex GUT scenarios, propose a Suggestion-aware Group Relative Policy Optimization (S-GRPO) strategy for our GUI- first employ Reinforcement Fine-Tuning Initialization (RFT cold-start) to
Critic-R1. 1. Specifically, stabilize the model's reasoning process. Then, the S-GRPO is adopted to further enhance the model's ability for GUI pre-critic.
3.3.1 RFT Cold-Start
We initiate model training with Reinforced Fine-Tuning (RFT) on collected GUI-Critic dataset De:
Crft R (c.a,,.a)-D,
(t.a.Ia..)=PPeeat ll - log( (1, 8, āle,a)) a)).
(2)
The cold-start stage distills GUI knowledge from human annotations (i.e. correct operations in actioo)) and distill reasoning experience from existing MLMs (i.e., progressive reasoning processes in Dn De_ ), thereby equipping the model with foundational capabilities needed for generating operation critiques and valid feedbacks.
3.3.2 Suggestion-aware Group Relative Policy Optimization
After the RFT cold-start, we enable the pre-critic model to self- self-improve its reasoning ability via online reinforcement learning. Standard GRPO [8] approach samples group of generated outputs for each input from the policy model, where each output includes CoT thought and an answer. Subsequently, the model is encouraged favor better answers with a high reward value within the group. Since the pre-critic model is required to output additional valuable critiques and suggestions of the operation, only employing the typical reward that focus on the correctness of format and answer is not enough. Therefore, introduce suggestion-awar GRPO method, incorporating a novel suggestion reward specifically designed for GUI pre-critic.
Suggestion Reward. Given the suggestion annotation s', the suggestion reward r, evaluates the model's ' constructive outputs follows:
r=(o) [simitar (s,
(3)
where (1, c) is the output of pre-critic model Teritie- [limitar(() calculates the similarity between the inputs leveraging large language models, and output a binary judgment. Based on Eq. (3), rs provides quantitative and objective metric assess the correctness of suggestions, thereby facilitating more precise evaluations of Terstie efficacy. Moreover, this direct feedback enables models to learn from their outputs iteratively, refining their progressive CoT reasoning process to produce more accurate action critique and corrective suggestions.
Format and Accuracy Reward. Additionally, adopt the rewards introduced in DeepSeek-RI [8]: format reward (o) and accuracy reward ra (o). The former evaluates whether the outputs follow the structure described in Section 3.2.2. The latter evaluates whether the generated correctness score align with ground-truth. These rewards imposes constraints solely on the format and final answer, leaving the thought process unregulated, resulting in stimulating the model's 's intrinsic ability to reason.
Optmization. The final reward function r(o) can be defined as: ro) = x (r) + r. + (1- A ),) (o), where Xy and X, adjust the importance of the format and suggestion rewards. For each input (c, a), we first sample a group of generated outputs G = {01, 02, ... .G6 from the frozen policy model Toree where each output (c, I, s) includes critique c, score and the suggestion s. Then we maximize the following objective and optimizes the critic model 7 Teritio"
LORPO
1
Σ
min Meritic a) A., clip Tew (a. a)aa) Ta (a a,
Teritic e,a)) 11-,, L+E)A.) -3 D (Teriti || Tret),
G
(4)

---

where and | are the PPO clipping hyperparameters and the coefficient controlling the Kull-
(KL) penalty, respectively. (Frriti Teer) = sello]]e.a)
(oi log Fericie(o:]f..0
is the KL divergence. The relative advantage for the i-th response is computed by normalizing the rewards across the group: A, = [fox)-mentl
sad su([[(())),..0000)))
4 Experiment
4.1 Experiment Settings
Dataset and Benchmark. (1) GUI-Critic-Train The data in GUI-Critic-Train were sourced from publicly available GUI operation datasets, including AITZ [54], AMEX [3], Odyssey [18], and AITW [28]. We utilize VLL-2.5-VL-7B [2] generate negative operations, which were simply deemed incorrect based on rule-based criteria. GPT-4o [13] and Qwen2.5-VL-72B [2] are utilized to generate the Chain of Thought (CoT) processes for De_ outlined in Section 3.2.2. Consequently, GUI-Critic-Traii dataset comprises approximately 11k entries, with 6k annotated by high-quality Chain-of- Thought (CoT) processes. (2) GUI-Critic-Tess To comprehensively evaluate the pre- critic capabilities of MLLMs, we established three main benchmark settings in GUI-Critic-Tess Test: Mobile-Instructie Generalization (GUI-I), Mobile-Scenarte Generalization (GUI-S), and Web- Scenario Generalization (GUI-W). Specifically, GUI-I test data are sourced from the AMEX [3], with instructions different from those in the training set. GUI-S test data are drawn from the Odyssey [18], comprising mobile applications that differ from those seen in the training set. Besides, shifting from mobile to web, GUI-Web contains the samples about the web automation, which are randomly sampled from the GUICourse [4]. Each of the settings, GUI-I, GUI-S, and GUI-Web, is manually annotated to ensure label accuracy, resulting in dataset sizes of 656, 114, and 418, respectively. We present critic accuracy and suggestion accuracy as the metrics. The former reflects the ability assess the correctness of GUI operations, while the latter reflects the suggestion quality by quantifying the similarity between the generated suggestion and the annotation, which is calculated by prompting the Qwen2.5-VL VL-72BB [2].
Implementation Details. We employ Qwen2.5-VL-7B [2] the backbone. For the RFT cold-start, it is conducted for one epoch on mobile scenarios (GUI-I and GUI-S) and two epochs on web scenarios (GUL-W) via supervised fine-tuning (SFT). Next, train the model the D action by S-GRPO for 10 epochs, with a learning rate of 3e 36-6 and batch size of 128. The suggestion and format reward weights ()., Xy) are both set 0.1, while the group size is set to 6. In accordance with the methodology outlined in [57], the KL divergence coefficient is set to 1e by default. All experiments are conducted on eight NVIDIA A100 Tensor Core GPUs.
4.2 Comparision Results
To analyze the performance of our GUI-Critic-R) comprchensively, we construct experiments from both static and dynamic aspects. Firstly, we evaluate the model's ability to determine operational correctness and suggest the corrective operation on our static GUI- GUI-Critic-Tess benchmark. Addi- tionally, crucial application of GUI-Critic lies in its functionality a pre-critic model within the dynamic GUI automation processes. To this end, we implement online evaluation experiments on AndroidWorld [27] to validate the effectiveness of our approach.
4.2.1 Static Evaluation
Table illustrates the quantitative results of our GUI-Critic-RI and a variety of baselines on GUI- Critic-Tes Test benchmark. It showcases that GUI-Critic-RR R1 performs competitively on different scenarios compared closed- and open-source MLMs both critic accuracy and suggestion accuracy. For close source MLMs, Claude-3.5 achieves best performance on most settings. Despite the capabilities of close source MLMs, our GUI-Critic-RR achieves state-of-the-art (SoTA) performance on the GUI-I test dataset. Compared to the base model, GUI-Critic-R achieved a significant improvement in critic accuracy from 54.88%, to 69.05%. Compared to the excellent GPT-4o, our model also achieves a 3.19% critic accuracy improvement and 11.89% advantage in suggestion accuracy. When faced with GUI-S, the novel application scenarios containing complex cross- applicaapo instructions from Odyssey [18], model demonstrated commendable generalization capabilities well. Although
7

---

Table 1: Static evaluation performance comparison of closed-soures and open-source MLLMs on our GUI-Critic-Tess, including three different settings: GUI-I, GUI-S, and GUI-W.
Model
GUI-1
GUI-S
GUI-W
Critic Accuracy
Suggestion Accuracy
Critic
Suggestion Accuracy
Critic Accuracy
Suggestion Accuracy
Accuracy
Close Source MLLMs
Claude-3.5
67.26
40.71
64.27
46.11
65.55
37.64
GPT-4o
66.01
40.54
62.28
33.33
68.45
28.27
Gemini-2.0-Flash
42.98
64.91
38.59
62.85
38.78
Open Source MLMs
Deepscek-V1.2-7BB [41]
44.36
0.00
43.85
0.00
10.28
0.00
Intern [5]
53.96
19.35
52.63
14.91
54.67
24.53
IntermVL.2.588BB-MB [5]
51.06
20.42
51.75
19.30
55.37
24.06
Qwen2.0-VL--B [37]
52.59
21.04
54.42
19.30
53.50
38.78
Qwen2.0-VL-72BBI1 [37]
54.27
30.03
53.51
29.80
51.86
21.96
Qwen2.5-VL-7B [2]
54.88
43.14
57.02
37.72
59.11
36.21
Qwen2.5-VL-72B [2]
56.40
49.08
59.65
38.79
60.05
38.79
Ours (GUT-Critic-RI)
69.20
52.43
58.77
47.37
63.08
39.48
(Ours Qwen2.5-VL-7B)
+14.32
+9.29
+1.75
+9.65
+3.97
+3.27
our model achieves a relatively insignificant improvement in critic accuracy (1.75%), it performs 9.65% advantage in suggestion accuracy. It demonstrates that model has robust understanding of operations even in a new environment. In the web scenarios, where domain differences are more significant, our model exhibits commendable performance, indicating that only enhances the ability to determine the operation correctness but also generates effective suggestions. These results validate the superiority and robustness of our proposed S-GRPO.
4.2.2 Dynamic Evaluation
Table 2: Dynamic evaluation results on the AndroidWorld [27] benchmark.
We further evaluate our model on the AndroidWorld [27] benchmark, which provides a live Android emulator and tasks across mobile apps. Specifically, this
Model
SR(() EAR(A)
platform, the GUI agent can perform operations ulated Android phone to attain human instructions, and the results are evaluated automatically. Our pre-critic model can serve as an error diagnosis module in the framework in plug-and-pla -lay manner. For fair comparison, conduct both post-operative and pre-operative critic with existing MILMs as the baseline. The results in Table 2 illustrate that GUI-Critic-R achieves the best success rate the benchmark, verifying the ability of error correction and suggestion of our model. We observe that both post- and pre-critic can improve the performance avoid some potential mistakes, and post-critic can
em-
Baseline
22.4
Post- Critic
GPT-4o
25.0
18.3
Pre-Critic
Qwen2.5-vl-7B [2]
20.3
21.8
Dwen2.5-v1-72B [2]
23.2
24.4
GPT-4o [13]
22.4
26.1
Ours
27.6
31.8
of the GUT agent, indicating that pre-critic also remedy some accomplished errors. Despite the capabilities of GPT--4o, we find that it can sometimes produce inaccurate critiques and suggestions mislead the agent when performing the pre-critic, primarily due to its insufficient common-sense knowledge of the GUI. GPT-4o outperforms the prior-critic on the post-critic because the prior-critic requires the model to predict the potential outcomes of GUI operations, which GPT-4o lacks the common- sensons-eeseeee reasoning ability. Furthermore, we evaluated the efficiency of GUI task execution across different approaches. We introduced metric called Efficiency Advantage Rate (EAR) for fair comparison, which measures the proportion that the "baseline + critic" model achieves an efficiency advantage (fewer steps) than the baseline, for tasks where they achieve consistent results. The results shown in Table 2 reveal that our model tends to complete tasks fewer steps, thereby demonstrating its efficiency. contrast, post-critic methods typically require greater number of steps.

---

Table 3: Ablation study on the dataset collec- tion pipeline.
Table 4: Ablation study on training strategies.
S-GRPO
GUI-I
GUI-S
GUI-I
GUI-S
RFT
Model
Critic Accuracy
Suggestion Accuracy
Critic Accuracy
Suggestion Accuracy
Suggestion Accuracy Accuracy
Critic
Critic Accuracy
Suggestion Accuracy
Tf
/
63.16
45.61
55.26
34.21
w/o NOS
50.46
1.22
50.30
5.26
X
67.98
43.44
54.38
39.47
w/o DF
67.23
49.54
54.39
42.98
X
J
v
69.05
49.24
56,14
42.10
w/o GCG
67.84
51.22
56.14
42.11
/
J
<
66.01
47.71
57.89
40.35
Ours
69.20
52.43
58.77
47.37
V
J
J
J
69.20
52,43
58.77
47.37
Task Instruction: How many atfendees were present in the meeting filedd 'Enplaye Performance Exdlustion in the Zaplin opp?
Criie Ausracy - Segyatien Aueraay
Suggettion Reward Weight
Group Size
Stop
Accurccy (59)
-0-
Scroil
Varoloua perfomanee bokktt
60
6.33 mmee
Accuraay
Sopu
50
$132
Ortigee
Step augeetic
(
Step
Figure Analysis of suggestion reward weight X and Group Size on the GUI-I setting.
Pa 25 etherdees
Cldc soarch byy
Figure 4: A case of pre-critic on GUI automation.
4.3 Ablation Study
Analysis of Dataset Collection Pipeline. We first conduct ablation studies for the proposed data collection pipeline verify the contributions of three key components GUT-I and GUI-S settings. Corresponding to Table 3, we first substitute Negative Operations Sampling (NOS) with a strategy of random decision replacements acquire negative samples. It underscores that negative samples sampled by random substitution are too naive to undermine the model's capability to detect errors and suggest corrections. Secondly, we substitute the data filtering (DF) phase with a random sample selection approach. Without DF, the quality of training samples may be compromised, potentially degrading the effectiveness of the training process. Finally, we omit the GUI Critique Generation (GCG) process by utilizing only D De_oction in the RFT cold-start stage. The absence of GCG leads to a 1.36% decline in critic accuracy on GUI-I, highlighting its crucial contribution to the model's ' generalization and cognitive capabilities. These findings collectively emphasize the pivotal importance of our data collection process, establishing as an integrated mechanism for boosting model training and operational efficiency.
Analysis of Training Strategy. In order to analyze the impact of training strategies on model performance, we conduct ablation experiments focusing on the RFT cold-start and reward components in the GRPO stage. As depicted in Table 4, the results in the first and third lines indicate that the utilization of RFT cold-start provided robust initial boost to the model's decision decision-making capabilities, and S- GRRP further increases the ability. Towards the second line, we observe that employing GRPO alone was insufficient to activate the model's GUI-Critic abilities, as the model does not yet possess the basic GUI critic ability, making it difficult to produce high-quality outputs when sampling randomly. In the fourth line, we ablate the suggestion reward, which led to an obvious decrease in suggestion accuracy, indicating this reward is necessary for the model to suggest correct GUI operations. These observations underscore the crucial significance of the RFT cold-start and the proposed suggestion reward in S-GRPO, incrementally enhancing the capability of our GUI-Critic-R,
Analysis of parameters. Figure 3 illustrates the impact of different suggestion reward weight X, or the group size the R-GRPO phase. Note that when varying As the weights for ra and ry are maintained at ratio of 4:1, ensuring that the total sum of weights remains equal to 1. By adjusting A. we observe that excessively low values inadequately constrain the suggestion, while excessively high values may compromise accuracy. Consequently, value of 0.1 was selected as an optimal parameter. Additionally, the size of the group plays a crucial role in balancing performance with resource utilization. Therefore, compromise was made by selecting group size of 6.

---

Task Instruction: Take one video.
Step
Step
Stop
Step
Step
Step
Swipe
Change Mode
Click 'Start'
Click 'Stop'
Click 'Start' Finish
APP
LIST
7
<thinking
"OOtvervatoon scrcenshot camona upp
visible
typically used for recording vidces. The sereen displays reem scene, indicating the camera ready to capture. ** "Pssiible Result**; Clicking the video betton will likely recording video, the standard function such
button in comera apps.
instruction necessitates taking video, and one has already been produced, starting another recording is unnceessary.
"Crittque**
decision
Video (540,
<thinkings
<spore- incorrect <iscore
<suggestion>>
The approprite decision is to terminate the task.
siuggestion
Finish
Figure 5: An instance of GUI-Critic-RI avoiding redundant action within GUI automation through a pre-operative critique approach.
4.4 Case Study
Figure illustrates an example drawn from the AndroidWorld benchmark [27], demonstrating how our model guides the GUT agent to achieve a correct and effective trajectory. Specifically, the agent tasked with finding file in the Joplin app but encounters interface without the target file visible, prompting it to mistakenly consider rolling back. Our model suggests clicking the search box to locate the target file, thereby assisting the agent in successfully completing the task.
In Figure 5, depict another example of pre-operative critic for GUI automation. The agent successfully navigates steps through 6, initiating the camera app and completing video recording. Nonetheless, step 7, the agent erroneously decides to press the record button again. This action is incorrect, as the instruction clearly stipulates recording only one video. Given that the agent has fulfilled this requirement, the task should be stopped rather than continuing the recording. Our model precisely delineates the state of the current interface and, based on historical operation, identifies the agent's decision as redundant. Consequently, our model advises the agent terminate the task at this step.
Conclusion
In this paper, we introduce pre-operative critic mechanism aimed at performing error diagnosis in GUI Automation, prior executing actions within interactive environments. To develop our pre-critic model GUI-Critic-R1, we propose a Suggestion-awand (S-GRPO) strategy with novel suggestion reward. It is designed to ensure effective corrective discrimination for operations and provides reliable feedback for improvement. Futhermore, We develop a reasoning-bootstrenee bootstrppptttnggggggggoooo based data collection pipeline to construct a GUI-Critic-Trans and GUI-Critic- Test dataset, addressing existing gaps in GUI critic data. Through extensive static and dynamic experiments, we demonstrate the the efficacy our approach across diverse GUT scenarios, highlighting its potential to significantly enhance the success rate of automated tasks.
Gradient Policy Optimization
10

---

Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grants 62322212. U23A20387, U25B2070), the Beijing Natural Science Foundation (No. L221013), and the CAS Project for Young Scientists in Basic Research (YSBR-116)
References
[1] Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang. Ang Li, and Xin Eric Wang. Agent s2: compositional generalist-specialis framework for computer use agents. arXiv preprint arXiv:2504. 00906, 2025.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 2rXiv:2502.1392: 2025.
[3] Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents. arXiv preprint arXiv:2407.1 174990 2024.
[4] Wentong Chen. Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu. Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to 2024.
versatile gui agents. arXiv preprint arXiv:2406.1133
[5] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 24.0224/,. 2024.
[6] Zi-Yi Dou, Cheng-Fu Yang, Xueqing Wu, Kai-Wei Chang, and Nanyun Peng. Reflection--reiffrreed self-training for language agents. arXiv preprint 2024.
Re-rest:
[7] Dayuan Fu, Keging He, Yejie Wang, Wentao Hong, Zhuoma Gongque, Weihao Zeng. Wei Wang, Jingang Wang. Xunliang Cai, and Weiran Xu. Agentrefine: Enhancing agent generalization through refinement tuning. arXiv preprint arXiv:2501.0 702, 2025.
[8] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint 1..Xiv:2501.1294: 2025.
[9] Priyanshu Gupta, Shashank Kirtania, Ananya Singha, Sumit Gulwani, Arjun Radhakrishna, Sherry Shi, and Gustavo Soares. Metareflection: Learning instructions for language agents using past reflections. arXiv preprint arXiv:2405. 2024.
[10] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281-14290. 2024.
[11] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprini 20.0v 09800 2023.
[12] Wenxuan Huang. Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06743 2025.
[13] Aaron Hurst. Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arXiv preprint arXiv:2410.21276 2024.
[14] Noman Islam, Zeeshan Islam, and Nazia Noor. A survey on optical character recognition system. arXiv preprint arXiv: 17 2017.
[15] Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent v2: Advanced agent for flexible mobile interactions, arXiv preprint arXix:2408. 11844, 2024.
11

---

[16] Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, et al. Pe-agent: A hierarchical multi multi-agent collaboration framework for complex task automation on pe. arXiv preprint arXiv:2502. /4282, 2025.
[17] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0/785 2025.
[18] Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang. Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406. 08451, 2024.
[19] Jie Ma, Pinghui Wang, Dechen Kong, Zewei Wang.
Jun Liu, Hongbin Pei, and Junzhou Zhao. Robust visual question answering: Datasets, methods, and future challenges. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[20] Xinbei Ma, Zhuosheng Zhang, and Hai Zhao. Coco-agent: A comprehensive cognitive milm agent for smartphone gui automation. arXiv preprini arXiv:2402.1 / 1941, 2024.
[21] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534 2023.
[22] Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebac7, and Jan Leike. Llm critics help catch Ilm bugs. arXiv preprint arXiv:2407.0021 2024.
[23] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang. Junjun He, Kaipeng Zhang, et al. Mm-eureka: Exploring visual aha moment with rule- large-scale reinforcement learning. arXiv preprint arXiv:2503.07365, 2025.
[24] Jiay Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.06474, 2024.
[25] Yingzhe Peng, Gongrui Zhang. Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprini urXiv:2503.075.5 2025.
[26] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprini 2r0iv:2501. /2326, 2025.
[27] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405. /4573, 2024.
[28] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36:59708 -59288, 2023.
[29] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8652. 2023.
[30] Yunpeng Song. Yiheng Bian, Yongtao Tang, Guiyu Ma, and Zhongmin Cai. Visiontasker: Mo- bile task automation using vision based ui understanding and 11m task planning. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1-17, 2024.
[31] Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. Advances in neural information processing systems, 36:58202-58255, 2023.
[32] Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, et al. A survey on (m) Ilm-based gui agents. arXiv preprint arXiv:2504.133. 2025.
12

---

[33] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changju Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv: (0w::2501//299, 2025.
[34] Gladys Tyen, Hassan Mansoor, Victor Cärbune, Peter Chen, and Tony Mak. Lims cannot find reasoning errors, but can correct them given the error location. arXiv preprint (rXiv:23/1.085// 2023.
[35] Haoyu Wang, Tao Li, Zhiwei Deng, Dan Roth, and Yang Li. Devil's advocate: Anticipatory reflection for Ilm agents. arXiv preprint arXiv:2405. 16334, 2024.
[36] Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang. Fei Huang, and Jitao Sang. Mobile-agent--2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint wxxiv24466010044 2024.
[37] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing rision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[38] Zhenhailong Wang. Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang. and Heng Ji. Mobile-agent-ee Self-evolving mobile assistant for complex tasks. arXiv preprint arXiv:2501.11733 2025.
[39] Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang. Yunhao Liu, Yagin Zhang, and Yunxin Liu. Autodroid: LIm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 543-557,2003
[40] Hao Wen, Hongming Wang, Jiaxuan Liu, and Yuanchun Li. Droidbot-gpt: Gpt-powered ui automation for android. arXiv preprint arXiv:2304.07066 2023.
[41] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang. Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang. Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong. Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang. and Chong Ruan. Deepseek-v12: Mfxture-of-experts vision-language models for advanced multimodal understanding, 2024.
[42] Yufei Xiang, Yiqun Shen, Yeqin Zhang, and Nguyen Cam-Tu. Tu. Retrospex: Language agent meets offline reinforcement learning critic. In Proceedings of the 2024 Conference Empirical Methods in Natural Lunguage Processing, pages 4650-4666, 2024.
[43] Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. arXiv preprint
1rX0022410.02712, 2024.
[44] Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. 2024.
arXiv preprint arXiv:2412.04444
[45] An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07566) 2023.
[46] Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng. Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615 2025.
[47] Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. Agent-r: Train- ing language model agents to reflect via iterative self-training. arXiv preprint arXiv:2501.11425, 2025.
[48] Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, et al. Large language model-brained gui agents: A survey, arXiv preprint arXiv:2411.188 18279, 2024.
[49] Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction. arXiv preprint arXiv:2402.07939, 2024.
13

---

[50] Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents smartphone users. arXiv preprint arXiv:2312.13771, 2023.
[51] Di Zhang, Junxian Li, Jingdi Lei, Xunzhi Wang, Yujie Liu, Zonglin Yang, Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, et al. Critic-v: Vlm critics help catch vlm errors in multimodal reasoning. arXiv preprini arXiv:241 1.1220,, 2024.
[52] Jiaqi Zhang, Chen Gao, Liyuan Zhang, Yong Li, and Hongzhi Yin. Smartagent: Chain-of-user- thought for embodied personalized agent in cyber world. arXiv preprint arXiv:4412.07472. 2024.
[53] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. /2937, 2025.
[54] Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao. Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-aactionttie arXiv:2403.02713, 2024.
for gui agents. arXiv preprint
[55] Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. Agent- pro: Learning to evolve via policy-level reflection and optimization. arXiv preprint arXiv:2402. /7574, 2024.
[56] Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang. Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Vem: Environment-free exploration for training gui agent with value environment model. arXiv preprint arXiv:2502.18906 2025.
[57] Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi muuti-mmdallty rl training framework. https: /tops://giii thub. EasyR1, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's contributions and scope.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in the Appendix.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have provided the necessary information for reproduction Section ??.
5. Open access to data and code
Question: Does the paper provide open to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
The codes for reproducing all experimental results will be released as the paper is accepted NeurIPS.
Justification: Justification:
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have provided the training and test details necessary understand the results Section 4.1.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide the compute workers details Section 4.1, and the amount of compute are illustrated in Appendix.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eurrip.pee httppppcc..ppppocecccccccccceeed
15

---

Answer: [Yes]
Justification: The research conforms with the NeurlPS Code of Ethics in every respect.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper prioritizes discussing technical methodologies and results rather than societal impacts.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper exploring GUI automation on public datasets poses no such risks.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly credited the used assets. All the datasets introduced this paper are available for free to researchers for non-commercial use.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The codes will be released once upon the paper's 's acceptance.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes].
Justification: The paper uses LLM and describes in detail the type of model used (Sec 4.1) and prompt in Appendix.
16

---

Appendix
A Limitations and Future Works
In this paper, we explore the pre-operative critic mechanism for GUI automation enhance success rates and decrease the number of operational steps. Future endeavors have the potential to extend our method to lighter models, such as Qwen2.5-VL-3BB, critic performance. Furthermore, our algorithm is based on single-step GUI visual information and semantic operation history.
achieve more efficient and improved
Our approach can be enhanced by integrating the trajectory-level critic in future studies, which involves taking a sequence of screenshots input. It has the potential to provide more comprehensive insights.
B More Dynamic Evaluation
In this paper, we conduct the dynamic evaluation on AndroidWorld [27] and illustrate the experiment results the main text (Section 4.2.2). this section, we introduce more details of the implementation of the experiments and more experimental results.
B.1 Supplementary Implementation Detials
We adopt the M3A (Multimodal Autonomous Agent for Android) framework introduced in Android- World [27] as the backbone. In the M3A framework, the decision agent is provided with a list of available action types, guidelines for operating the phone, and list of UI elements derived from the leaf nodes of the Android accessibility tree. The agent receives the current screenshot along with Set-of-Mark annotated screenshot, which includes bounding boxes with numeric labels the top-left corner of each UI element. At this stage, the agent attempts to execute the generated action, referencing specific marks. Prior action execution, insert pre-operative critic evaluate whether the agent's 's proposed action is conducive to achieving the instruction. If the action is deemed correct, proceeds to execution as normal. Conversely, if the action is considered incorrect, we input the critic model's analytical recommendations to the decision agent, prompting it to reassess and formulate a new decision. After action execution, another agent adopted deliver a concise summary following the execution of the action. We leverage such semantic summaries to serve as a record of the action history. We follow the Androidworld configure the UI element detection procedure and the definition of the action space.
B.2 Supplementary Experiments
We extend the dynamic evaluation experiment in- corporate GPT-4o [13] as the decision agent. As evidenced in Table 5, employing GPT-4o [13] the decision-making agent yields performance benefits over Qwen2.5-ViL LL-72B [2], underscoring GPT-4o's robust perception and reasoning capabilities. The pre- operative critie improves accuracy and exemplifies the efficacy of the baseline in detecting erroneous decisions and offering constructive improvement sug- gestions. In comparison with other pre-operative models, our GUI-Critic-R exhibits superior perfor- mance outcomes. When utilizing GPT-4o as the base- line, the advantage of the pre-critic in terms of the Efficient Advantage Rate (EAR) becomes more pro- nounced. GPT-4o tends to engage excessive explo- ration during GUI automation, often opting to attempt further actions rather than terminating early, thereby resulting in a greater number of steps. In contrast, the pre-critic effectively anticipates erroneous or redundant exploration, thereby reducing unnecessary steps and manifesting superior EAR metric.
Table 5: Supplementary dynamic evaluation results on the AndroidWorid [27] benchmark with GPT-4o [13] as the baseline.
Model
SR(1) EAR(t)
Baselines
Qwen2.5-VIL-22 [2]
22.4
GPT-4o [13]
23.9
GPT Post-rris
GPT-4o [13]
26.4
31.6
GPT40+
Qwen2,5-VL--BB [2]
16.9
32.4
Qwen2.5-VL-72B [2]
25.9
36.8
GPT-4o [13]
24.0
43.1
Ours
29.4
46.2
17

---

Task Instruction:
Delete all but one of any expenses in expense that are exact duplicates, ensuring at least one instance of each unique expense remains.
Step
Step 2
Stop
Step 4
Step 5
Step 6
Step
Click 'Pro Expense APP
Click Menu'
Click 'Statistic'
Swipe up
Swipe up
Click Item
Click "Delete'
Click 'Fapense Logg
Step 5
Click 'Statistic'
Step g
Step 9
<thinking
Observation The screenshot shows Pro Expense app's with options like "Home," " "Expense Logs." "Sttissiccs"" "Backup," " "I "Feedhack," " "About," and "Settings." "Stutisries" oprion highlighned, indicating might the target interaction, * Possibbee Result* Clicking on "Statisties" will likely open new screen or related expenses, which may directly help identifying duplicates
6..0
to statistical
delction.
*Critique** The decision to click on "Statistics" incorrect because it does not align with the usar' instruction delete duplicate expenses. The needs the experse logs similar functionality to identify duplicates before proceeding with deletion. The current action does contribate the task's completion. <>thinking>
<score-Tneoe cecore--ncrroo--ssooooo
suugestioon=
action would click "Expense Lugs" view detailed expense records. </suggestion
Click 'Confirm'
Finish
Click 'Expense Log'
Figure 6: An instance of GUI-CriticRRI correcting an erroneous decision within GUI automation through a pre-operative critique approach.
B.3 Additional Case Study
Figure illustrates an operational process of GUI automation with pre-operative critic by our GUI- Cri In this example, the GUI agent initially demonstrates correct behavior by opening the Pro Expense app and accessing the menu. However, at step 5, the agent erroneously decides to click the "statistics" button, which is incorrect. Our GUI-Critic-RR R1 model successfully identifies this as an incorrect action and analyzes the reason behind the error. The model determines that selecting this button would navigate new interface displaying expense statistics, which is irrelevant to the instruction requiring the removal of duplicate expenses. Furthermore, our model provides suggestion for correction: click on "Expense Logs" view detailed expense records.
C GUI-Critic Dataset
To construct the GUI-Critie-Trase and GUI-Critic-Tess datasets, we extract step-level data from pub- licly available GUI operation datasets, encompassing user instructions, current interface screenshots, text-based operative history, and corresponding correct actions. the following section, we provide an overview of the datasets employed in our study.
18

---

GUI-Critie-Trann w/o CoT
GUI-Critic-Train CoT
4000
possitive
4000
possitive
GUI-Critic-Train
3500
negative
3500
negative
2500
2500
5455
1500
1500
GUll-Crite- Trin CoT
CUl-Criéc-Tman wi CoT
500
500
AMEX
AITZ
AITW
Odyssey
AMEX
AITZ
AITW
Odyssey
Figure 7: Illustration the data composition for the GUI-Critic- -rain dataset. The left displays the proportion of data without Chain-of-T ofoThh (CoT) annotations versus with CoT. The central part shows the sources of data without CoT annotations, along with the ratio of correct to incorrect actions. The right section depicts the sources of data with CoT annotations, also detailing the ratio of correct incorrect actions.
C.1 Public Datasets
Firstly, review the publicly available datasets employed in research.
Android in the Wild (AITW) [28] dataset comprises human demonstrations of device interactions, encompassing both screen captures and actions, alongside corresponding natural language instructions. includes substantial collection of 715,000 episodes covering 30,000 unique instructions, across four versions of the Android operating system (v10-13) and eight different device models ranging from the Pixel XL to Pixel 6, each with distinct sereen resolutions. The dataset involves complex multi-step tasks necessitating nuanced semantic understanding of both language and visual context.
CeddoiddIn-Thh-ZZee (AITZ) [54] dataset contains 18,643 screen-action pairs together with chain- of-action-thouut thought annotations, spanning over 70 Android apps, coupled with 4x useful annotations compared with action coordinate labels only. Based on the screen episodes from AITW [28], the authors generate candidate answers for the screen descriptions, action thinkings and next action descriptions. These candidates are further validated and refined by human to guarantee alignment with the screenshots.
GUT Odyssey [18] is comprehensive dataset for training and evaluating cross-app navigation agents. GUI Odyssey comprises 7735 episodes, meticulously curated from different mobile devices such as Pixel Pro and Tablet. It encompasses types of cross-app navigation tasks spanning from general system tool use to media entertainment, requiring navigating through 201 different apps and 1399 app combos from various fields such as video, music, reading
Android Multi-annotation EXpo (AMEX) [3] a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents. Their capabilities of completing complex tasks by directly interacting with the graphical user interface (GUI) on mobile devices are trained and evaluated with the proposed dataset. AMEX comprises over 104K high-resolution screenshots from 110 popular mobile applications, which are annotated at multiple levels. AMEX includes three levels of annotations: GUI interactive element grounding, GUI screen and element functionality descriptions. and complex natural language instructions, each averaging 13 steps with stepwise GUT-action chains.
GUICourse [4]. In this paper, the author constructed GUIAct. GUI navigation dataset in website and android scenarios for enhancing VLMs' knowledge of GUI systems, including 67k single-step and 15k multi-step action instructions.
C.2 GUI-Critic Dataset Components
In this subsection, we delineate the origins, constituent parts, and foundational principles of the construction of our GUI-Critic- Train and GUI-Critic-TE -est datasets.
19

---

Instruction: Open Wego. Find multi-citty flights from Amsterdam to Athens on May 5, from Athens Cairo on May 8 and from Cairo back to Amstendam on May 12, I adult. First class, Sort by price and filter only book Wego. price must be under USS3000 set as May 12.
Screenshot marked with the candidate action.
wo Pights
Sup Top the "Wego Flights upp open Step on back search screes, Step 3: Tap on the "MULTLCITT oprion change the search type, Step 4: Tap on the fist city change to Ameoenaan Step Type Ammsterdan into the scarch ficki Step Tap the Ammterraai option sclect it Step Top GGV to change to 'Athens SSep type 'Athens'
History:
flight
4011
arrival city, Step 'Atheus, to select it as the destinatonn city. Step 10: Tap oa the destination city the second segusest and change Madiid to Cain. Step II: Swips up is navign: to the morth of May and locase May 8 the calendar Step 12: Top on May 6, 2014, to select it the departare date from Athens Cairo. Step 13: second segment. Step 14: Sclect Cairo. Egypt Boa list popular cities. Step
search to
on
agcion,
change the destination from Madrid to Cairo the
Calm,, Egypt' the destination.
1 Acd oin right
Action: ("action": "click", "coordinate": [639, 836])
Addes 2 onidee
Generated CoTT process:
<thinking?
Clars
*Obbeeraatoon"* The screenshot is of the "Flights Search" interface within the Wego app. The search type to "MULTI- CITY. The current itinerary includes the follewing segments:: 1. A.msserdaam (AMS) to Athens (ATH), departing May 5, Sunday.
2. Athens (NTH) to Cairo (CAD) departing May 8, Wednesday.
teanhh
3, Cairo (CAJ) to Rome (ROM), departnng July 12, Friday.
Cordina sAaCc
The red translucent marker highlighting the city "Cairo" in the second segment of the itinerary. Below options to add passengets, change cabin class (currently set to "Business Class"), and payment types. The history so for involves setting the multi-city search and filling in destinations (Amsterdam, Athens, Cuiro) for the first two segments. Based on the user instruction, the third flight segment should have Cairo as the departure city and Amstendam as the destination, with the departure date set to May 12.
100 10 19
Possiblee * If the agent clicks on "Cairo" as marked, it will likely allow editing Caire as the destination city for the second flight segnentt However, action redundant since city "Cairo" is already correctly assigned
sccond seguennt The focus should now shift to modifying the third flight segment to Cairo Amsterdam with the departure date set as May 12. This click neither promotes progress toward completing the intended user instruction nor aligns with the detailed requirements of modifying the third segment.
destination
"Crilique**" * This decision is incorreet. Based on the bistory, Cairo is already conrectly set as the destination for the second segment. The agent should now focus on correcting the third flight segment, currently set Cairo Rome on July 12, to reflect Cairo - Amsterdam on May 12 (per user instructions). Clicking "Cairo" in the second segment achieves no necessary action for advancing the care task.
"tthinking-
<score> Incorrect </score >
"suggestion>
Change the third segment of the itinerur: Click on "ROM" and type 'Ammterdam."
<buggestion>>
Figure 8: A data sample in GUI-Critic- Train dataset.
C.2.1 GUI-Critic-Train
Figure 7 provides a visualization of the data sources utilized in our training set. For the construction of our training dataset, we predominantly selected data from AMEX[3] due to its extensive repository of high-quality GUT operation data. The raw data encompasses decision- processes enriched with reasoning, which contributes to the reliability of the annotated correct actions. Detailed methodologies for the generation of incorrect decisions and the formulation of Slain-of-Thougee (CoT) critiques are elaborated in the main text (Section 3.2). We preserved a higher proportion of samples containing incorrect decisions to facilitate the model's 's acquisition of enhanced error correction competencies. Figure 8 illustrates a data sample with the generated CoT process in -UI-Critic-Trall dataset.
C.2.2 GUI-Critic-Test
Mobile-Instruction Generalization (GUI-I). Data in GUI-I originates from the AMEX dataset [3]. We ensure that the instructions in GUI-I are distinct from those in GUI-Critie- After human annotation, GUI-I retains 656 GUI screenshots, each accompanied by a user instruction, operational history, and a set of candidate actions.
Mobile-Scenario Generalization (GUI-S). We collect 114 test data from the Odyssey dataset [18] for GUI-I. We select the data with applications different from those in GUI-Critie-Train Specifically, the novel applications are:
1. TikTok: A popular social media application for sharing and watching short-form videos, fostering a global community of creators and viewers.
2. Chaton: An innovative application powered by intelligent conversation models, designed to enhance user interactions through Al-driven dialogues.
20

---

3. ClevCalc: A multifunctional calculator tool app that offers wide range of mathematical operations and unit conversions for everyday use.
4. Remix: A versatile audio editing application that allows users mix, cut, and enhance soundtracks with variety of editing features.
5. Redfin Houses: A comprehensive app for real estate transactions, providing listings, pricing, and market insights for prospective home buyers and sellers.
6. Tripadvisor: A travel review platform app where users can rate and explore travel destina- tions, accommodations, and restaurants worldwide.
Web-Scenario Generalization (GUI-W). contains 418 samples of web automation, which are randomly sampled from the GUICourse [4]. Although the action space in web environments differs from mobile platforms (e.g., limitations on swipe direction and the incorporation of double-tapping), the underlying operational logic largely remains consistent across both scenarios.
D
Agent Prompt
Table 6, Table 7. Table 8, and Table 9, we present the prompts for pre-critic, post-critic, action sim- ilarity evaluation, and adding critiques to the GUI agent decision process. Note that, conduct mini- mal modifications to the pre-critic prompt, resulting in the post-critic prompt. These modifications in- clude the removal of predictions concerning possible results and the adjustment of requirements from generating corrective suggestions to remedial suggestions in the <suggestion ... </suggestion>> The action similarity evaluation prompt is leveraged in calculating the suggestion reward in S-GRPO and the suggestion accuracy.
21

---

System
There is a multimodal agent that can perform a series actions on a smart device (phone or PC) to automate the completion of user instructions.
Possible actions include "click" / "left_ citcc_ccikk at (x.y) position, "long press" at (x.y) position, "swipe" from (xI.yl) to (x2.y2), "scroll" down up, "drag" from (xl.yl) to (x2.y2), "type" (text content), "back", "home", "enter" and so on. User instructions are usually complex and may include several detailed requirements. In some steps, the action decided by the mobile agent may be wrong. Now, you are a critic model used to evaluate the agent's decision. I will provide you with the following information:
1. User instruction.
2. History: The action history of the agent in the previous steps.
3. Decision: The decision of the agent for this step.
4. Image:
The screenshot before executing this action. If the action contains positional parameters (such as click and swipe), the interaction area marked with translucent red circle or red arrow. Firstly, you need understand the purpose of the decision. Pay attention analyzing the interface elements in the screenshot (such as button position, text content, etc.). there are red marks, focus on the action position. You can take appropriate account of the history information.
Then, based on the given information, carefully analyze the decision given by the agent for the current step:
I. Decision Analysis
(1). Observation: Observe the screenshot and analyze the state without considering the user's instruction.
Focus on the operable or informative elements related to the operational decision.
(2). Possible Result: Speculate the most possible result of executing this decision.
Predicts the screenshot change after the operation.
Whether to promote the progress of core lasks.
(3). Critique: Determine whether the decision correct and explain why.
Focus on historical operations.
Based the previous analysis and the history, determine if this decision supports the completion of the instruction.
Only perform actions specified in the instructions.
Home button is the correct choice for switching apps
Both clicking suggestion and Enter correct when searching.
2. Based on the above analysis, determine whether this decision "Correct" or "Incorrect".
3. Reflection: If correct, retell the action; if incorrect, suggest better action. Propose one-step action for the current observation, like click, swipe (with direction), type (with information), Home, Back, or Terminate (in 20 words).
Assess the current decision's 's correctness in the following format: <thinking>
** *Observation** Describe the screenshot.
* Possible *e Analysis from the possible result perspective.
** *Critiqu=*; Criticize why the decision correct incorrect.
</hinkkng>>
<score>
Correct or Incorrect
</score>
<suggestion>
If correct, provide a brief summary, if incorrect, suggest a better decision brielly.
</suggestion>
User
Below is the information for the current step:
1. User instruction: ( (User's instruction)
2. History: (Operation History)
Decision: (Action)
Image is the screenshot this step. <image>
Table 6: The prompt for the pre-operative critic.
22

---

System
There is multimodal agent that can perform a series of actions on a smart device (phone or PC) to automate the completion of user instructions.
Possible actions include "click" beft__liikk" at (x.y) position, "long press" at (x.y) position, "swipe" from (xl.yl) to (x2.y2), "scroll" down or up, "drag" from (xl.y1) to (x2.y2), "type" (text content). "back", "home", "enter" and so on. User instructions are usually complex and may include several detailed requirements. In some steps, the action decided by the mobile agent may be wrong. Now, you
critic model used to evaluate the agent's decision. I will provide you with the following information:
1. User instruction.
2. History: The action history of the agent in the previous steps.
3. Decision: The decision of the agent for this step.
4. Images: The screenshots before and after executing this action.
Firstly, you need to understand the purpose of the decision. Pay attention to analyzing the interface elements in the screenshot (such as button position, text content, etc.). there are red marks, focus on the action position. You can take appropriate account of the history information.
Then, based on the given information, carefully analyze the decision given by the agent for the current step:
1. Decision Analysis
(1), Observation: Observe the screenshots and analyze the state without considering the user's instruction.
Focus on the operable or informative elements related to the opcrational decision.
(2). Critique: Determine whether the decision correct and explain why.
Focus on historical operations.
Ensure compliance with each specific requirement in the instructions.
Only perform actions specified in the instructions.
Home button is the correct choice for switching apps.
Both clicking suggestion and Enter correct when searching.
2. Based on the above analysis, determine whether this decision "Correct" or "Incorrect".
3. Reflection: If correct, retell the action; if incorrect, suggest a remedial action on the screen after executing this action. Propose a one-step action for the current observation, like click, swipe (with direction), type (with information), Home, Back, or Terminate (in 20 words).
Assess the current decision's correctness in the following format: <thinking>
* *bservation Describe the screenshots.
* *Critiuu=*; Criticize why the decision correct incorrect. </thinking>
<score>
Correct or Incorrect
</score>
<suggestion>
If correct, provide a brief summary; if incorrect, suggest a remedial decision briefly.
</suggestion>
User
Below is the information for the current step:
1. User instruction:
( (User's instruction)
History:
(Operation History)
Decision:
(Action)
Images are screenshots before and after executing this action.
Table 7: The prompt for the post-operative critic.
The following two sentences describe the operation that a mobile agent needs to perform on mobile order to complete a certain user instruction: (Annotated Suggestion;
2. [Generated Suggestion
Determine whether these two sentences describe a similar action? If yes, answer 1, if not 0, explanation required.
Table 8: The prompt for evaluating the similarity between two action-descriptive sentences.
Note that your last decision may be incorrect! Please make remedial decision based on the following Critiques, and consider adhering to the Suggestion provided: (Critique)
Table 9: The prompt for integrating critique into the GUI decision-making process.
23


Reviews:
[{"review_id": "31eytcrGSt", "reviewer": "Reviewer_arvH", "summary": "This paper proposes a critic model, GUI-Critic-R1, which can detect errors and provide corrective suggestions before actually executing actions in GUI tasks. To train this model, the authors first construct a dataset consisting of successful trajectories and sampled incorrect operations. Based on this dataset, critiques are collected via rejective sampling. GUI-Critic-R1 is trained with 1 epoch of SFT, followed by 10 epochs of GRPO. As the dataset contains two supervised signals, correctness and suggestion, the paper introduces an additional suggestion reward into GRPO, alongside the accuracy and format rewards. Experiments demonstrate that GUI-Critic-R1 outperforms its base model, Qwen-7B, and is comparable to some closed-source models.", "questions": "* In Section 3.3.1, the RFT stage involves two datasets: one with CoT and one without. Are different system prompts used for these two datasets to indicate whether the response should include a CoT?\n* After the critic model outputs the correctness score, critique, and suggestion, how exactly does the agent utilize this information? If an action is judged to be incorrect, does the agent directly follow the suggested correction, or does it treat the critique and suggestion as context and re-generate a new action?\n* Typo in Table 4: the position of $r_s$ seems to be incorrect.", "limitations": "The data collection process relies on successful trajectories from public datasets and filtering with GPT-4o, which limits the scalability of the approach. Step-level correctness signals and suggestions may become unavailable when GUI tasks extend beyond the coverage of existing datasets or exceed the reasoning capabilities of GPT-4o.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength**\n* This paper presents a complete pipeline, covering both data collection and model training.\n* This paper introduces a novel perspective by making the critic model to provide not only a correctness signal but also suggestions.\n\n**Weakness**\n* The idea of a pre-operative critic is interesting, but its necessity is not fully convincing. Since the dataset already contains step-level correctness labels and suggestions, it is natural to ask why not use this data to directly train a better policy model, instead of training a separate critic. Adding a critic makes the system more complex, and it is not clear if the extra component brings more benefit than directly improving the policy. Also, in some dynamic or partially observable GUI environments, it might be hard for the critic to predict the outcome of an action without actually running it.\n* The overall technical novelty is somewhat limited. This paper mostly adapts existing RFT + RL techniques to a specific GUI automation context, with relatively limited insights for broader domains.\n* The data collection process relies on successful trajectories from public datasets and filtering with GPT-4o, which limits the scalability of the approach. Step-level correctness signals and suggestions may become unavailable when GUI tasks extend beyond the coverage of existing datasets or exceed the reasoning capabilities of GPT-4o.", "was_revised": true, "final_justification": "I am satisfied with the rebuttal.", "rebuttal": "**We greatly appreciate the insightful and constructive feedback provided. We have carefully addressed each concern below.**\n\n## **W1. Necessity of pre-critic**\nWe agree that improving the policy model can enhance GUI interaction performance. However, pre-operative critic is also necessary in GUI automation. \n\n**1. The decoupled relation between policy and critic model and the necessity of our pre-operative critic.**\n\n**(1). The policy and critic models complement the system in different dimensions.**\n\n+ Policy generates actions while critic predicts errors and provides suggestions. Both components contribute uniquely to task completion, and the combination of decision-making and safety verification creates our efficient system. \n+ According to Table 2 of main text and Table 1 of Appendix, upgrading the policy from Qwen2.5-VL-72B to GPT-4o improves success rate from 27.6% to 29.4% on AndroidWorld (using GUI-Critic-R1 as pre-critic). Similarly, enhancing the critic from Qwen2.5-VL-7B to GUI-Critic-R1 improves performance from 20.3% to 27.6% (using Qwen2.5-VL-72B as policy), showing both components' upgrades enhance overall performance. \n\n**(2). Pre-operative critic is necessary in GUI automation.**\n\n+ In dynamic GUI environments, generating accurate decisions is challenging for the GUI agent, leading to high error rates [36 ,27]. While self-reflection strategies fail in complex reasoning tasks [34, 52], an external pre-critic can objectively detect overlooked mistakes and provide corrective suggestions. \n+ Dynamic evaluation on AndroidWorld (Table 2) and case studies (Figure 4) confirm that the pre-critic can effectively identify errors and provide remedial feedback in GUI automation, demonstrating the necessity of an explicit critic module. \n\n**2. Training an effective policy model with GUI-Critic-Train dataset is challenging due to its larger action space versus critic's binary classification.** Policies typically need extensive training data (e.g., UI-Tars [26]). Following reviewer's suggestion, we constructed an equally-sized policy training dataset based on **GUI-Critic-Train** and applied cold-start and GRPO, yielding a policy model _Qwen2.5-VL-7B-Policy_. We evaluate it on AndroidWorld as follows. \n\n|  | Model | SR |\n| --- | --- | --- |\n| Policy | Qwen2.5-VL-7B  | 14.6 |\n| | Qwen2.5-VL-7B-Policy | 15.5 |\n| Policy + Pre-Critic | Qwen2.5-VL-7B + **GUI-Critic-R1** | **20.6** |\n| | Qwen2.5-VL-7B-Policy + **GUI-Critic-R1** | **21.5** |\n\n\nThe results clearly show that _Qwen2.5-VL-7B-Policy_ only obtains a small gain after training (14.6% to 15.5%), which indicates that training a good policy model is difficult. Besides, adding our critic to _Qwen2.5-VL-7B-Policy_ improves the performance from 15.5% to 21.5% obviously, demonstrating the complementary roles of policy and critic model and the necessity of critic model. \n\n**3. Focused Scope and Proven Effectiveness.** Regarding the challenge of predicting outcomes in complex GUI environments, we acknowledge this limitation but emphasize that **our critic primarily focuses on identifying obvious errors through static analysis of the GUI state, rather than predicting all possible outcomes.** Though not perfect in anticipating consequences, the 7% SR improvement on AndroidWorld (Table 3) validates its value in detecting errors and providing suggestions, enhancing task completion and efficiency. This confirms pre-critic as a promising research direction. \n\nWe will incorporate these clarifications in the revised manuscript. \n  \n  \n\n## **W2. Technical novelty and insights for broader domains**\nWe respectfully disagree with the assessment of limited technical novelty. Instead of simply adapting existing RFT + RL techniques to a specific GUI automation context, we **advance and specialize these methods to meet the distinctive demands of the GUI pre-critic task.**\n\n**1. Introduce technical innovations when compared with the standard GRPO method.**\n+ **The GUI pre-critic task** presents unique technical challenges that traditional RFT+RL methods cannot directly address, as it **requires a more reliable thought process (critique)**. Specifically, traditional RL rewards only focus on the accuracy of the final answer. In the GUI pre-critic, simply adopting the same technique leads to a critical credit assignment problem: when the model makes a correct judgment, it's difficult to determine whether this resulted from proper reasoning or merely superficial pattern matching. This problem is acute in GUI scenario where reliable reasoning is crucial for both action judgment and suggestion generation.\n+ To address this challenge, we introduce **a suggestion reward as an innovative indirect supervisory signal for CoT reasoning**, through measuring the semantic reliability of the generated suggestions. This reward is crucial as producing accurate revision suggestions requires proper analysis of the GUI operation, thereby encouraging the model to develop robust reasoning capabilities. The experiments in Table 4 demonstrate that this **simple yet effective** innovation of reward raises Suggestion Accuracy (SA) by 4.72 % and 7.02 % on GUI-I and GUI-S, respectively.\n\n| RFT | $r_f$ | $r_a$ | $r_s$ | CA | SA | CA | SA |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| ✓ | ✓ | ✓ | ✗ | 66.01 | 47.71 | 57.89  | 40.35 |\n| ✓ | ✓ | ✓ | ✓ | 69.20  | 52.43 | 58.77  | 47.37 |\n\n\n**2. Introduce two key methodological innovations in reasoning-bootstrapping based data collection pipeline.** To address the lack of suitable training data for GUI critics, we propose the following two key modules:\n+ A progressive Chain-of-Thought paradigm specifically designed for step-by-step GUI operation analysis,\n+ A Reasoning-Bootstrapping pipeline that enables automatic generation of high-quality critique data\n\nTable 3 in the main text shows that incorporating these two components yields consistent gains, validating their effectiveness.\n\n**3. Broader applicability to multimodal interactive domains**. Our core methodological contributions are domain-agnostic and can be readily adapted to other online domains.\n+ The S-GRPO framework naturally extends to scenarios where reliable reasoning and error prevention are crucial, such as **robotic manipulation** where operational mistakes can be costly.\n+ Our reasoning-bootstrapping method also provides practical insights for generating high-quality training data in **dynamic** environments.\n\n  \n  \n## **W3. Scalability of the data collection process**\n**1. Regarding public datasets concern, our data pipeline supports multiple data sources**.\n+ We support various data. For example, private/custom data from real-world applications, synthetic data generated by automated systems can be used. **Public datasets are chosen for their accessibility, quality, representation and diversity**.\n+ We acknowledge that **successful trajectories are necessary as ground truth for correct GUI operations**. It fundamentally aligns with how pre-critic model should be learned - by referencing successful examples combined with generated negative samples.\n+ When encountering GUI tasks beyond existing datasets, we can use automated GUI exploration methods [1*][2*] to collect successful trajectories. Negative sampling strategy and rule-based validation criteria can then generate training data, as in Section 3.1. Notably, **both modules are domain-agnostic**: the former capitalizes on mllm's general understanding of GUI interactions and focuses on universal principles (e.g., visibility and clickability) rather than being tailored to specific GUI scenarios, while the latter applies consistent rules across different GUI contexts.\n\n**2. Regarding GPT-4o concern, our GPT-4o filtering is optional, not mandatory.**\n+ As described in Section 3.1, **GPT-4o filters erroneous critic annotations for data quality**. Without filtering, the ablation study w/o DF in Table 3 reveals **only a 2% performance drop** and still demonstrates strong performance versus open-source baselines in Table 1. These results indicate that our pipeline's effectiveness isn't entirely GPT-4o dependent. \n+ GPT-4o's reasoning capability concern is also addressable. The GPT-4o filtering is optional with minimal impact on results. Moreover, GPT-4o is just one possible choice. Our framework can incorporate newer, more capable models as they emerge, and multiple models or verification methods can be combined to enhance reliability.\n\nWe will incorporate the aforementioned analysis in the final version of the paper.\n\n[1*] UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents\n\n[2*] GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior\n\n### **Q1. Distinct prompts for CoT vs. no-CoT sets**\nYes. Different system prompts are used for CoT and no-CoT datasets. CoT dataset uses the prompt in Appendix Table 2, while no-CoT dataset removes thought-related format, requesting only score and suggestion. We'll add this clarification to Section 4.1 and include both prompts in Appendix. \n\n### **Q2. How do agent use critic output?**\nThe agent treats the critic’s feedback as context and regenerates a new action. \n\nAt each step, if critic gives a positive correctness score, the action is executed; otherwise, agent replans with critique and suggestion in context. The feedback integration prompt is in Appendix Table 5.  We will add the clarification to Section 4.4.2 in the revised manuscript. \n\nThis design leverages complementary roles: policy generates actions while critic detects errors and provides suggestions. Due to GUI environments' dynamic nature, policy alone has high error rates [36]. An external pre-critic offers actionable suggestions, making it essential for reliable GUI automation.\n\n### **Q3. Typo**\nWe sincerely appreciate your catching this issue. In Table 4, the ✗ in the fourth row should be under $r_s$. We will correct this in the manuscript, with the updated line shown in the W2 response."}, {"review_id": "APtD6iHUNw", "reviewer": "Reviewer_1CqV", "summary": "The paper tackles the problem of catching step-level errors before they cripple an LLM-driven GUI agent. It introduces a pre-operative critic architecture embodied in GUI-Critic-R1, a vision-language model fine-tuned to judge a candidate action, explain why it might back-fire and suggest a fix and all before the click lands.\n\nAt the heart of the system is Suggestion-aware Group Relative Policy Optimization (S-GRPO). After a lightweight RFT *cold-start*, S-GRPO samples groups of candidate critiques; a tri-part reward (format, accuracy, and a novel suggestion reward that enforces actionable fixes) ranks them, and policy gradients push the model toward the best-scoring alternative. Algorithm 1 (S-GRPO) orchestrates this loop, while a built-in KL term keeps the critic tethered to a reference policy for stability.\n\nTraining data come from a reasoning-bootstrapping pipeline: positive actions are harvested from existing GUI trajectories; negative actions are hallucinated by an off-the-shelf agent and filtered by GPT-4o; Chain-of-Thought critiques are then boot-strapped in a progressive <thinking/score/suggestion> format to fill the gaps. The result is GUI-Critic-Train (~11k samples, 6k with full CoT) and GUI-Critic-Test spanning instruction, scenario and domain generalisation settings (GUI-I/S/W).\n\nWith a 7B Qwen-VL backbone, GUI-Critic-R1 lifts static critic accuracy on GUI-I from 54.9% to 69.2%, and suggestion accuracy from 43.1% to 52.4%, outperforming GPT-4o and other open models. Dropped into AndroidWorld as a plug-and-play pre-critic, it pushes the downstream task success rate from 22.4% to 27.6% while shortening action traces, beating both post-hoc reflection and larger pretrained critics.\n\nOverall, the work shows that coupling boot-strapped GUI reasoning data with S-GRPO yields a compact yet reliable critic that lets GUI agents look before they leap, preventing irreversible clicks and trimming redundant detours in real-time automation.", "questions": "1. Exactly how much end-to-end delay does the critic add per GUI action (mean, p95, worst-case) and how does it scale with screen resolution?\n2. How much does each S-GRPO reward head (format, accuracy, suggestion) contribute, and how sensitive are results to group size k?\n3. At the chosen threshold, how often does the critic wrongly block a valid action or permit a harmful one?\n4. Does a critic trained on AndroidWorld transfer to desktop or web GUIs?\n5. Will GUI-Critic-Train/Test and training scripts be released (licence + timeline)?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n**Critical Safety Angle:** Tackles the overlooked problem of *pre-emptively* vetoing bad GUI actions, turning reactive agents into agents that can “look before they leap.”\n\n**Novel Pre-Critic Architecture:** Couples a vision-language backbone with explicit <thinking/score/suggestion> reasoning slots, so the critic not only flags errors but explains and repairs them in natural language.\n\n**S-GRPO Innovation:** Introduces Suggestion-aware Group Relative Policy Optimization - an RL recipe that adds a *suggestion reward* on top of format and accuracy terms, directly optimising for actionable fixes rather than mere classification. \n\n**Boot-Strapped Dataset Contribution:** Releases GUI-Critic-Train/Test, built with a reasoning-bootstrapping pipeline that mixes real trajectories, hallucinated negatives, and GPT-4o-verified critiques.\n\n**Good Empirical Gains:** Lifts static critic accuracy on GUI-I from 54.9% to 69.2% and downstream AndroidWorld task success from 22.4% to 27.6%, outperforming GPT-4o and larger open models.\n\n**Lightweight Footprint:** Achieves those gains with a 7B model, showing that safety and efficiency can co-exist without resorting to larger models.\n\n**Plug-and-Play Integration:** Can be inserted as a decision gate in existing agents with no retraining of the actor, immediately shortening action traces and avoiding irreversible clicks.\n\n---\n## Weaknesses\n**Latency Overhead:** Every action now triggers a VLM forward pass, adding perceptible delay; the paper reports gains but no detailed timing budget for real-time RPA scenarios.\n\n**Limited Ablations:** The ablation table omits variants without the suggestion reward or with alternative optimisers, leaving open how much each component truly matters. \n\n**False-Positive Risk:** A conservative critic occasionally blocks valid but unfamiliar actions, lowering overall task throughput and yet false-negative/false-positive trade-offs are not quantified.", "was_revised": false, "final_justification": "", "rebuttal": "**We greatly appreciate your recognition of our work and your insightful and constructive feedback. We have carefully addressed each of your concerns and are confident that the resulting revisions will further improve the quality of our study.** \n\n### **W1 & Q1. Latency Overhead**\n1. **Timing budget**: Following the reviewer's suggestion, we conducted a comprehensive runtime analysis on AndroidWorld benchmark as shown in the table below. The experiments measure task runtime, step counts, and success rates. To ensure fair comparison, all models (except GPT-4o) are deployed with identical hardware resources. The latency for each step includes environment interaction time, model inference time, and action execution time. \"Avg Success Steps\" denotes the average step number of all successful tasks, and \"Avg success latency\" represents the average latency of all successful tasks.\n\n\n|  | Model | Avg Steps | Avg Success Steps | Avg Latency (s) | Avg Success Latency (s) | Success Rate (%) |\n| --- | --- | --- | --- | --- | --- | --- |\n| Baseline | Qwen2.5-VL-72B | 10.48 | 9.88 | 162.06 | 130.81 | 22.4 |\n| Baseline+Pre-Critic | + Qwen2.5-VL-7B | 10.22 | 8.37 | 286.02 | 184.35 | 20.3 |\n| | + Qwen2.5-VL-72B | 7.86 | 7.27 | 188.09 | 190.07 | 23.2 |\n| | + GPT-4o | 8.11 | 7.90 | 232.52 | 202.22 | 22.4 |\n| | + GUI-Critic-R1（Ours） | **7.19** | **7.09** | **156.91** | **127.61** | **27.6** |\n\n\nThe results show that:\n\n+ **Our GUI-Critic-R1 achieves clearly efficiency improvement over Baseline**: We achieve **lower average latency** (156.91s vs 162.06s) while substantially reducing the average number of steps (7.19 vs 10.48). Moreover, our model maintains a higher success rate (27.6% vs 22.4%), showing the effectiveness of the pre-operative critic mechanism.\n\n+ **Analysis of Latency Reduction**: Although the critic mechanism introduces an additional VLM forward pass at each step, **the significant reduction in the total number of steps** (~31% reduction from baseline) more than compensates for this overhead. This is because our pre-operative critic mechanism can prevent erroneous actions before execution, thus **avoiding time-consuming retry attempts**. This early error detection capability enables the model to make more accurate decisions in fewer steps, leading to more efficient task completion.\n\n+ **Our GUI-Critic-R1 shows superior efficiency compared with other pre-critic models.** It operates faster than both Qwen2.5-VL-72B critic (156.91s vs 188.09s) and GPT-4o critic (156.91s vs 232.52s). This performance advantage highlights our model's ability to strike an optimal balance between computational cost and effectiveness.\n\nIn conclusion, our proposed pre-operative critic mechanism achieves both high operational efficiency and improved task success rates, making it a practical solution for real-world GUI automation deployment. We will integrate these new experimental results into the revised manuscript.\n\n2. **End-to-end delay:** As shown in the table below, the critic introduces end-to-end latency per GUI action: 6.28s on average, with 7.11s at p95 and 7.26s in the worst case. \n\n3. **Screen Resolution Influence:** Additionally, we evaluate the inference time cost of our model at different screen resolutions. Specifically, we resize the original screenshot by the corresponding scale factor and evaluate the average inference time per test case on the GUI-I test set. As shown in the following table, we can find higher resolutions lead to increased inference times and the time difference is relatively small. This indicates that the image resolution does not greatly affect the inference delay.\n\n| Scale Factor | 0.1 | 0.5 | **1.0** | 1.5 | 2.0 |\n| --- | --- | --- | --- | --- | --- |\n| Avg Time (s) | 6.20 | 6.27 | **6.28** | 6.35 | 6.42 |\n| p95 (s) | 7.10 | 7.10 | **7.11** | 7.26 | 7.44 |\n| Worst (s) | 7.23 | 7.18 | **7.26** | 7.40 | 7.63 |\n\n\n### **W2 & Q2 Limited Ablation**\n\n1. **Ablations on S-GRPO Rewards:** Due to our oversight, the annotation in the fourth row of Table 4 was incorrect; this row should present the ablation results for $r_s$. We apologize for this error and will correct it in the revised manuscript. As shown in the table, when the $r_s$ term is removed, the model's performance decreases on both the Critic Accuracy and Suggestion Accuracy metrics. Furthermore, we have added ablation experiments for $r_a$ and $r_f$. Removing either of these components leads to a performance drop, as the accuracy reward ($r_a$) and format reward ($r_f$) are both essential reward components for ensuring the reliability of the intra-group advantage estimation for GRPO. \n\n| Model | Critic Accuracy | Suggestion Accuracy |\n| --- | --- | --- |\n| w/o $r_f$ | 64.65 | 51.31 |\n| w/o $r_a$ | 63.82 | 49.28 |\n| w/o $r_s$ | 66.01  | 47.71 |\n| Ours | 69.20  | 52.43 |\n\n\n2. **Ablations on Group Size:** We analyze the impact of group size in the S-GRPO phase in Section 4.3. The group size balances performance with resource utilization, leading to our selection of 6 as optimal (see Figure 3).\n\n### **W3 & Q3. False-Positive Results**\n\nThe false-positive (FP) and false-negative (FN) trade-offs are indeed crucial evaluation metrics for assessing a model's ability to accurately identify decision errors without misclassifying correct actions as errors. We evaluated the FP and FN metrics on the GUI-I dataset, and the results are presented in the table below. CA denotes critic accuracy.  \n\n| Model | FN (%)↓ | FP (%)↓ | CA (%)↑ |\n| --- | --- | --- | --- |\n| GPT-4o | 10.24 | 23.17 | 66.01 |\n| Qwen-7B  | 19.44 | 25.93 | 56.40 |\n| Qwen-72B  | 20.37 | 24.75 | 54.88 |\n| Ours | **8.69** | **21.84** | **69.20** |\n\n\nAs shown, our model achieves a 10.48% reduction in FN and 4.09% reduction in FP relative to GPT-4o, while remaining competitive with other MLLMs. \n\n+ Our method significantly reduces the rate - achieving only 8.69% FN (wrongly blocking valid actions or permitting harmful ones), substantially outperforming Qwen-7B (19.44%) and even surpassing GPT-4o (10.24%). \n\n+ This proves that our method achieves a strong balance between efficiency and effectiveness in GUI automation, significantly reducing both false rejections of valid operations and false approvals of harmful ones.\n\n### **Q4.  Extend to desktop or web** \n\n1. **Cross-Domain generalization to web interfaces.** While the model was trained on mobile GUI data from GUI-Critic-Train dataset, we extensively evaluated its transferability to **web scenarios through the GUI-W task in our GUI-Critic Benchmark**. As shown in Table 1 in the main text, our GUI-Critic-R1 achieves the **best performance** (63.08%) compared to other open source MLLMs, indicating its effective domain adaptation from mobile to web interfaces despite the inherent differences in UI layouts and interaction patterns.\n\n2. **Additional validation on a dynamic desktop benchmark OSWorld**[1*]. To more thoroughly validate the model's efficiently in desktop scenarios, we add a new experiment on the **OSWorld** benchmark, which is a **challenging dynamic benchmark on a real computer environment** that contains 369 computer tasks involving real web and desktop apps. We take Qwen 2.5-VL-72B as the action model (baseline), and the results are shown below.\n\n\n\n\n|  | Model | Success Tasks Number |\n| --- | --- | --- |\n| Baseline | Qwen2.5-VL-72B | 38 |\n| Baseline + Pre-Critic | + Qwen2.5-VL-7B  | 43 |\n| | + GUI-Critic-R1（Ours） | **50** |\n\n\nFrom the table, we can find that our GUI-Critic-R1 significantly boosts the number of successful tasks on OSWorld **(50 vs 38)** and outperforms the configuration that employs Qwen 2.5-VL-7B as the pre-critic **(50 vs 43)**. The results show that our model also exhibits satisfactory generalization ability in desktop and web environments. \n\n[1*] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurISP'24. \n\n### **Q5.  Release plan**\n\nWe have submitted the test code in the supplementary materials, and we plan to open-source the model parameters, dataset, and related resources in the future. \n\n### **Q6.  Typos**\n\nWe thank the reviewer for pointing out the issues in our abstract and references; we will correct these in the manuscript."}, {"review_id": "W6pic2x97H", "reviewer": "Reviewer_Znsj", "summary": "This work constitutes a significant advance for reliable GUI automation. Its formalization of pre-operative critique, coupled with S-GRPO and high-quality datasets, resolves critical safety/efficiency challenges. The dual static/dynamic validation demonstrates both diagnostic precision and operational efficacy, offering a foundational framework for safe human-AI interaction.", "questions": "- How is Isimilar(s,s′)*I*similar​(*s*,*s*′) computed in Equation (3)? The paper lacks detail here. Is the similarity judged by a vision-language model (e.g., Qwen2.5-VL-72B), or is it rule-based? More clarity would help understand how reliable and generalizable this reward is.\n\n- The critic model appears to perform reasoning-like functions similar to those found in advanced GUI action models with built-in \"thinking\" capabilities. This raises an important question: does introducing a separate critic reduce the need for reasoning within the action model itself, or do they complement each other? Clarifying this relationship would provide deeper insight into the system design and its modularity.\n\n- The paper does not discuss the computational overhead introduced by the real-time pre-critic mechanism. Since the critic is designed to operate before the GUI action model, it may lead to increased latency in practical applications. A discussion on runtime efficiency and its trade-offs would help assess the feasibility of deployment in real-world systems.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n- The work introduces the first formalized *pre-operative critic* mechanism for GUI automation, marking a conceptual shift from post-hoc correction to proactive error prevention. It proposes S-GRPO, a novel suggestion-aware reward formulation that extends GRPO to support multimodal critique generation. Furthermore, the authors develop a reasoning-bootstrapping pipeline to collect high-quality data without ground-truth, effectively addressing the lack of existing GUI critic datasets.\n\n- The paper demonstrates strong experimental design, evaluating the approach on both static (GUI-Critic-Test) and dynamic (AndroidWorld) benchmarks across mobile and web platforms. Comprehensive ablation studies (Table 3–4, Fig. 3) provide clear justification for key design choices.\n\nWeakness:\n\nI don’t have major concerns, but there are a few minor points worth noting:\n\n- The method shows relatively weak performance on the GUI-W benchmark (63.08% vs. 69.20% on GUI-I), raising concerns about its cross-platform generalization, particularly in desktop or web-based environments. This aspect is not sufficiently discussed in the paper.\n\n- The experiments are primarily conducted on scripted benchmarks like AndroidWorld. It remains unclear how well the method handles more complex, real-world conditions—such as dynamic UIs, unpredictable layout changes, or network-induced delays—which are common in practical GUI automation tasks.", "was_revised": true, "final_justification": "This paper presents a significant contribution to reliable GUI automation through its formalization of pre-operative critique, combined with the S-GRPO training strategy and a high-quality dataset. The approach effectively addresses key challenges in safety and efficiency, with strong empirical results.", "rebuttal": "**Thank you for your valuable feedback and for recognizing the contributions of our work. Your insights are greatly appreciated and will help us further improve the quality of our manuscript.** \n\n### **W1. Weak performance on the GUI-W**\n\n1. **About Domain Gap.** \n\n    * The performance difference between GUI-W (63.08%) and GUI-I (69.20%) benchmarks **stems from the domain gap between web and mobile scenarios, as our model was trained on mobile GUI scenarios.**\n\n    * We strategically focused on mobile applications because they present many challenges (e.g., frequent interface updates, diverse interaction patterns, and heterogeneous UI layouts). These characteristics make mobile GUI a more challenging and representative test bed for our approach.\n\n2. **Cross-domain Generalization and Methodology.**\n\n    * Despite this domain-specific training, our model demonstrates strong cross-domain generalization capabilities. As shown in Table 1 in the main text, **it achieves superior critic accuracy on the GUI-W benchmark compared to other MLLMs** with many more parameters (e.g., Qwen2.5-VL-72B at 60.05%), even though web interfaces differ significantly from mobile UIs in terms of layout structure and action space. \n\n    * Furthermore, **our core methodological contributions** - the S-GRPO training strategy and reasoning-bootstrapping based data collection pipeline - **are domain-agnostic** and can be readily adapted to other GUI environments including web and desktop platforms. We'll incorporate this analysis into Section 4.2.1 (Static Evaluation) of the revised manuscript.\n\n3.\t**Additional Validation on a dynamic desktop benchmark OSWorld [1\\*].** To more thoroughly validate the model's efficiency in desktop scenarios, we have added experiments on the OSWorld benchmark, which is a challenging dynamic benchmark on a real computer environment that contains 369 computer tasks involving real web and desktop apps. We take Qwen 2.5-VL-72B as the action model (baseline), and the results are shown below.\n\n\n||Model|Success Tasks Number|\n|---|---|---|\n|Baseline\t|Qwen2.5-VL-72B\t|38|\n|Baseline + Pre-Critic|\t+ Qwen2.5-VL-7B \t|43|\n| |+ GUI-Critic-R1（Ours）\t|**50**|\n\nFrom the table, we can find that our GUI-Critic-R1 significantly boosts the number of successful tasks on OSWorld **(50 vs 38)** and outperforms the configuration that employs Qwen2.5-VL-7B as the pre-critic **(50 vs 43)**. The results show that our model also exhibits satisfactory generalization ability in desktop and web environments. \n\n[1*] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurISP'24.\n\n### **W2. Real-world GUI Conditions**\n\n1.\t**Diversity of training sets.** Although dynamic testing was conducted on AndroidWorld, the training data included over 100 real-world apps, covering a wide range of realistic conditions. \n2. **Real-world static evaluation.** In this paper, we introduce a static evaluation benchmark including GUI-I/S/W test datasets (Section 4.1) that covers a broad spectrum of **real-world** applications and heterogeneous UI layouts. The consistent superiority of our method on this suite of tests provides evidence of its effectiveness. \n3.\t**Real-world experiments.** Currently, there is no established suitable real-world dynamic benchmark in mobile scenarios. In order to verify the capabilities of our model in real-world scenarios, we added **real-device experiments on Mobile-Agent-v2 test set** [36]. In this experiment, Qwen2.5-VL-72B serves as the actor agent, while our proposed GUI-Critic-R1 serves as the pre-critic model. We assess the performance on 24 advanced instructions targeting external applications (Table 5/6 in Reference [36]). \n\n|Model|SR|\n|---|---|\n|Baseline （Qwen2.5-VL-72B）|9/24|\n|Baseline + GUI-Critic-R1（Ours）|15/24|\n\nThe results show that with the assistance of our pre-critic, the agent successfully executed **6** tasks it had previously failed to execute. For instance, while executing “Find a video on Bilibili and give it the triple—like, coin, and favorite,” the agent assumed the task was complete after liking and favoriting, but the critic noticed that the coin had not yet been given and guided the agent model to finish the instruction. \n\n### **Q1. How is Isimilar(s,s′) computed?**\n\nWe leverage LLM (Qwen2.5-72B) to compute the semantic similarity between each generated suggestion and its ground-truth annotation (Lines 225–226 and 264). The prompt used for this calculation is given in Appendix Table 4. We will add a detailed explanation of this similarity-scoring procedure, with reference to Eq. 3, in the implementation section of the revised manuscript.\n\n### **Q2. Critic reduce the need for reasoning within the action model itself?**\n\nThank you for this valuable suggestion. No, introducing a separate critic does not and should not reduce the need for reasoning within the action model itself. \n* **Clarification**: Foremost, we aim to clarify that the introduction of the critic is intended to complement the action model, **rather than to supplant its reasoning capabilities**.\n\n* **Sequential workflow of the system**: In our workflow, action model first independently makes decisions, followed by the critic's evaluation and potential feedback (The relevant prompt is provided in Appendix Table 5). If the action model were too simplified and more error-prone, **it would lead to frequent critic interventions and multiple revision cycles**, impacting the system's efficiency and response time. This is particularly crucial in GUI automation where timely and accurate first-attempt decisions are important for a smooth user experience.\n* **Advantage**: Based on this consideration, both models should maintain their full reasoning capabilities and complement the system at different points. The action model is responsible for generating concrete GUI actions, whereas the critic model focuses on detecting erroneous decisions and providing corrective guidance. **This combination of strong initial decision-making and additional verification makes our system more robust and efficient**. \n\n### **Q3. Computational overhead**\n\nFollowing the reviewer's suggestion, we conducted a **comprehensive runtime analysis** on AndroidWorld benchmark as shown in the table below. The experiments measure task runtime, step counts, and success rates. To ensure fair comparison, all models (except GPT-4o) are deployed with identical hardware resources. The latency for each step includes environment interaction time, model inference time, and action execution time. \"Avg Success Steps\" denotes the average step number of all successful tasks, and \"Avg success latency\" represents the average latency of all successful tasks.\n\n||Model|Avg Steps|Avg Success Steps|Avg Latency (s)|Avg Success Latency (s)|Success Rate (%)|\n|---|---|---|---|---|---|---|\n|Baseline|Qwen2.5-VL-72B|10.48|9.88|162.06|130.81|22.4|\n|Baseline+Pre-Critic|+ Qwen2.5-VL-7B|10.22|8.37|286.02|184.35|20.3|\n| |+ Qwen2.5-VL-72B|7.86|7.27|188.09\t|190.07|23.2|\n| |+ GPT-4o|8.11|7.90|232.52\t|202.22\t|22.4|\n| |+ GUI-Critic-R1(Ours)|**7.19**|**7.09**|**156.91**|**127.61**|**27.6**|\n\nThe results show that:\n\n1.\t**Our GUI-Critic-R1 achieves clearly efficiency improvement over Baseline**: We achieve lower average latency (156.91s vs 162.06s) while substantially reducing the average number of steps (7.19 vs 10.48). Moreover, our model maintains a higher success rate (27.6% vs 22.4%), showing the effectiveness of the pre-operative critic mechanism.\n\n    * **Analysis of Latency Reduction**: Although the critic mechanism introduces an additional VLM forward pass at each step, **the significant reduction in the total number of steps** (~31% reduction from baseline) more than compensates for this overhead. This is because our pre-operative critic mechanism can prevent erroneous actions before execution, thus **avoiding time-consuming retry attempts**. This early error detection capability enables the model to make more accurate decisions in fewer steps, leading to more efficient task completion.\n\n2.\t**Our GUI-Critic-R1 shows superior efficiency compared with other pre-critic models**. It operates faster than both Qwen2.5-VL-72B critic (156.91s vs 188.09s) and GPT-4o critic (156.91s vs 232.52s). This performance advantage highlights our model's ability to strike an optimal balance between computational cost and effectiveness.\nIn conclusion, our proposed pre-operative critic mechanism achieves both high operational efficiency and improved task success rates, making it a practical solution for real-world GUI automation deployment. We will integrate these new experimental results into the revised manuscript.\n\nIn conclusion, our proposed pre-operative critic mechanism achieves both high operational efficiency and improved task success rates, making it a practical solution for real-world GUI automation deployment. We will integrate these new experimental results into the revised manuscript."}, {"review_id": "JvE0N9UGqN", "reviewer": "Reviewer_y9qJ", "summary": "This paper proposes a pre-operative critic mechanism for MLLM-based GUI agents. The critic can provide potential errors and suggestions for an action before execution. To train the critic model, the authors propose Suggestion-aware Group Relative Policy Optimization (S-GRPO). It contains a suggestion award besides a right or wrong signal. Besides, a data collection pipeline called reasoning bootstrapping is also developed for critic training and testing. Evaluation results show priority of this method.", "questions": "1. The experiment for critic evaluation is quite sufficient, however, the main experiment of dynamic evaluation is mainly conducted on Android environments. Is this method also suitable for more broader scenarios such as desktop applications? \n2. It requires high real-time abilities for online GUI agents, so how efficient the critic is? \n- In Sec. 4.2.2, the authors introduce a metric that measures the proportion that the ‘baseline + critic’ model achieves fewer steps than the baseline, called Efficiency Advantage Rate (EAR).  However, besides step number, the token number and latency are also factors that influence efficiency. If the critic tends to generate long suggestions, the task execution time may be longer. \n- Besides, for samples where the 'baseline+critic' model generates more steps than the baseline, the EAR metric does not evaluate how worse they are. So why choosing the EAR metric rather than computing average step numbers?\n3. A typo: in line 11 of the abstract, 'Suggestion-aware Gradient Relative Policy Optimization (S-GRPO)'. Gradient -> Group.", "limitations": "Yes. The authors have addressed the limitations in Appendix.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The idea of a pre-operative critic is novel and useful for GUI agent development.\n- The paper is well-written and logically organized. The method and results are presented clearly.\n- The dataset of GUI-Critic-Train and GUI-Critic-Test offers data support for future research on GUI agent critics.\n\nWeaknesses:\nPlease refer to the questions.", "was_revised": true, "final_justification": "1. The authors further conduct experiments on OSWorld benchmark, showing the ability of GUI-Critic-R1 to handle tasks of broader scenarios.\n2. The authors provide a more comprehensive efficiency comparison on average (success) steps and average (success) latency, making the evaluation more reliable.", "rebuttal": "**Thank you for your thoughtful review and acknowledging the potential and contribution of our work. We appreciate your insightful comments, which have provided us with an opportunity to refine our manuscript and address critical aspects that will enhance the clarity and impact of our research.**\n\n\n### **Q1. Experiments on broader scenarios such as desktop application**\n\n**Our GUI-Critic-R1 is designed to be domain-agnostic and can generalize well across different GUI environments.** As suggested by reviewer, to validate the model's effectiveness in broader scenarios such as desktop, we add a new experiment on the **OSWorld** benchmark, which is a **challenging dynamic benchmark on a real computer environment** that contains 369 computer tasks involving real web and desktop apps. We take Qwen2.5-VL-72B as the action model (baseline), and the results are shown below.\n\n\n||Model|Success Tasks Number|\n|---|---|---|\n|Baseline\t|Qwen2.5-VL-72B\t|38|\n|Baseline + Pre-Critic|\t+ Qwen2.5-VL-7B \t|43|\n| |+ GUI-Critic-R1（Ours）\t|**50**|\n\nFrom the table, we can find that our GUI-Critic-R1 significantly boosts the number of successful tasks on OSWorld **(50 vs 38)** and outperforms the configuration that employs Qwen 2.5-VL-7B as the pre-critic **(50 vs 43)**. The results show that our model also exhibits satisfactory generalization ability in desktop and web environments. \n\n[1*] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurISP'24. \n\n\n### **Q2. How efficient the critic is?**\n\n**1. Comprehensive efficiency analysis.** To make the efficiency assessment more comprehensive and credible, we recorded the step number, token number, and the latency required to complete tasks on the AndroidWorld benchmark as follows. \"Avg Success Steps\" denotes the average step number of all successful tasks, and \"Avg success latency\" represents the average latency of all successful tasks.\n\n\n||Model|Avg Steps|Avg Success Steps|Avg Latency (s)|Avg Success Latency (s)|Success Rate (%)|\n|---|---|---|---|---|---|---|\n|Baseline|Qwen2.5-VL-72B|10.48|9.88|162.06|130.81|22.4|\n|Baseline+Pre-Critic|+ Qwen2.5-VL-7B|10.22|8.37|286.02|184.35|20.3|\n| |+ Qwen2.5-VL-72B|7.86|7.27|188.09\t|190.07|23.2|\n| |+ GPT-4o|8.11|7.90|232.52\t|202.22\t|22.4|\n| |+ GUI-Critic-R1(Ours)|**7.19**|**7.09**|**156.91**|**127.61**|**27.6**|\n\n\nThe results clearly show that:\n- **Our GUI-Critic-R1 achieves clearly efficiency improvement over Baseline.** We achieve lower average latency (156.91s vs 162.06s) while substantially reducing the average number of steps (7.19 vs 10.48). Moreover, our model maintains a higher success rate (27.6% vs 22.4%), showing the effectiveness of the pre-operative critic mechanism. Besides, our GUI-Critic-R1 generates suggestions averaging less than 30 words. \n- **Analysis of Latency Reduction**: Although the critic mechanism introduces an additional VLM forward pass at each step, **the significant reduction in the total number of steps** (~31% reduction from baseline) more than compensates for this overhead. This is because our pre-operative critic mechanism can prevent erroneous actions before execution, thus **avoiding time-consuming retry attempts**. This early error detection capability enables the model to make more accurate decisions in fewer steps, leading to more efficient task completion.\n- **Our GUI-Critic-R1 shows superior efficiency compared with other pre-critic models**. It operates faster than both Qwen2.5-VL-72B critic (156.91s vs 188.09s) and GPT-4o critic (156.91s vs 232.52s). This performance advantage highlights our model's ability to strike an optimal balance between computational cost and effectiveness.\nIn conclusion, our proposed pre-operative critic mechanism achieves both high operational efficiency and improved task success rates, making it a practical solution for real-world GUI automation deployment. We will integrate these new experimental results into the revised manuscript.\n\n **2. The rationality of EAR (Efficiency Advantage Rate).** We believe that EAR effectively reflects whether our model, compared to the baseline, can complete the same tasks with fewer steps. Average execution time and average steps might be significantly skewed by substantial advantages of a small number of tasks. We fully acknowledge the reviewer's perspective that EAR alone is not comprehensive enough. Indeed, EAR, average step length, and average latency each measure model efficiency from different angles, and all these metrics are necessary for a complete evaluation. We will update the revised manuscript with these results. \n\n\n### **Q3. Typo**\n\nWe thank the reviewer for pointing out the issue in our abstract; we will correct this in the manuscript."}]

Author rebuttal:

Meta-review: The paper introduces GUI-Critic-R1, a pre-operative critic mechanism for Multimodal Large Language Models (MLLMs) in GUI automation. The key contributions are:  
1. Pre-operative Error Detection: Unlike post-hoc correction, the critic evaluates actions *before* execution, preventing irreversible errors (e.g., accidental deletions or payments).  
2. S-GRPO Training: A novel Suggestion-aware Group Relative Policy Optimization method that incorporates a suggestion reward to improve critique reliability.  
3. Dataset Creation: A reasoning-bootstrapping pipeline generates GUI-Critic-Train and GUI-Critic-Test, addressing the lack of GUI critic datasets.  
4. Empirical Results:  
   - Static Evaluation: Outperforms open-source MLLMs (e.g., Qwen-7B) and matches GPT-4o in critic accuracy.  
   - Dynamic Evaluation: On AndroidWorld, integrating GUI-Critic-R1 improves task success rates and reduces steps.  

All reviewers are satisfied with the rebuttal and give an acceptance recommendation, though minor weaknesses remain (e.g., domain generalization).

**Final Decision: Accept.**
Decision: Accept (poster)