Venue: NAACL 2025

# ReachAgent: Enhancing Mobile Agent via Page Reaching and Page Operation

Qinzhuo Wu, Wei Liu1 , Jian Luan, Bin Wang

XiaoMi AI Lab

{wuqinzhuo, liuwei40, luanjian, wangbin11}@xiaomi.com

Footnote 1: Corresponding author.

###### Abstract

Recently, mobile AI agents have gained increasing attention. Given a task, mobile AI agents can interact with mobile devices in multiple steps and finally form a GUI flow that solves the task. However, existing agents tend to focus on the most task-relevant elements at each step, leading to local optimal solutions and ignoring the overall GUI flow. To address this issue, we constructed a training dataset called MobilieReach, which breaks the task into page reaching and operation subtasks. Furthermore, we propose ReachAgent, a two-stage framework that focuses on improving its task-completion abilities. It utilizes the page reaching and page operation subtasks, along with reward-based preference GUI flows, to further enhance the agent. Experimental results show that ReachAgent significantly improves the Intersection over Union (IoU) Accuracy and Text Accuracy by **7.12% and 7.69%** on step-level and **4.72% and 4.63%** on task-level compared to the SOTA agent. Our data and code are available at [https://github.com/XiaoMi/reachagent](https://github.com/XiaoMi/reachagent).

## 1 Introduction

With the quick advancement of visual language models (VLMs) [58, 3], it has become more feasible to create mobile AI agents that can operate mobile devices [36, 37, 38]. Some early works [39, 39, 39] have attempted to combine powerful general VLM models, such as GPT-4V [42], with prompt engineering and retrieval modules to generate UI actions for mobile control. Other works have used GUI navigation datasets [39, 39] to fine-tune the base VLMs to improve their stability in generating UI actions. Subsequent works [1, 38] have built mobile-specific datasets to pre-train VLMs so that the model can better understand graphical user interface (GUI) elements and page structure. Specifically, mobile AI agents use the VLM as their base model and the mobile device as the environment. The agent iteratively interacts with the mobile's GUI for each mobile control task, by generating actions based on environmental information and executing those actions to update the GUI environment. This interactive process forms a GUI flow.

Although existing mobile AI agents have shown good performance, they still have some limitations. Existing agents focus on interacting with the most task-relevant elements of the current page, and ignore whether the entire GUI flow solves the task. This method sometimes causes them to focus on single-step action accuracy and select tasks-related actions greedily, thus falling into a local optimal solution. As shown in Figure 2, for the task of adding Xiaomi 14 to the shopping cart, the agent should first reach the product page, but the homepage does not contain elements related to the

Figure 1: An example of a task and its subtasks and possible GUI flows. Green boxes represent the pages that need to be reached, and orange arrows represent the actions that need to be operated. To complete this task, the agent must reach 5 pages and do 3 operations.



product name. In this case, it might go directly from the homepage to the shopping cart page.

Therefore, we break down the task into several subtasks and focus on the agent's subtask completion abilities. Intuitively, these subtasks can be divided into two categories, reach and operate. Reach only requires the agent to reach a specific page, regardless of the path taken. Operate requires the agent to reach a specific page and perform some specific operations. Figure 2 shows a task and its GUI flow, consisting of a 9-step chain, the key to solving this task is to reach the product page first, and then click the "Add to Cart" pop-up window and select the attributes and "OK" button in the pop-up window in sequence, as shown in Figure 1. For the product page reach subtask, it can be done either through product search or by browsing the product catalog. Similarly, the operate subtask only requires the agent to select the "B" attribute in the pop-up window, and browsing other attributes does not affect the task completion.

To address these limitations, we collected a training dataset called MobileReach, which contains three types of tasks: page navigation, page reaching, and page operation. In addition, we propose ReachAgent, a two-stage framework that focuses on the ability to complete subtasks. In the first stage, in addition to page navigation abilities, it also learns how to reach a specified page and complete operations on a specified page. In the second stage, a 4-level reward function is used to weigh different GUI flows and construct preference data to further reinforce ReachAgent, Furthermore, an action alignment mechanism is proposed to reduce the difficulty of action generation. The main contributions of this paper can be summarized as follows:

\(\bullet\) We break down the mobile control task into page reaching and page operation subtasks, and construct a mobile control dataset called MobileReach consisting of three types of tasks.

\(\bullet\) We proposed ReachAgent, a two-stage framework that utilizes page reaching and page operation to enhance the agent's subtask completion abilities. In addition, it uses reinforcement learning to further enhance the agent's overall task completion ability. ReachAgent includes an action alignment mechanism that decreases the number of candidate actions, thereby reducing the difficulty of the task.

\(\bullet\) Experimental results show that ReachAgent outperforms existing state-of-the-art agents. It also demonstrates stronger page reaching and operation abilities.

## 2 Related Work

**Mobile AI Agent.** Many recent studies have proposed mobile AI agents for device control. Apparent Yang et al. (2023) and Mobilagent

Figure 2: The complete 9-step GUI flow for a task. Green boxes represent the pages that need to be reached, and green circles represent the operations that need to be done. Orange arrows are the actions in the golden flow. Blue arrows are the actions in other GUI flows. Both the orange and blue flows can complete the task.

 

(Ding, 2024) use prompt engineering methods and rely on existing closed-source models (e.g., GPT-4V, GPT-4o) to achieve mobile control. Other studies like CogAgent Hong et al. (2023) and Agenttuning Zeng et al. (2023) use data-driven methods to fine-tune the open-source VLMs. In order to improve the device control abilities of mobile AI agents, large-scale datasets with diverse scenarios and accurate annotations are needed.

Rico Deka et al. (2017) and AITW Rawles et al. (2023) are two publicly available large-scale GUI datasets, containing 72,219 single-step GUI tasks and 715,142 multi-step page navigation tasks, respectively. They are widely used in multiple GUI modeling works Wang et al. (2021); Li et al. (2021); Hsiao et al. (2022). However, they are constructed by combining crowdsourcing workers and automated annotation, which also leads to noise and wrong labels. To this end, a series of published works clean and filter these datasets to improve their quality. Enrico Leiva et al. (2020), UI-Bert Bai et al. (2021), Vins Bunian et al. (2021) extended the Rico dataset and proposed new GUI tasks such as UI layout classification and UI element retrieval. AutoUI Zhan and Zhang (2023) further filtered the GooglePlay tasks in the AitW dataset. However, these datasets mainly contain screenshots and OCR text, lacking GUI raw data such as xml documents, which limits the agent to further obtain mobile environment information. MobileVLM Wu et al. (2024) and ScreenAI Baechler et al. (2024) build large-scale pre-training datasets to enhance the agent's UI understanding ability. Here, Mobile3M Wu et al. (2024) proposed by MobileVLM uses a breadth-first approach to explore the GUI pages of each APP and forms these pages into a graph structure. This allows us to obtain environmental information about different paths and operations, for example, the next pages after performing different actions in a page, and how many paths there are from one page to another. However, as a pre-training dataset, MobileVLM lacks fine-tuning tasks. Most of their tasks are UI understanding and single-step action generation. A small number of multi-step tasks can be summarized as navigate to a page with a certain button.

Therefore, based on Mobile3M, we extracted available GUI flows and annotated a page navigation dataset. Using the graph structure environment, we further extracted page reaching and page operation datasets, and scored and paired preference datasets for RL learning. We used the three page datasets for one-stage SFT fine-tuning of ReachAgent and the preference datasets for two-stage RL optimization.

**Reinforcement Learning (RL).** RL-based fine-tuning methods have shown great potential in

Figure 3: Actions and tasks for a GUI flow. The step-by-step description provides a set of action history, where each step corresponds to an action performed on that GUI page. The brief task is a concise task description that aligns with this GUI flow.

 

improving the generation quality of LLMs (Liu et al., 2023; Shen et al., 2023; Yuan et al., 2023). For mobile device control, some agents use RL methods to further optimize fine-tuned models.

DigRL (Bai et al., 2024) uses Gemini 1.5 Pro Team et al., 2024) as an automatic evaluator to assign rewards to each GUI flow generated by the agent, and updates the model with the annotated GUI flows through online RL. DistRL Wang et al., 2024) uses multiple workers to collect interaction data and then sends them to a central learner for learning, focusing more on distributed frameworks. These works are limited by data collection efficiency and labor-intensive manual annotation, and therefore cannot be well expanded.

Our work combines agent generation and graph path sampling to obtain a large number of GUI flows, uses page reaching and operation subtasks to set reward rules, and builds preference data for each step of the GUI flow.

## 3 Dataset Construction

### Data Definition

**GUI flow:** Given a user's instruction, an agent will conduct multiple rounds of interactions with the mobile device to complete the task and record the process as GUI flow. Specifically, the GUI flow is a chain structure with the GUI page as the head node, the agent's action as the edge, and the resulting GUI page after executing the action as the tail node. Figure 2 shows a GUI flow. It is defined as a 9-step chain sequence, which includes a complete set of 9 GUI Pages and 9 actions.

**GUI page:** Our algorithm uses an Android emulator to run the apps and display the GUI pages. A typical GUI page consists of a screenshot and an XML document containing multiple elements such as buttons, text boxes, icons, and images. These elements have unique identifiers, such as text, bounds, and resource IDs, used to interact with them programmatically. Figure 4 shows a GUI page example.

**Action:** Our action set includes four types of actions: Click, Scroll, Input, and Complete. Figure 3 summarizes the parameters and examples of these actions. When the agent interacts with the UI page, such as clicking a button or entering text, Appium1 sends an action command to the Android emulator, which then performs the corresponding action. The changes of the GUI page are then captured as a new screenshot and XML document, which are used as the environment for the next step.

Footnote 1: [https://appium.io/docs/en/latest/](https://appium.io/docs/en/latest/)

### Page Navigation Dataset

Our page navigation dataset is built using Mobile3M's GUI flows. Mobile3M Wu et al., 2024) uses random walks to explore the page jump paths of 49 apps and combines them into a graph format. We sample GUI flows from the graph and generate corresponding tasks to build the page navigation dataset. The construction pipeline is as follows:

1. GUI flow extraction: Select GUI flows with path lengths of 3-10 steps from Mobile3M.

2. Image Caption: Generate image captions for each GUI page in the flow with InternVLChen et al., 2024).

3. Task Generation: Use GPT-4V to generate a step-by-step description and a brief task for the GUI flow based on the captions of each page and the actions between pages, as shown in Figure 3. The prompt for task generation is in Appendix B.

4. Data cleaning: Filter out low-quality GUI flows with duplicate tasks or invalid GUI pages.

After the above steps, we have a total of 53,832 GUI flows and 259,742 action steps, approximately 4.8 steps per task. Each task includes a brief task, GUI pages, and corresponding actions, along with a step-by-step description of the task.

### Page Reaching and Operation Dataset

To improve the model's ability to reach and operate mobile GUI pages, we break down the GUI flow into multiple subtasks. There are two types of subtasks described as follows:

**Page reaching task.** It takes an instruction for reaching a specific page as input and a GUI flow from the app homepage to that page as output. Here, each GUI page is named in three ways.

1. If the corresponding step in the step-by-step description mentions the name of a page, we use that name directly, such as "search results page."

2. We extract the element name or text input from the action as the page name, such as "white" or "add to cart".

3. If the name is invalid or too common, we use GPT-4 to summarize the page name based on the GUI page's caption.

We use page names and templates to construct page reaching tasks. The pre-defined task templates are shown in Appendix B.

**Page operation task.** It requires the agent to complete a specific task on a page, which requires 

the model to have two abilities: navigate to a page that can complete the task, and then complete the specific task. There are two extraction methods for this task:

1. In the step-by-step description, if a step involves a scroll or input action, it is usually a page operation task. We extract the sub-flow from the GUI flow to pair with this step's description.

2. In the GUI flow, if an action has similar pages before and after execution, it is usually a page operation task. We use GPT-4V to construct a task for the sub-flow that ends with this action. Similar to Wu et al. (2024), the definition of similar pages is based on page similarity and the number of co-occurring elements.

Eventually, we have a page reaching dataset with 67,920 GUI flows and 374,834 action steps, a page operation dataset with 76,252 GUI flows and 368,942 action steps. See Appendix C for details.

## 4 Models

### Action Alignment Mechanism & Action Space Generation

The mobile control task can be represented as a multi-round interaction process, which generates an action sequence \(\overrightarrow{A}=\{a_{1},a_{2},...,a_{t}\}\) based on a given user instruction X and GUI pages \(\overrightarrow{P}=\{\text{P}_{1},\text{P}_{2},...,\text{P}_{t}\}\) in the mobile environment. This multi-round interaction process forms a GUI flow.

**Action Alignment Mechanism.** As shown in Figure 4, the GUI page P\({}_{t}\) contains two forms: screenshot \(\text{P}^{\rm vis}_{t}\) and XML document \(\text{P}^{\rm xml}_{t}\). The screenshot shows multiple elements of the GUI page, while the XML document provides text descriptions and attributes for these elements. While users can touch any point on the screen, their goal is to interact with an element on the page. To narrow down the range of candidate actions, we propose an action alignment mechanism. That is, we suggest that the agent only clicks on the center point of an element or scrolls along its central axis. For a candidate box [273, 84][324, 180], we can click or enter at position (298, 132). For a candidate box [0,528][720, 960], we can scroll left from (360, 744) to (180, 744).

**Action Space Generation.** With the action alignment mechanism, we can extract all the candidate actions in the GUI page as the candidate action space \(\text{P}^{\rm acts}_{t}\). Each element in the screenshot corresponds to an element description in the XML document. We extract the element's bounding box and its attributes, such as whether it is clickable, scrollable, and inputtable, from the description and form an action. As shown in Figure 4, the "search box" element can be formed into an input action and a click action. The "RecyclerView" element can be formed into scrolling actions in 4 directions.

### First Stage SFT Framework

Given a task X, in the t-th step, the screenshot \(\text{P}^{\rm vis}_{t}\) is encoded into the image representations \(\mathbf{h}^{\rm vis}_{t}\) by a frozen visual encoder as follows:

\[\mathbf{h}^{\rm vis}_{t}=\textbf{ViT}(\text{P}^{\rm vis}_{t}). \tag{1}\]

Here, visual encoder **ViT** is ViT-bigG (Dosovitskiy et al., 2020). The candidate action space \(\text{P}^{\rm crls}_{t}\) is extracted from \(\text{P}^{\rm xml}_{t}\) and encoded into the text representation \(\mathbf{h}^{\rm acts}_{t}\) as follows:

\[\mathbf{h}^{\rm acts}_{t}=\mathbf{E}(\text{ActionSpace}(\text{P}^{\rm xml}_{t })). \tag{2}\]

Here, \(\mathbf{E}(\cdot)\) is a wording embedding layer. The candidate action space \(\text{ActionSpace}(\text{P}^{\rm xml}_{t})\) is a list of all possible actions that can be used to interact with the page, as described in Section 4.1.

Similarly, instruction X and action history \(\overrightarrow{A}_{<t}\\ =\{a_{1},...,a_{t-1}\}\) are also encoded as \(\mathbf{h}^{\rm X}_{t}\), and \(\mathbf{h}^{\rm A}_{<t}\). ReachAgent uses a position-aware visual language adapter to align visual representations with text representations, and generate the the action \(a_{t}\):

\[\begin{split}\mathbf{h}^{\rm X}_{t},\mathbf{h}^{\rm A}_{<t}= \mathbf{E}(\mathrm{X}),\mathbf{E}(\overrightarrow{A}_{<t}),\\ \mathbf{h}^{\rm vriis}_{t}=&\text{Adapter}(\mathbf{ h}^{\rm vis}_{t}),\\ \mathrm{P}(a_{t})=&\text{ReachAgent}(\mathbf{h}^{ \rm X}_{t},\mathbf{h}^{\prime\rm vis}_{t},\mathbf{h}^{\rm acts}_{t},\mathbf{h }^{\rm A}_{<t}).\end{split} \tag{3}\]

After generating the action \(a_{t}\), we execute \(a_{t}\) in the mobile environment to update the GUI page P\({}_{t}\) to P\({}_{t+1}\). Therefore, in the next step, ReachAgent has instruction X, screenshot \(\text{P}^{\rm vis}_{t+1}\), candidate action space \(\text{P}^{\rm acts}_{t+1}\) and action history \(\overrightarrow{A}_{<t+1}\) as inputs for action \(a_{t+1}\) generation. ReachAgent needs to go through multiple rounds of action generation and execution loops until it finally generates the action "STATUS_task_COMPLETE" to end the mobile-control process, or until it exceeds the maximum number of interaction steps.

In the first stage, ReachAgent fine-tuned its ability to reach pages and perform actions on them by using all three datasets. The cross-entropy loss is defined as:

\[\mathcal{L}=\sum_{t}\log\mathbf{P}(a_{t}|\mathrm{X},\text{P}^{\rm vis}_{t}, \text{P}^{\rm acts}_{t},\text{A}_{<t}).\] (4) 



### Second Stage RL Framework

**Preference data construction.** To further reinforce ReachAgent, we constructed a preference dataset \(D=\{(X,P_{t},a^{w}_{t},a^{l}_{t})\}\). Here, \(a^{w}_{t}\) is the chosen action at step t, and \(a^{l}_{t}\) is the rejected action. Our construction approach is based on two key principles: 1. The GUI flow should reach the required page and complete the required operation in the instruction; 2. The GUI flow should be as short as possible.

Given an instruction X and the current page P\({}_{t}\), we can divide all potential actions into four levels:

**Golden:** The action is in the golden GUI flow, or in a GUI flow that can complete the instruction and has the same length as the golden flow.

**Longer:** The action is in a GUI flow longer than golden, but can still complete instruction X.

**Incomplete:** The action is not in a GUI flow that can complete the instruction.

**Invalid:** The action cannot be executed or is not in the action space of the current page.

Naturally, for these four levels, we expect their reward scores to be:

R(Golden)>R(Longer)>R(Incomplete)>R(Invalid) (5)

Given a task, "add product A to the shopping cart", the subtasks it needs to complete are simplified into Reach(P\({}_{2}\)): Go to the "Product A" page, and Operate(P\({}_{2}\), "add to cart"): Add Product A to the cart on the product page. Figure 4 shows several GUI flows for this task. "P\({}_{0}\), P\({}_{1}\), P\({}_{2}\), P\({}_{3}\)" and "P\({}_{0}\), P\({}_{1}^{1}\), P\({}_{2}\), P\({}_{3}\)" are the 4-step golden GUI flows. "P\({}_{0}\), P\({}_{1}^{2}\), P\({}_{2}^{2}\), P\({}_{2}\), P\({}_{3}\)" takes 5 steps to complete the task and marked as Longer. "P\({}_{0}\), P\({}_{1}^{1}\), P\({}_{3}^{1n}\) doesn't reach the P\({}_{2}\) page and marked as Incomplete.

Based on the 4-level reward function, we identify the chosen and rejected actions for the shared history flow. Start with GUI page P\({}_{0}\), \(a^{0}_{0}\) and \(a^{1}_{0}\) are in the Golden flow, while \(a^{2}_{0}\) leads to a Longer flow. Therefore, R(\(a^{0}_{0}|\)P\({}_{0}\))>R(\(a^{2}_{0}|\)P\({}_{0}\)) and (X, P\({}_{0}\), \(a^{0}_{0}\), \(a^{2}_{0}\)) is a preference data. Start with page P\({}_{1}^{1}\), \(a^{1}_{1}\), \(a^{3}_{1}\), and \(a^{4}_{1}\) lead to a Golden, Longer, and Incomplete flow respectively. The preference data could be (X, P\({}_{1}^{1}\), \(a^{1}_{1}\), \(a^{3}_{1}\)) and (X, P\({}_{1}^{1}\), \(a^{1}_{1}\), \(a^{4}_{1}\)).

To collect preference data, we use the SFT agent to regenerate the training split of the page navigation dataset. If the reward of an action is less than the golden action, we use it to construct the preference data. Otherwise, we randomly select an action with a lower reward in the action space of each page to pair with the golden action.

**DOO Optimization.** Direct Policy Optimization (DPO) Rafailov et al. (2024) does not require an explicit reward score, but only requires a preference for paired data, which is more suitable for our 4-level reward function. Therefore, we adopt DPO to

Figure 4: The overview of our proposed ReachAgent. (a) Extracting action space from XML document. (b) In the first stage, the framework generates a GUI flow through multiple interaction steps with the GUI page. (c) In the second stage, it uses the reward function to construct preference data to further reinforce the SFT framework.



optimize ReachAgent. The DPO loss is defined as:

\[\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{SFT}})=- ~{}\mathbb{E}_{(\texttt{X},\texttt{P},a^{w}_{t},a^{l}_{t})\sim D}[\log\sigma \tag{6}\] \[(\beta\log\frac{\pi_{\theta}(a^{w}_{t}|\texttt{X},\texttt{P}_{t} )}{\pi_{\text{SFT}}(a^{w}_{t}|\texttt{X},\texttt{P}_{t})}-\beta\log\frac{\pi_{ \theta}(a^{l}_{t}|\texttt{X},\texttt{P}_{t})}{\pi_{\text{SFT}}(a^{l}_{t}| \texttt{X},\texttt{P}_{t})})].\]

Here, \(\pi_{\theta}\) is the DPO agent to learn and \(\pi_{\text{SFT}}\) is the first stage SFT agent. \(\theta\) is the sigmoid function and \(\beta\) is a hyperparameter that controls the deviation from \(\pi_{\text{SFT}}\).

## 5 Experiment

### Datasets and Benchmarks

The MobileReach dataset contains 3 splits: page navigation, page reaching, and page operation. We used the GUI graph from Mobile3M to build the MobileReach dataset. The detailed information of the MobileReach dataset is shown in Table 2. For preference data, 48,013 rejected actions are generated by the agent, and 211,729 rejected actions are sampled from the action space.

We used the page navigation split of the MobileReach dataset and the Auto-UI dataset Zhan and Zhang (2023) for testing. For the Auto-UI dataset, we follow the official split and method for finetuning. For the MobileReach dataset, we randomly selected 5% of the GUI flows as a test set before data construction. Therefore, there was no overlap between the training set and the test set.

### Evaluation Metrics

We use two objective metrics to evaluate the position and text of the generated action. IoU Acc evaluates the bounding box accuracy. Text Acc evaluates the text accuracy. Step-level accuracy evaluates whether each action in a GUI flow is correct, while task-level accuracy evaluates whether all actions in an entire GUI flow are correct. See more metric details in Appendix A.

### Implementation Details and Baselines

ReachAgent chose MobileVLM as the backbone model. We use 8 80GB Nvidia A100 GPUs for fine-tuning. The learning rate is 1e-5, and the agent's max length is 4,096. For the MobileReach dataset, in the first stage, ReachAgent was trained for 2 epochs on all three splits. In the second stage, it was trained for 2 epochs on 259,742 preference data. For the Auto-UI dataset, ReachAgent is fine-tuned for 1 epoch on its training data using the first-stage framework. For specific information on parameters and baselines, refer to Appendix A.

### Main results

The main experimental results are presented in Table 1 and 4. We can observe that:

\(\bullet\) For MobileReach dataset, ReachAgent improves the IoU Accuracy, and Text Accuracy by **+7.12%**, **+7.69%** on step-level and **+4.72%** and **+4.63%** on task-level compared to fine-tuned MobileVLM. It also raises **+6.59%** on Task Success Rate. This proves that ReachAgent is not only good at generating actions at each step, but also provides a more efficient GUI flow to successfully complete the task.

\(\bullet\) At both step-level and task-level, ReachAgent-stage1 has a significantly higher IoU and Text accuracy than other SFT baselines. We attribute this to our action alignment mechanism and the introduction of page reaching and page operation

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt}|c c|c c|c} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c|}{Step-level Acc} & \multicolumn{2}{c|}{Task-level Acc} & \multicolumn{2}{c|}{Task Success} \\ \cline{3-7}  & Method & IoU & Text & IoU & Text & \\ \hline GFT-to OpenAI, 2023 & FewShot & 19.44\% & 17.06\% & - & - & - \\ Qwen-VL Bai et al. (2023) & FewShot & 0.05\% & 1.49\% & - & - & - \\ MobileVLM\({}_{\text{unified}}\) Wu et al. (2024) & FewShot & 2.43\% & 4.91\% & - & - & - \\ MobileVLM\({}_{\text{experate}}\)Wu et al. (2024) & FewShot & 1.75\% & 10.60\% & - & - & - \\ \hline Qwen-VL Bai et al. (2023) & SFT & 73.38\% & 72.14\% & 30.13\% & 26.22\% & 35.77\% \\ Auto-UL\({}_{\text{unified}}\)Zhan and Zhang (2023) & SFT & 73.26\% & 70.88\% & 29.60\% & 22.20\% & 33.40\% \\ MobileVLM Wu et al. (2024) & SFT & 76.20\% & 74.08\% & 34.03\% & 28.43\% & 39.78\% \\ \hline
**ReachAgent-stage 1** & SFT & **83.34\%** & 81.47\% & 37.82\% & 31.31\% & 44.85\% \\
**ReachAgent-stage 2** & SFT+RL & 83.32\% & **81.77\%** & **38.75\%** & **33.06\%** & **46.37\%** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Main Result(%) on MobileReach. SFT baselines are fine-tuned for 2 epochs on page navigation split.

\begin{table}
\begin{tabular}{p{113.8pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline Dataset & \multicolumn{2}{c}{Train} & \multicolumn{2}{c}{Test} \\ \cline{2-4}  & Chain & Step & Chain & Step \\ \hline Page Navigation & 53,832 & 259,742 & 2,689 & 12,922 \\ Page Reaching & 67,920 & 374,834 & 3,385 & 16,253 \\ Page Operation & 76,252 & 368,942 & 3,798 & 18,338 \\ \hline \end{tabular}
\end{table}
Table 2: The statistics of MobileReach dataset.

 

subtasks. ReachAgent achieves better page navigation abilities by learning how to go to specific pages and complete specific operations.

\(\bullet\) Compared to ReachAgent-stage1, ReachAgent significantly outperforms it at the task level and is competitive with it at the step level. This can be attributed to our reinforcement learning mechanism, which encourages the agent to generate GUI flows that can complete the task.

\(\bullet\) For the Auto-UI dataset, ReachAgent outperformed the SOTA baselines in all tasks. Note that ReachAgent achieves this result without building subtask data on Auto-UI. This proves the generalizability of the agent's page reaching and page operation abilities.

### Ablation Study

Table 5 shows several ablation experiment results.

**Effects of Action Alignment Mechanism:** Through the action alignment mechanism, the performance of the agent (+Al) is improved by 5.55% and 1.46% on average at the step level and task level. This is because the action alignment mechanism helps to narrow down the range of possible actions that the agent can take, reducing the generation difficulty.

**Effects of Page Reaching and Page Operation Subasks:** Adding the page reaching subtask and the page operation subtask both improve the agent's performance. The accuracy of the "+ Al & Re & Op" increases by an average of 1.7% and 1.87% at the step level and task level compared to "+ Al". These subtasks enhance the agent's understanding of the task goal and the order of actions, making the agent generate actions that are more aligned with the overall goal of the task. This leads to better performance at both the step and task levels.

**Effects of Reinforcement Learning:** ReachAgent further adds RL to "+ Al & Re & Op". It mainly improves the task-level accuracy. This is because our reward focuses more on whether the agent can generate a short GUI flow that can complete all subtasks, rather than whether the action of each step is exactly the same as the golden answer.

**Ability to complete subtasks:** We also compared the ability of different ablations to complete the Reach and Operate subtasks. As shown in Table 3, ReachAgent improves the overall IoU accuracy by 1.9% and the Text accuracy by 2.18%, outperforming the agent without RL in all subtasks. This shows that the RL mechanism improves the agent's ability to complete various subtasks. In addition, adding the page operation subtask contributes more to the agent's performance than adding the page reaching subtask. This is because the page operation subtask not only improves the agent's ability to operate pages, but also requires the agent to be able to reach the specified page.

\begin{table}
\begin{tabular}{l|c c c|c} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c}{Step-level} & \multicolumn{2}{c}{Task-level} \\ \cline{2-5}  & IoU & Text & IoU & Test \\ \hline MobileVLMsFT & 76.20\% & 74.08\% & 34.03\% & 28.43\% \\ + AI & 81.55\% & 79.86\% & 35.37\% & 30.01\% \\ + Al \& Re & 82.34\% & 80.49\% & 36.37\% & 29.64\% \\ + Al \& Op & 82.54\% & 81.22\% & 37.19\% & 32.87\% \\ + Al \& Re \& Op & **83.34\%** & 81.47\% & 37.82\% & 31.31\% \\ \hline ReachAgent & 83.32\% & **81.77\%** & **38.75\%** & **33.06\%** \\ \hline \end{tabular}
\end{table}
Table 4: Main Result(%) on Auto-UI dataset. The first column shows few-shot agents. The second column shows the SFT agents.

\begin{table}
\begin{tabular}{l r r r r|r r} \hline \multirow{2}{*}{Model} & \multicolumn{2}{c}{Reach SubTask} & \multicolumn{2}{c|}{Operate SubTask} & \multicolumn{2}{c}{All SubTask} \\ \cline{2-7}  & IoU Acc & Text Acc & IoU Acc & Text Acc & IoU Acc & Text Acc \\ \hline MobileVLMgTT & 51.44\% & 53.87\% & 56.62\% & 37.89\% & 53.19\% & 48.48\% \\ + action alignment & 53.90\% & 54.91\% & 53.97\% & 39.39\% & 53.92\% & 49.82\% \\ + action alignment \& page reaching & 55.79\% & 56.83\% & 59.57\% & 43.39\% & 57.03\% & 52.42\% \\ + action alignment \& page operation & 57.36\% & 58.24\% & 60.93\% & 46.42\% & 58.53\% & 54.37\% \\ + action alignment \& page reading \& page operation & 57.84\% & 58.94\% & 59.96\% & 43.33\% & 58.53\% & 53.82\% \\ \hline ReachAgent & **59.91\%** & **60.57\%** & **62.35\%** & **47.45\%** & **60.71\%** & **56.27\%** \\ \hline \hline \end{tabular}
\end{table}
Table 3: The task level accuracy (%) of different ablations for completing subtasks.

\begin{table}
\begin{tabular}{l|c c c|c} \hline Model & General & Install & GoogleApps & Overall \\ \hline GPT-4V & 43.01\% & 46.14\% & 49.18\% & 48.91\% \\ Qwen-VL Max & 46.22\% & 50.30\% & 49.16\% & 49.21\% \\ GPT-4o & 47.06\% & 49.12\% & 52.30\% & 52.04\% \\ \hline Llama z=plan & 53.77\% & 69.10\% & 61.19\% & 61.57\% \\ MobileAgent & 55.80\% & 74.98\% & 63.95\% & 64.51\% \\ Auto-Ulunified & 68.24\% & 76.89\% & 71.37\% & 71.71\% \\ CoCo-LAVA & 58.93\% & 72.41\% & 70.81\% & 70.72\% \\ CogAgent & 65.38\% & 78.86\% & **74.95\%** & 75.05\% \\ MobileVLM & 69.58\% & 79.87\% & 74.72\% & 74.99\% \\ \hline
**ReachAgent** & **70.27\%** & **80.76\%** & 74.94\% & **75.27\%** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablation study on the action alignment mechanism (AI), page reaching subtask (Re), page operation subtask (Op), and reinforcement learning.

 



### Case Study

Figure 5 shows two generation cases. MobileVLM tends to greedily choose task-related actions at each step, leading to local optima. In the first case, it goes directly to the "Public Transportation" page on the homepage, or clicks on "Running Routes" before finding nearby restaurants. In the second case, when the task is to search for a property that is about to be sold, MobileVLM keeps scrolling down, trying to browse directly to find the required property, and gets stuck in an endless loop. In contrast, ReachAgent improves the ability to reach and operate on task-related pages, and further completes the task based on these subtasks.

## 6 Conclusion

In this work, we propose ReachAgent, a two-stage framework that leverages page reaching and page operation tasks to enhance the agent's subtask completion abilities. Besides, we use a 4-level reward function to collect preference GUI flows, and further enhance the agent's overall task completion ability with reinforcement learning. In addition, we construct a mobile control dataset called MobileReach, which contains 3 categories of tasks: page navigation, page reaching, and page manipulation. Experimental results show that ReachAgent significantly improves IoU and Text accuracy by **7.12% and 7.69%** at the step level and **4.72% and 4.63%** at the task level, respectively.

Our ReachAgent shows strong task completion abilities by addressing the challenge of mobile AI agents focusing more on single-step action accuracy rather than completing the entire task flow. We hope that MobileReach can serve as a useful resource for breaking down tasks, solving page reach and operation subtasks, and provide assistance for future research.

### Limitations

Our training data is built on a graph consisting of GUI flows of 49 commonly used apps. Since the method of exploring the GUI graph is random walk, this may not cover all the functions of the app. In addition, we select GUI flows by random sampling, which may result in many invalid GUI flows that do not have corresponding tasks.

### Ethics Statement

This paper was conducted in accordance with the ACM Code of Ethics. Our MobileReach dataset is constructed using publicly available platforms and data sources, ensuring that there are no privacy issues or violations. All data used in our research was obtained following legal and ethical standards, and we do not collect any personally identifiable information.

Figure 5: Two cases of generated GUI flow by ReachAgent and MobileVLM.





## References

* [1]G. Baechler, S. Sunkara, M. Wang, F. Zubach, H. Mansoor, V. Etter, C. Carbune, J. Lin, J. Chen, and A. Sharma (2024) Screenai: a vision-language model for ui and infographics understanding.. arXiv preprint arXiv:2402.04615. Cited by: SS3.1.
* [2]C. Bai, X. Zang, Y. Xu, S. Sunkara, A. Rastogi, J. Chen, et al. (2021) Uibert: learning generic multimodal representations for ui understanding. arXiv preprint arXiv:2107.13731. Cited by: SS1.
* [3]H. Bai, Y. Zhou, M. Cemri, J. Pan, A. Suhr, S. Levine, and A. Kumar (2024) Digirl: training in-the-wild device-control agents with autonomous reinforcement learning. arXiv preprint arXiv:2406.11896. Cited by: SS1.
* [4]J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. (2023) Qwen technical report. arXiv:2309.16609. Cited by: SS1.
* [5]S. Bunian, K. Li, C. Jermali, C. Harteveld, Y. Fu, and M. Seif Seif El-Nasr (2021) Virus: visual search for mobile user interface design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-14. Cited by: SS1.
* [6]Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198. Cited by: SS1.
* [7]B. Cheng, R. Girshick, P. Dollar, A. C. Berg, and A. Kirillov (2021) Boundary iou: improving object-centric image segmentation evaluation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15334-15342. Cited by: SS1.
* [8]B. Deka, Z. Huang, C. Franzen, J. Hibschman, D. Afergan, Y. Li, J. Nichols, and R. Kumar (2017) Rico: a mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and technology, pp. 845-854. Cited by: SS1.
* [9]T. Ding (2024) Mobileagent: enhancing mobile control via human-machine interaction and sop integration. arXiv preprint arXiv:2401.04124. Cited by: SS1.
* [10]A. Dosovitskiy, L. Beyer, A. Kolessnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. Cited by: SS1.
* [11]W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y. Wang, Z. Wang, Y. Dong, M. Ding, et al. (2023) CGagNET: a visual language model for gui agents. arXiv preprint arXiv:2312.08914. Cited by: SS1.
* [12]Y. Hsiao, F. Zubach, G. Baechler, V. Carbune, J. Lin, M. Wang, S. Sunkara, Y. Zhu, and J. Chen (2022) SCreenqa: large-scale question-answer pairs over mobile app screenshots. arXiv preprint arXiv:2209.08199. Cited by: SS1.
* [13]L. A. Leiva, A. Hota, and A. Oulasvirta (2020) Enrico: a dataset for topic modeling of mobile ui designs. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 1-4. Cited by: SS1.
* [14]T. Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers (2021) Screen2vec: semantic embedding of gui screens and gui components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-15. Cited by: SS1.
* [15]Y. Li, J. He, X. Zhou, Y. Zhang, and J. Baldridge (2020) Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776. Cited by: SS1.
* [16]J. Liu, Y. Zhu, K. Xiao, Q. Fu, X. Han, W. Yang, and D. Ye (2023) RLIF: reinforcement learning from unit test feedback. arXiv preprint arXiv:2307.04349. Cited by: SS1.
* [17]X. Ma, Z. Zhang, and H. Zhao (2024) COCO-agent: a comprehensive cognitive mlm agent for smartphone gui automation. In Findings of the Association for Computational Linguistics ACL 2024, pp. 9097-9110. Cited by: SS1.
* [18]OpenAI (2023) GPT-4 technical report. Preprint. External Links: 2303.08774 Cited by: SS1.
* [19]R. Rafailov, A. Sharma, E. Mitchell, C. D Manning, S. Ermon, and C. Finn (2024) Direct preference optimization: your language model is secretly a reward model. Advances in Neural Information Processing Systems36. Cited by: SS1.
* [20]C. Rawles, A. Li, D. Rodriguez, O. Riva, and T. Lillicrap (2023) Android in the wild: a large-scale dataset for android device control. arXiv preprint arXiv:2307.10088. Cited by: SS1.
* [21]B. Shen, J. Zhang, T. Chen, D. Zan, B. Geng, A. Fu, M. Zeng, A. Yu, J. Ji, J. Zhao, et al. (2023) Pangu-coder2: boosting large language models for code with ranking feedback. arXiv preprint arXiv:2307.14936. Cited by: SS1.
* [22]G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. (2024) Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.



Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2words: Automatic mobile ui summarization with multimodal learning. In _The 34th Annual ACM Symposium on User Interface Software and Technology_, pages 498-510.
* Wang et al. (2024) Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. 2024. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents. _arXiv preprint arXiv:2410.14803_.
* Wang et al. (2023) Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023. Cogvlm: Visual expert for pretrained language models. _arXiv preprint arXiv:2311.03079_.
* Wu et al. (2024) Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, and Shuo Shang. 2024. Mobilevlm: A vision-language model for better intra-and inter-ui understanding. _arXiv preprint arXiv:2409.14818_.
* Yan et al. (2023) An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. 2023. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. _arXiv preprint arXiv:2311.07562_.
* Yang et al. (2023) Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. Appagent: Multimodal agents as smartphone users. _arXiv preprint arXiv:2312.13771_.
* Yuan et al. (2023) Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Zeng et al. (2023) Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023. Agenttuning: Enabling generalized agent abilities for llms. _arXiv preprint arXiv:2310.12823_.
* Zhan and Zhang (2023) Zhuosheng Zhan and Aston Zhang. 2023. You only look at screens: Multimodal chain-of-action agents. _arXiv preprint arXiv:2309.11436_.
* Zhang et al. (2024) Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. 2024. Android in the zoo: Chain-of-action-thought for gui agents. _arXiv preprint arXiv:2403.02713_.

 

Experiment Settings

### Baselines

ReachAgent was compared to four other baselines as follows: GPT-4o, Qwen-VL, Auto-UI, MobileVLM.

* GPT-4o, GPT-4V (OpenAI, 2023) are most advanced VLMs currently available.
* Qwen-VL, Qwen-VL-Max (Bai et al., 2023) are large-scale visual language models designed to perceive and understand text and images. They have demonstrated significant performance in tasks such as image captioning, question answering, visual or document visual question answering, and localization, and have been applied as the base model for multiple mobile AI agents.
* Auto-UI (Zhan and Zhang, 2023) is derived from AITW (Rawles et al., 2023), which uses a chain-of-action composed of a series of action histories and future action plans to improve the agent's action prediction ability.
* MobileVLM (Wu et al., 2024) builds a large-scale Mobile3M dataset in the graph structure and uses multiple pre-training tasks to enhance the agent's UI understanding ability.
* MobileAgent (Ding, 2024) builds a GUI agent based on prompt engineering based on GPT-4V.
* CoCo-LLaVA (Ma et al., 2024) leverages comprehensive environment perception (CEP) and conditional action prediction (CAP) to enhance the agent's understanding of GUI pages and tasks.
* CogAgent (Hong et al., 2023) is built on CogVLM, adding pre-training tasks and supporting higher-resolution images.

For in-context learning, we provided them with several few-shot examples. For SFT learning, we use the page navigation split of the MobileReach dataset to fine-tune them for two epochs. We maintain consistent hyperparameters across all the baselines for fair comparisons.

### Metric

We use two objective metrics to evaluate the model at the step-level:

**IOU Acc:** Intersection over Union (Cheng et al., 2021) evaluates if the bounding box of the generated action intersects with the golden action. We allow a 14% margin of error relative to the screen size.

**Text Acc:** It evaluates if the text in the generated action matches the golden one. This includes the name of the click action, the direction of the scroll action, the name and input text of the input action, and the text of the complete action. For the input text, which may have varied descriptions, we require an F1 value greater than 0.8 to be considered consistent. The remaining text must be entirely consistent.

We use two metrics to evaluate the model at the task-level:

**Task-level Acc**: As described in Section 5.2, task-level accuracy evaluates whether all actions in an entire GUI flow are correct. That is, all actions in the reasoning process are exactly the same as all actions in the ground truth. This is a very strict method for judging task success and has been applied in previous works.

**Task Success Rate**: This metric considers a GUI flow that completes all subtasks as correct. In our reward scoring principle, if the model can complete all subtasks, it does not need to be exactly the same as the ground truth to be considered complete. However, since the Auto-UI dataset does not contain explored paths other than the golden answer, we only use this evaluation metric in the Mobile3M dataset.

### HyperParameter

We present the hyperparameters for the first stage SFT framework and second stage RL framework in Table 6. ReachAgent chose MobileVLM as the backbone model. We use 8 80GB Nvidia A100 GPUs for fine-tuning. The learning rate is 1e-5, and the agent's max length is 4,096.

For the MobileReach dataset, in the first stage, the model was trained for 2 epochs on all three splits with a batch size of 4. In the second stage, it was trained for 2 epochs on 259,742 preference data with a batch size of 1. During testing, the max step is set to 15.

For the Auto-UI dataset, we fine-tuned the MobileReach-stage1 on its training split for 1 

epoch. Similar to its official method, we only used 10% of the GoogleApps data of AITW to save the training time. All SFT baselines are also fine-tuned for 1 epoch on this split with the same parameters. During testing, we removed WebShopping tasks because they are control tasks performed on a computer screen rather than a mobile screen.

## Appendix B Prompt

For the **page navigation task**, we use the few-shot prompt to guide the GPT-4 in building tasks, as shown in Figure 6. The inputs are image captions generated by Intern-VL and actions between GUI pages. The outputs are step-by-step description and brief task.

For the **page reaching task**, we use the following pre-defined task templates to build tasks. We replace the {text} in the template with page names.

Figure 6: The prompt for step-by-step description and brief task generation.

 



\begin{tabular}{|l|c|c|} \hline Hyperparameter & Stage 1 & Stage 2 \\ \hline epoch & 2 & 2 \\ batch\_size & 4 & 1 \\ learning\_rate & 1e-5 & 1e-5 \\ warmup\_ratio & 0.02 & 0.02 \\ optimizer & Adam & Adam \\ max\_sequence\_length & 4096 & 4096 \\ GPUs & 8 & 8 \\ \hline \end{tabular}

Table 6: Hyperparameters.



7. On the parameter page, select "white" to confirm the parameters of the phone.
8. On the parameter page, click the "Confirm" button at the bottom to confirm the parameter configuration of the phone.

**Brief Task:**

Help me find detailed information about xiaomi 14, and add a white one to the shopping cart.

Here, each step \(s_{i}\) corresponds to a GUI Page \(P_{i}\) and an action \(a_{i}\) in the GUI flow.

**Filter:**

We filter out some tasks and GUI flows with low quality. For example:

A. The brief task already appears in the current training set.

B. The number of steps in the distribution description is inconsistent with the number of steps in the GUI flow.

Figure 7: Page reaching subtasks and page operation subtasks extracted from the GUI flow example in Figure 3.

 

C. The brief task is too long.

### Subtask Generation

After obtaining the GUI flow, step-by-step description, and a brief task, we can split them into page arrival and page operation subtasks. This splitting operation is mainly based on the step-by-step description and actions.

**Page Reaching Subtask Generation**

1. If a page is referred to by a unique name in the step-by-step description, we split out the sub-page flow leading to that page and assign a task through template to reach that page. For example:

Description: 4. On the search results page, select the detailed information of "xiaomi 14".

Task: Visit the search result page.

GUI Flow: \(P_{1}\) > \(a_{1}\) > \(P_{2}\) - \(a_{2}\) - \(P_{3}\) - \(a_{3}\) - \(P_{4}\)

2. If the element name of a click action does not appear in the existing dataset, we name the next page of clicking on the element with that element name and assign a task to reach that page. For example:

Action: \(a_{4}\), click("Xiaomi 14", [425,1074][628, 1125]).

Task: Help me navigate to "Xiaomi 14" interface.

GUI Flow: \(P_{1}\) -> \(a_{1}\) -> \(P_{2}\) -> \(a_{2}\) -> \(P_{3}\) -> \(a_{3}\) -> \(P_{4}\) -> \(a_{4}\) -> \(P_{5}\)

**Page Operation Subtask Generation**

1. If the action type is scroll or input, we assign an operation task as referenced in the step-by-step description, including reaching the current page and performing this action.

Action: \(a_{6}\), scroll([0,585][720,1088], "up").

Task: On the parameter page, scroll the page to view more parameter information.

GUI Flow: \(P_{1}\) -> \(a_{1}\) -> \(P_{2}\) -> \(a_{2}\) -> \(P_{3}\) -> \(a_{3}\) -> \(P_{4}\) -> \(a_{4}\) -> \(P_{5}\) -> \(e_{5}\) -> \(P_{6}\) -> \(a_{6}\) -> \(P_{7}\)

2. If an action has similar GUI pages before and after execution, we assign an operation task according to the step-by-step description, describing the process of reaching this page to perform this action.

Similar Pages: \(P_{7}\), \(P_{8}\)

Task: On the parameter page, select "white" to confirm the parameters of the phone.

GUI Flow: \(P_{1}\) -> \(a_{1}\) -> \(P_{2}\) -> \(a_{2}\) -> \(P_{3}\) -> \(a_{3}\) -> \(P_{4}\) -> \(a_{4}\) -> \(P_{5}\) -> \(a_{5}\) -> \(P_{6}\) -> \(a_{6}\) -> \(P_{7}\) -> \(a_{7}\)