CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation (2024)

Abstract: Multimodal large language models (MLLMs) have shown remarkable potential as human-like autonomous language agents to interact with real-world environments, especially for graphical user interface (GUI) automation. However, those GUI agents require comprehensive cognition including exhaustive perception and reliable action response. We propose a Comprehensive Cognitive LLM Agent, CoCoAgent, with two novel approaches, comprehensive environment perception (CEP) and conditional action prediction (CAP), to systematically improve the GUI automation performance. First, CEP facilitates the GUI perception through different aspects and granularity, including screenshots and complementary detailed layouts for the visual channel and historical actions for the textual channel. Second, CAP decomposes the action prediction into sub-problems: determining the action type and then identifying the action target conditioned on the action type. With our technical design, our agent achieves state-ofthe-art performance on AITW and META-GUI benchmarks, showing promising abilities in realistic scenarios. Code is available at https: //github.com/xbmxb/CoCo-Agent.

heading: 1 Introduction
text: Graphical user interface (GUI) automation aims to enable human-like operations on operating systems with artificial intelligence instead of human efforts. Large language models (LLMs) have demonstrated commendable performance as human-like agents, showing emergent abilities of perceiving (Yao et al., 2022), reasoning (Li et al., 2023b; Park et al., 2023), and acting (Wang et al., 2023; Richards, 2023). With the multimodal enhancement, MLLM agents
* Corresponding authors. This research was supported by the Joint Research Project of Yangtze River Delta Science and Technology Innovation Community (No. 2022CSJGG1400), the Joint Funds of the National Natural Science Foundation of China (Grant No. U21B2020).
become promising autonomous GUI assistants to deal with complex tasks on behalf of human operators. To interact with the GUI environment, those agents require comprehensive cognition, including exhaustive perception and reliable action response.
Current vital challenges for autonomous agents lie in two aspects. One is (i) the dependence on strong (M)LLMs, and the other is (ii) the insufficient GUI environment modeling.
Although strong (M)LLMs like GPT-4V (OpenAI, 2023) and ChatGPT (Ouyang et al., 2022) ignite the development of autonomous agents, they exhibit shortcomings in realistic use. First, the alignment requires a trustworthy domain transfer as there is a large disparity between GUI commands and natural languages. GUI agents are expected to generate accurate and well-formulated responses as executable GUI commands, which is non-trivial for zero/few-shot prompting. For example, given the GUI that accepts commands as “{action: click, touch_point:[y0, x0], touch_point:[y1, x1], typed_text: ‘’}”, semantically equivalent generations like “Open the address book on your phone” is plausible but unavailable. Second, the black-box APIs are likely to cause unexpected safety issues. Risks to privacy and integrity may arise when granting personal device authority to a black-box API. This significantly reduces realistic usability. Third, agent performance relies on the prompt design. The issues mentioned above leave heavy burdens on the design of prompt lines for those agents. Besides necessary environment descriptions, the prompts (and post-processing) must be sophisticated for domain alignment, instruction following, and security risk mitigation in different circumstances.
Second, GUI agents necessitate a comprehensive multimodal perception for the informative GUI environment modeling. Existing methods for visual language models are mainly endowed with favorable abilities in semantic alignment between the
1 9097
vision and language modalities (Dai et al., 2023; Ye et al., 2023; Zhao et al., 2023). However, GUI contains fine-grained details and intricate semantic connections, presenting a challenge for agents to comprehend (Rawles et al., 2023; Li et al., 2023a). Consider a screenshot that includes a magnifier icon, where the conventionally accepted meaning of “search” is conveyed. It implies a potential action through implicit semantics despite its small pixel size. Thus, only leveraging general image semantics like captioning is insufficient for GUI environment modeling. In addition, the GUI environmental perception is limited by the finite input window, where a balance between the visual and textual feature length needs to be struck.
This work proposes CoCo-Agent, a Comprehensive Cognitive MLLM Agent, to address the challenges above for smartphone GUI automation. CoCo-Agent adopts a multimodal backbone of LLaVA (Liu et al., 2023) and further enhances comprehensive cognition, respectively for exhaustive perception and reliable action response. The two proposed approaches are comprehensive environment perception (CEP) and conditional action prediction (CAP). Specifically, CEP integrates GUI perception elements of textual goal, historical action, and high-level and detailed description of the vision channel. CAP decomposes the complex and redundant GUI action commands into sub-problems following a top-down order. Our experiments cover diverse tasks in two GUI benchmarks, AITW (Rawles et al., 2023) and META-GUI (Sun et al., 2022a), including application manipulation, web operation, and dialogues. CoCo-Agent achieves SOTA performance with a limited parameter size. Subsequently, we present deep analyses including element ablation, visual module selection, and future action prediction. We show the significant effect of each perception element and the favorable choice of visual module. We also analyze the limitations of existing datasets and illustrate the additional potential of CoCo-Agent for realistic scenarios.
Our contributions are summarized as follows: ◦ We propose CoCo-Agent, an autonomous agent with comprehensive cognition for GUI, with novel approaches to enhance the perception and action response, namely comprehensive environment perception (CEP) and conditional action prediction (CAP).
◦ CoCo-Agent achieves state-of-the-art perfor-
mance on representative GUI benchmarks, demonstrating superior performance. ◦ Extensive analyses for a systematic study of GUI automation demonstrate our significant effectiveness and realistic potential.

heading: 2 Related Work
text: This section introduces studies on autonomous language agents and multimodal perception of LLMs.

heading: 2.1 Autonomous Language Agents
text: Recent studies (Li et al., 2023b; Richards, 2023) use the term language agent to refer to language models that interact with an environment or other agents to solve a problem. This paper investigates the autonomous language agents that perceive the environment and then act on the environment.
One research line relies on the strong fundamental competencies of (M)LLMs. Based on ChatGPT or GPT-4, autonomous agents can be built by only well-written prompts. Existing works have proved the reasoning, planning, and generalizing abilities of GPT-based agents, e.g., AutoGPT (Richards, 2023), BabyAGI (Nakajima, 2023), AgentGPT (Reworkd, 2023), HuggingGPT (Shen et al., 2023), and MM-Navigator (Yan et al., 2023).
However, when we expect practicality and reliability, we pursue the trainable language agent that can be customized and privatized to align with given environments (Shao et al., 2023). Thus, another research line turns to training methods for open-source language models. m-BASH (Sun et al., 2022b) adopted ROI pooling to present GUI icons in a BERT-based multi-task system. The Auto-GUI (Zhang and Zhang, 2023) was trained on a multimodal T5 (Raffel et al., 2020), formulating the GUI interaction to a first-principal form. CogAgent (Hong et al., 2023) integrated an extra attentionbased high-resolution visual module with alignment pre-training. This paper follows the second research line to discuss the trainable, open-source language agent.

heading: 2.2 Multimodal Perception
text: Beyond language modeling, recent works have studied the fusion with channels of other modalities. Because of the development of LLMs, mainstream methods usually follow a language-centric framework, i.e., encoding information of other modalities into the language embedding space. These models consist of a pre-trained encoder of other
2
modalities, a language model, and an adapter (or a projector) as the bridge. For example, LLaVA (Liu et al., 2023) uses a linear layer to map the vision encoding from CLIP, while BLIP-2 (Li et al., 2023c) adopts a Q-former to learn a query vector to represent the image. This endeavor has given rise to the emergence of various multimodal LLMs, such as Flamingo (Alayrac et al., 2022), mPLUG (Ye et al., 2023), MiniGPT-4&v2 (Zhu et al., 2023; Chen et al., 2023), Video-LLaMA (Zhang et al., 2023b), and SpeechGPT (Zhang et al., 2023a).
However, the multimodal perception is even more challenging for GUI agents. Because GUI contains extensive detailed information with intricate semantic connections, such as very small icons conveying customary meanings (shown in Figure 1). A gap remains between existing visual LLMs and the perception necessitated for GUI agents.

heading: 3 Methodology
text: In this section, we will first formulate the GUI automation task and then propose our CoCo-Agent. Concretely, we will describe our technical designs of cognition, namely, comprehensive environment perception (CEP) and conditional action prediction (CAP), to improve the GUI automation performance systematically. Figure 1 shows an overall illustration.

heading: 3.1 Task Formalization
text: The task of GUI automation is defined as an interactive sequence generation problem. First, the user instructs the agent with a goal g that can be achieved in the GUI environment in several steps. At each step, the agent first perceives the present GUI state, st, and predicts the next action at, lead-
ing to the next GUI state st+1. The sequential (s, a) that accomplishes a goal forms an episode. An interaction record is formulated as
EPISODE = (g, [(st, at)] n t=1). (1)
The action space is a finite operation command set with limited parameters. Examples are illustrated in Table 1. The state space includes any possible display from the smartphone. As the output recipient of agents is not human but a GUI, accurate actions are expected instead of flexible expressions like natural language.

heading: 3.2 Backbone
text: Our backbone follows LLaVA (Liu et al., 2023), which uncovers the generalization of LLM to vision modality. LLaVA consists of a language model (LM), Llama-2-chat-7B (Touvron et al., 2023), a vision encoder (ENCODERimage), CLIP (Radford et al., 2021), and a one-layer linear projector (PRJ) to bridge the image features to the space of language embedding (EMBEDtext). The input is denoted as X , including text Xtext and image Ximage, The output is denoted as Y . The backbone can be formulated as
Htext = EMBEDtext( Xtext ◦ Ŷ 0:t−1 ), Zimage = ENCODERimage( Ximage ),
Himage = PRJ( Zimage ),
HDecodert = DECODER( Himage ◦Httext), Pt = LMhead( HDecodert ), L = ∑
t
CE( Pt, Yt ),
(2)
where ◦ denotes the concatenation operation. The training objective L is cross entropy (CE).
3

heading: 3.3 Comprehensive Environment Perception
text: Environment perception is a crucial prerequisite for action responses. The environment can be simplified to only a GUI screenshot (Zhang and Zhang, 2023), which is highly subject to the upper-bound ability of the vision encoder. However, there is a bottleneck for the vision channel. First, the size of the encoder is restricted to a relatively low resolution, e.g., 224 × 224. Second, the existing pre-training objectives on vision encoders mainly focus on general, high-level semantic modeling, like image captioning (Radford et al., 2021; Li et al., 2023c). Thus, fine-grained information on the screen needs to be enhanced to complement the high-level perception.
Our proposed comprehensive environment perception fully leverages tools like optical character recognition (OCR) and IconNET (Sunkara et al., 2022), which gives fine-grained layouts with readable textual hints, e.g., “ICON_SETTINGS: [0.1783, 0.8701]”. Besides the global goal, g, the environment state is perceived from three aspects, the present screenshot, Ximage, the layouts from OCR, L, and the previous actions in the present episode, at−h:t−1. The total input can be noted as
Xtext = PROMPT( g, L, at−h:t−1 ), Ximage, (3)
where PROMPT is our prompt template (Appendix A). h denotes the number of action histories involved. The layouts L are listed (item name, item coordinate), where items are OCR results or icons.

heading: 3.4 Conditional Action Prediction
text: Regarding action response, we propose to refactor GUI actions following the thinking order. As is shown in the left part of Table 1, existing GUI actions involve redundant parameters of each command, including the action type, the beginning coordinates, the ending coordinates, and the possible input text. However, these parameters are not inde-
pendent of each other but show significant relations. For example, the coordinates depend on the action type. If the action is to click on an icon, then the touch and lift coordinates are accordingly determined. Predicting such JSON-formatted parameters would cause inconsistency and waste efforts.
Thus, we propose conditional action prediction. The GUI actions are refactored for relation decomposition as illustrated in Table 1. The actions are decomposed into two sub-problems to address, (i) action type prediction and (ii) optional action target prediction conditioned on the action type prediction. Also, we use natural language-like expressions without compromising the accuracy. As illustrated in Table 1, we change the action to a prompt line step-by-step, explicitly decomposing and clarifying those actions. Notably, the dual_point action is refined into three types: (i) scroll action, if the beginning and ending points are farther apart than the threshold (Rawles et al., 2023); (ii) click action involving item name, if the tap point falls in a bounding box; (iii) tap action, if it is not a scroll action but matches no bounding box.
In this way, the action prediction follows a topdown order. First, the agent decides on action types, conditioned on which the agent further decides on the target item and coordinates. Normalization. Based on CEP and CAP, the actions are normalized to alleviate noise, which is inevitable in real-world data. Specifically, the target coordinates of click actions are normalized to the centroid of the bounding box from OCR. The scroll actions are normalized into four-direction swipes (Zhang and Zhang, 2023).

heading: 4 Experiments
text: This section will introduce the experimental settings including the dataset, implementation details, and baselines, followed by our empirical results.
4

heading: 4.1 Dataset
text: The following benchmarks of GUI automation are considered in the empirical evaluation. The dataset statistics are presented in Table 2.
AITW (Rawles et al., 2023) is a benchmark for smartphone GUI, containing 715K operation episodes under 30K reality intentions. Each entry includes a goal in natural language, screenshots, and actions as Eq. 1. Humans collect the data on various devices and operation systems in various screen resolutions. According to the applications domain, AITW consists of five subsets: General, Install, GoogleApps, Single, and WebShopping.
META-GUI (Sun et al., 2022b) smartphone dataset is originally released for multimodal dialogue generation. Differently, the agent is enabled to communicate with the user to verify the present state or further operation, e.g., “Is this good?”. Eq. 1 is expanded with optional utterances,
(st, at) → (st, at, uagentt , uusert ), (4) where utterances from the agent and the user are denoted as uagent and uuser. These utterances cut an episode into several dialogue turns. METAGUI consists of 1k episodes with 18k steps. The data diversity lies in 11 applications of 6 domains including weather, calendar, search, etc.

heading: 4.2 Implementation
text: Our implementation adopts LLaVA (Liu et al., 2023) with a LLaMA-2-chat-7B and a vision encoder, CLIP.1 The maximum input length is 2048 following the vision instruct tuning. For the subsets of AITW, our experiments include two setups, i.e., separate training on each subset and unified training on the whole set. Details are shown in A. Metrics. Accuracy is computed at each time step of all parameters as our metric. The refactored action is parsed to JSON format and each parameter
1openai/clip-vit-large-patch14.
is compared to the action label following existing work (Rawles et al., 2023). The predicted coordinate is considered correct if it falls in the same element bounding box as the labeled one or falls within a 14% screen distance from the labeled one. A scroll action is correct if its main direction is correct. The accuracy of other parameters are exact match except for typed_text or dialogue responses. The typed text of AITW is correct if the label is in the predicted text. For META-GUI, F1 is computed for input text, and BLEU is computed for response generation. One action is regarded as correct if all the JSON fields are correctly predicted.

heading: 4.3 Baselines
text: For AITW, our proposed approach is compared with the following baselines.
• Uni-modal API-based methods. Rawles et al. (2023) and Zhang and Zhang (2023) have evaluated 5-shot performance on PaLM-2 (Anil et al., 2023) and ChatGPT (Ouyang et al., 2022). The images are represented by pseudo HTML codes. The action target prediction is the item name or index, without verifying the coordinate numbers. • Multimodal API-based methods. MMNavigator (Yan et al., 2023) is a GPT-4V-based multimodal agent, achieving few-shot SOTA. • Training-based methods. (i) Behavioural Cloning (Rawles et al., 2023) is a Transformerbased specialized agent that models goals, screens, and historical information using a BERT (Devlin et al., 2019). (ii) LLaMA-2 is shown as a representative trainable uni-modal LLM with pseudo HTML code inputs instead of images. The results are from Zhang and Zhang (2023). (iii) AutoGUI (Zhang and Zhang, 2023) bases on a multimodal encoder-decoder language model with T5 and BLIP. (iv) CogAgent (Hong et al., 2023) is a 9B-parameter visual LLM pre-trained for specializing in GUI understanding with a novel highresolution cross module, which tops on AITW.
For META-GUI, we present baselines following Sun et al. (2022b) including LayoutLMs (Xu et al., 2020, 2021), BERT, and m-BASH (Sun et al., 2022b). All of those need training. m-BASH achieves SOTA which is a multi-task Transformer with Faster R-CNN (Ren et al., 2015) and ROI pooling for vision modeling.

heading: 4.4 Main Results
text: Tables 3 and 4 present the main experimental results. Our method surpasses the baselines sig-
5
nificantly and achieves overall state-of-the-art performance (except for Single subset). The unified model shows consistent advances compared to the separate training, indicating that the model learns generality across various situations.
The lower part of Table 3 shows the detailed performance on AITW. CoCo-Agent is enabled to mimic the behavior patterns on GUI, while the limitations lie in predicting target items and scroll directions. (i) The action type scores achieve around 90%. This high level indicates that the agent can learn the action patterns to operate GUI. The lower CoT type scores indicate that it is harder for the agent to differentiate dual point actions. This is reasonable as these action types are more flexible than others whose effects are more definite (like
press back and press home). (ii) Predicting items (the locations to click) and directions are more difficult for agents. Especially, the item accuracy is close to the action accuracy. (iii) Input prediction is relatively easy, and F1 scores are up to 97%.
On META-GUI, we can also observe a significant improvement in action accuracy (12%). The accuracy of the item increases by a very large margin, while the action type and swipe direction accuracy is close to the perfect score. The input and utterance predictions are relatively consistent.

heading: 5 Analysis
text: This section presents further analysis and discussions. Section 5.1 shows the effects of comprehensive cognition elements by ablation and replace-
6
ment. Section 5.2 discusses the capability of visual modules, followed by future action prediction in Section 5.3. Dataset features are analyzed in Section 5.4, including action type categories and human evaluation for realistic scenarios.

heading: 5.1 Effects of Environment Elements

heading: 5.1.1 Ablation
text: Our method combines CEP and CAP to characterize the GUI environment. We ablate each CEP element along with the refactoring method of CAP to observe their significance for the CoCo-Agent. Results are shown in Table 5. The improvement of each element is significant, especially for layouts (+5.82%) and action history (+5.63%).
Besides the coordinates, the layouts provide the item names like icon names and texts shown on the screen. When combined with CAP, they bridge the prediction through rationales, making predictions easier than direct coordinate grounding. Notably, although no historical screens are provided, historical actions with complete parameters lead to better scores than historical action types.

heading: 5.1.2 Replacement
text: Actions on GUI can be flexible and the dependency on the environment is complex. Thus, we design a related task to probe the environment modeling further. We randomly choose one element from the goal, the screen, the layouts, and the action history and replace it with another from a random data point. With such corrupted input, the agent is trained to select the replaced element and also predict the next action.
The results are consistent with ablations and intuitions. (i) The wrong image and goal are more obvious replacements and get higher detection accuracy, while the layouts and action history are more complex to distinguish. (ii) Regarding action prediction, the accuracy decreases more with wrong layouts or wrong action history. Those are hard to memorize and require complex modeling, therefore rely more on correct inputs. (iii) The damage caused by a wrong image is limited due to the complement from layouts. This suggests again that layouts are important fine-grained complements for the screen image, while the image gives an overall impression and the layouts describe detailed information.

heading: 5.2 Visual Capability
text: The vision encoder and projector largely influence the visual capability of GUI agents. We compare a range of visual LMs with various vision encoders.
The results are shown in Table 7. Our integrated CLIP with projector encodes an image to a 256- length vector with a 4096 hidden size.1 With finegrained layouts, CoCo-Agent receives exhaustive visual information. Auto-GUI uses a BLIP-2 with pooling leading to a 1-length image vector, which is integrated into language embedding by an attentionbased fusion module. Differently, MMICL adopts Q-Former to learn a 32-length query vector, and mPLUG adopts Abstractor to learn a 64-length vector. 2 CogAgent uses an EVA2-CLIP-E (Sun et al., 2023) as a low-resolution encoder and an EVA2CLIP-L (Sun et al., 2023) with cross-attention layers as a high-resolution encoder. In cross-attention
2The first 1000 samples in General, the most difficult subset of AITW.
7
layers, the high-resolution image features interact with each layer of the language model.
The models with only an image encoder outperform those with learnable queries. Learning queries from image-text attention can be unsuitable for GUI tasks, as input texts are complex and different from generic captions. Besides the overall semantic impression of images, straightforward textual layouts can work as an even better highresolution module for image detail enhancement.

heading: 5.3 Future Actions
text: This section considers a more challenging setting of n−next actions prediction. The task is much less trivial as the agent only receives the environment state st, without the perception of future states st+1:t+n−1. Thus, predicting future actions at:t+n−1 involves harder reasoning, planning, and environment simulation. Table 8 shows the results with n = 3. Although the next action can be predicted with 70.96% accuracy, predicting the following actions without environmental feedback remains to be improved.

heading: 5.4 Dataset Features

heading: 5.4.1 Action Type Categories
text: Each action type has very different proportions in AITW, which is decided by the unbalanced distribution in natural operations. For example, the click action is the most frequent but the complete action appears at most once in each episode. Thus, we divide datasets into categories according to the ground truth action type. Table 9 shows the proportion and action accuracy. (i) Dual point action (including click, tap, and scroll) accounts for 69.26% - 86.09% in long-episode tasks and accounts for around half even in the Single subset. Other types, especially press back and press enter consistently
account for low proportions. Such an unbalance can limit the performance of less frequent actions. (ii) For dual point type, the data is sufficient and type accuracy scores are up to above 90%. The action accuracy scores are limited by the difficulties in predicting target items and directions as shown in Table 3. (iii) Single subset shows better performance on all action types and more significant on less frequent actions. This is because the samples in Single are segmented sub-goals that give clear instructions and require fewer steps.

heading: 5.4.2 Potential for Realistic Scenarios
text: There is a disparity between the evaluation and realistic scenarios. The benchmarks show randomness because the actions can be stochastically chosen with different paths for the same goal. However, when the predicted action indicates an alternative path, it is reasonable but fails to match the label. This leads to an underestimation. Thus, the agent has extra potential in practice.
We study the first 500 samples of General dataset to see this phenomenon. In the first 500 actions, there are 118 (23.6%) actions whose predictions are different from the labels. Our human evaluation considers two criteria. We check the predicted action with the goal, the last action, the present screen, and the next screen to see if (i) the predicted action can result in a situation that is similar to the next screen, or (ii) the predicted action is consistent with the goal. We observed that only 54 (10.8%) samples strictly contradict the episode, while other 64 predictions do not betray the goal. Among the 64 samples, there are 25 (5%) predicted actions that can lead to a similar next state. For example, after typing a query in the search bar of a search engine, many actions can lead to the search results including press enter, click the magnifier icon, or click a proper query suggestion (Figure 3 and 4).
8

heading: 6 Conclusion
text: This paper proposes CoCo-Agent, an MLLMbased autonomous agent for smartphone GUI. Our method facilitates comprehensive cognition with exhaustive perception and reliable action response. CoCo-Agent is enhanced with two approaches. Comprehensive environment perception (CEP) integrates GUI perception, and conditional action prediction (CAP) enables a decomposition of complex commands. CoCo-Agent achieves SOTA performance in experiments on two GUI datasets. Further analysis shows that the agent learns the behavior patterns in smartphone GUI. The significance of the proposed enhancements is also verified. We discuss the unbalanced category distributions and the underestimation of agent performance. Our work reveals the promising capabilities of MLLMs as autonomous agents, especially for complex environment perception and action response.
Limitations
We acknowledge the limitations of this work. (i) Resource consumption. The data is of a large amount. The training process costs computational resources compared to the zero-shot methods. Whereas, the training process only needs to be conducted once. And our unified model achieves the SOTA performance. Efficient LLM training or inference methods can improve the balance between resource cost and performance. (ii) More complex settings. As is shown in Section 5.3, the predictions of future action remain to be improved. The ultimate goal is to empower the agent to operate the full episodes in a simulated GUI environment or a realistic device.
Future Work
The following directions for future work are proposed based on our study. First, GUI agents require generalization ability to support new instructions, new applications, or even new operating systems. The effectiveness of CoCo-Agent in task adaptation across domains of AITW has been preliminarily validated. Second, GUI agents require improved multimodal training strategies. Multimodal LLMs can be strengthened by integrating GUI perception into vision-language pre-training or instruction tuning. Third, GUI agents require comprehensive measurements. The measurements can be improved to reflect different paths for the same goal in practical scenarios. Last but not least, there is still room for
performance improvement (79.05% for AITW and 88.27% for META-GUI) and future action reasoning and planning for full-episode prediction.
Ethics Statement
This section presents ethics statements in the following aspects. (i) Data Privacy. The research dataset AITW (Rawles et al., 2023) has declared control over Personal Identifiable Information. The instructions contain no toxic, biased, or misleading content. The META-GUI is based on a taskoriented dialogue dataset, SMCalFlow (Andreas et al., 2020), crowdsourced on Amazon Mechanical Turk with qualification requirements. CoCoAgent does not rely on LLM APIs, preserving privacy data from leakage to LLM companies. (ii) System security. Compared to the command-line interface (CLI), the GUI is more interpretable and controllable. Since GUI actions follow human behavior, security considerations can better align with human-oriented mechanisms, which are already implemented in existing GUIs for operating systems. However, the potential risks of GUI agents have yet to be well explored (Yuan et al., 2024; Yang et al., 2024). (iii) Potential social impacts. On the one hand, GUI automation can facilitate efficiency and save human resources for more high-level work. On the other hand, malicious actors could abuse GUI agents to achieve undesirable purposes. For practical applications of GUI agents, platforms may need to update detection, authorization, and governance mechanisms to control potential risks for social impacts.

heading: B Examples
text: We show examples of a full episode in Figure 2. Examples of the randomness in AITW are illustrated here. Figure 3 is a case where the predicted action leads to the same situation but does not match the label (i, in Section 5.4.2). Figure 4 is a case where the predicted action is different from the label but is still reasonable for the goal (ii, in Section 5.4.2). Blue boxes highlight the target items on the screen for label actions and orange boxes for predicted actions.
12
13
14

heading: Future Work
text: 3, the predictions of future action remain to be improved. The ultimate goal is to empower the agent to operate the full episodes in a simulated GUI environment or a realistic device. Future Work
The following directions for future work are proposed based on our study. First, GUI agents require generalization ability to support new instructions, new applications, or even new operating systems. The effectiveness of CoCo-Agent in task adaptation across domains of AITW has been preliminarily validated. Second, GUI agents require improved multimodal training strategies. Multimodal LLMs can be strengthened by integrating GUI perception into vision-language pre-training or instruction tuning. Third, GUI agents require comprehensive measurements. The measurements can be improved to reflect different paths for the same goal in practical scenarios. Last but not least, there is still room for
performance improvement (79. 05% for AITW and 88. 27% for META-GUI) and future action reasoning and planning for full-episode prediction