You Only Look at Screens: Multimodal Chain-of-Action Agents (2024)

Abstract: Autonomous graphical user interface (GUI) agents aim to facilitate task automation by interacting with the user interface without manual intervention. Recent studies have investigated eliciting the capabilities of large language models (LLMs) for effective engagement in diverse environments. To align with the input-output requirement of LLMs, most existing approaches are developed under a sandbox setting where they rely on external tools and application-specific APIs to parse the environment into textual elements and interpret the predicted actions. Consequently, those approaches often grapple with inference inefficiency and error propagation risks. To mitigate the challenges, we introduce Auto-GUI, a multimodal solution that directly interacts with the interface, bypassing the need for environment parsing or reliance on applicationdependent APIs. Moreover, we propose a chainof-action technique—leveraging a series of intermediate previous action histories and future action plans—to help the agent decide what action to execute. We evaluate our approach on a new device-control benchmark AITW with 30K unique instructions, spanning multi-step tasks such as application operation, web searching, and web shopping. Experimental results show that Auto-GUI achieves state-of-the-art performance with an action type prediction accuracy of 90% and an overall action success rate of 74%. Code is publicly available at https://github.com/cooelf/Auto-GUI.

heading: 1 Introduction
text: Building intelligent autonomous agents that are capable of task planning, decision making, and action execution in a particular environment is a longstanding goal of artificial intelligence (AI) (Searle, 1969; Wooldridge and Jennings, 1995; Maes, 1995; Hendler, 1999). The advent of large language models (LLMs) (Brown et al., 2020; Chowdhery et al.,
*Work done at Amazon Web Services.
2022; OpenAI, 2023) has flourished promising opportunities for developing autonomous agents to assist users in completing tasks in distinct environments such as operation systems, specific applications, and web browsers (Adept, 2022; Rawles et al., 2023; Liu et al., 2023; Zhou et al., 2023; Wang et al., 2023c; Koh et al., 2024; Gao et al., 2023; Yang et al., 2023).
Recent studies have explored prompt engineering (Richards, 2023; Nakajima, 2023; Reworkd, 2023; Sumers et al., 2023; Liu et al., 2023) and fine-tuning techniques (Rawles et al., 2023; Wen et al., 2023; Sun et al., 2022) to elicit the capability of language models to execute actions in interactive environments. However, there are at least two major challenges that have limited real-world applications of autonomous agents.
First, existing approaches commonly rely on external tools such as optical character recognition (OCR) and icon detectors (Zhang et al., 2021; Sunkara et al., 2022) to parse the environment into textual elements (e.g., HTML layouts) as inputs to a language model (Figure 1(a)) (Rawles et al., 2023). On the one hand, the parsed elements generate lengthy inputs, thus leading to inference inefficiency. Since computational latency is a key measure in deployment, using lengthy inputs would increase inference cost and may even exceed the input length limit of the language model. On the other hand, parsing the visual environment into textual elements may also be prone to error propagation or information loss because parsing mistakes are inevitable using external tools.
Second, most existing approaches are under the sand-box setting that requires accessing internal APIs to interact with the environment (Zhou et al., 2023; Gur et al., 2023), e.g., using a JavaScript element selection on a webpage or a Python interpreter to execute actions. However in practice, the API interface is often inaccessible in third-party applications (Apps).
3132
These challenges have motivated more advanced techniques that are capable of first principles thinking (Aristotle; Irwin, 1989)—allowing direct interactions on the screen without needing access to intermediate environment parsing or interval application-dependent APIs (Figure 1(b)). To address the challenges, we introduce Auto-GUI, a multimodal approach that directly interacts with the graphical user interfaces (GUIs). To further strengthen the agent’s action prediction capability, we propose a novel chain-of-action technique, where a chain of action is a series of intermediate previous action histories and future action plans.
We evaluate Auto-GUI on a new device-control benchmark AITW (Rawles et al., 2023) with 30K unique instructions, spanning multi-step tasks of application operation, web searching, and web shopping. Experimental results show that AutoGUI achieves state-of-the-art performance with an action type prediction accuracy of 90% and an action success rate of 74%.
In summary, our work makes the following technical contributions:
(i) We introduce Auto-GUI, a multimodal agent for autonomous GUI control that can directly interact with the screens, thus circumventing the constraints of environment parsing and applicationspecific API access.
(ii) We propose a chain-of-action technique that leverages the previously executed actions and future action plans to help the agent decide what action to execute at each step.
(iii) Auto-GUI achieves state-of-the-art perfor-
mance with an action type prediction accuracy of 90% and an action success rate of 74%. Notably, Auto-GUI is able to infer an action in less than one second.

heading: 2 Related Work
text: Our work falls into the field of language agents. This section will first review the recent progress in building language agents and then discuss the approaches to conduct user interface control with language agents.

heading: 2.1 Language Agents
text: Language agents refer to those agents that can follow user instructions and interact with environments to complete tasks. Such agents expand the landscape of language models to compete in specific fields, including application operation, web searching, and web shopping. There are two popular types of language agents, autonomous agents and communicative agents. Autonomous agents aim to assist humans to achieve specific goals in the real world. Typical examples of autonomous agents are AutoGPT (Richards, 2023), BabyAGI (Nakajima, 2023), and AgentGPT (Reworkd, 2023). In contrast, communicative agents are personalized and socialized agents (Park et al., 2023; Wang et al., 2023b; Zhu et al., 2023; Hong et al., 2023a) with human behaviors that can communicate and collaborate with each other. They are often deployed in immersive environments.
Inspired by the potential in real-world applications, this work focuses on autonomous agents, es-
pecially those working in mobile devices. We aim to assist users by completing multi-step tasks (e.g., manipulating Apps, web shopping, and question answering) without any manual intervention. Given a user instruction in natural language, the agent is required to interpret the instruction and execute actions by directly controlling its user interface. Due to the requirement in real-world applications, the agent is expected to be both effective and efficient.

heading: 2.2 GUI Control with Natural Language
text: Recently, LLMs have shown promise in building autonomous GUI agents with abilities of instruction following (Sanh et al., 2022; Taori et al., 2023b; Chiang et al., 2023) and chain-of-thought (CoT) prompting (Nye et al., 2022; Wei et al., 2022). Especially, CoT prompting (Wei et al., 2022; Kojima et al., 2022; Zhang et al., 2023a) elicit LLMs’ capacities of step-by-step planning, decision making, and action execution. Those capacities have been shown to be effective in GUI control tasks (Rawles et al., 2023).
However, the task environments are GUIs instead of natural language that LLMs can process directly. Therefore, the GUI states and actions are required to be converted to textual formats to be applicable to LLMs. For example, it is feasible to parse the GUI screens by icon recognition and OCR (Zhang et al., 2021; Sunkara et al., 2022; Song et al., 2023) and organize the parsed elements into HTML layouts. As a compromise, existing approaches are restricted in a sandbox setting where they rely on external tools (Rawles et al., 2023; Wen et al., 2023) and application-specific APIs
(Zhou et al., 2023; Gur et al., 2023) for environment parsing and action interpretation; thus, commonly suffer from inference inefficiency and error propagation. Although there are studies that have considered multimodal architecture for processing inputs in different modalities (Sun et al., 2022; Yan et al., 2023), those studies still rely on fine-grained environment parsing to ensure competitive performance. As GUI tasks have shown prerequisites to fine-grained grounding to the GUI contents, more recent concurrent studies (Cheng et al., 2024; Hong et al., 2023b) have explored GUI grounding pretraining to improve the agent’s performance.
In contrast to the studies above, this work is established upon first principles thinking, which directly reads GUI without additional environment parsing and provides the action (e.g., action type, gesture coordinate, and typed text) that can be efficiently executed without needing any extra APIs. Concretely, our approach diverges from existing studies in several key aspects.
(i) Conceptually, Auto-GUI represents a novel approach as a multimodal agent designed for autonomous GUI control, enabling direct interaction with screens. This approach effectively bypasses the need for environment parsing and access to application-specific APIs, thus enhancing adaptability and versatility.
(ii) Methodologically, our proposal introduces a chain-of-action technique. This method capitalizes on previously executed actions and future action plans to inform the decision-making process of the agent at each step.
(iii) As a scientific contribution, our paper ar-
ticulates that a unified multimodal model out of first principles thinking can serve as a strong autonomous agent evidenced by state-of-the-art performance. In addition, we facilitate a series of studies to understand the contributing factors regarding the vision features and model scale.

heading: 3 Methodology
text: In this section, we will first introduce the basic concepts for the GUI control task and then describe the design of our proposed Auto-GUI framework.

heading: 3.1 Problem Formalization
text: Given a user instruction (also known as a goal), the agent needs to complete the task with multiple steps of interactions. The entire process is called an episode, which is composed of a series of screens. For each step in the episode, the agent will be provided with a screenshot, and the agent is required to predict the action until the task is complete. Detailed examples can be found in Appendix A.1.

heading: 3.2 Framework Overview
text: Auto-GUI is a multimodal agent that decides what action to take given the input screenshot and a user instruction. To empower the agent’s decision making capability, we introduce a chain-of-action approach by leveraging a series of intermediate previous action histories and future action plans to predict actions.
The model architecture of Auto-GUI is illustrated in Figure 2. On a high level, Auto-GUI consists of three stages. First, we acquire encoded features from both vision and language inputs. Specifically, the vision input, i.e., a screenshot, is encoded by a frozen vision encoder. Meanwhile, the language input, consisting of the goal and a chain of previous action histories—each history contains a tuple {action type, touch point, lift point, and typed text}, is encoded by a language encoder. Second, the encoded vision and language representations are integrated by a self-attention module. Third, the fused representation is fed to the decoder to generate a chain of future action plans (i.e., action types to execute in future steps) followed by action prediction. A chain of action consists of two parts in the procedure above: a chain of previous action histories on the input side and a chain of future action plans on the output side. In the following, we describe the entire procedure in detail.
Encoding Suppose that an episode consists of k steps of interactions. Given a screenshot Xscreen ∈ Rh×w×3 with height h and width w at step t ∈ [1, k], we first feed it to a frozen image encoder (e.g., BLIP-2 (Li et al., 2023)) and extract vision features Hscreen ∈ R1×ds where ds is the dimension of the vision features. Additionally, we leverage a language encoder to extract the language features Hlanguage ∈ Rn×dl of the input goal Xgoal where n is the number of tokens and dl is the dimension of the language features. If t > 1, there will be a chain-of-action history already executed before step t. We denote the chain of action histories as Xhistory = [m1, . . . ,mt] where mi contains a tuple of action type, touch point, lift point, and typed text.1 Otherwise, if t = 1, Xhistory will be set empty:
Xhistory = { [m1, . . . ,mt], if t > 1 <empty>, otherwise (1)
We concatenate Xgoal and Xhistory as the input to the language encoder: Xlanguage = {Xgoal, Xhistory}.
Then, we obtain the encoded representations of the vision and language inputs as follows:
Hscreen = VisionExtractor(Xscreen), (2) H ′ screen = WHscreen, (3) Hlanguage = LanguageEncoder(Xlanguage),(4)
where W is a trainable projection matrix to convert Hscreen into the same dimensionality as Hlanguage.
Interaction We correlate H ′screen and Hlanguage with a single-head self-attention network (Vaswani et al., 2017),2 where the query (Q), key (K), and value (V ) are Hlanguage, H ′ screen, and H ′ screen, respectively. The attention output Hattnscreen ∈ Rn×d is defined as: Hattnscreen = Softmax( QK⊤√ dk
)V , where dk is the same as the dimension of Hlanguage because a single head is used.
Then, a gated fusion (Zhang et al., 2020; Wu et al., 2021; Zhang et al., 2023b) is adopted to fuse
1Following the design philosophy of AITW (Rawles et al., 2023), the agent is presumed to execute a sequence of actions until the task is completed. For practical use, it is feasible to halt the agent using straightforward rules when required.
2It is possible to use a unified multimodal Transformer architecture (Bavishi et al., 2023). Here we opt for a singlelayer interaction to maintain the lightweight nature of our method, with parameters less than 1 billion. This design choice also ensures flexibility to accommodate new modalities in the future, such as integrating additional feature extractors.
Hlanguage and Hattnscreen. We have the fused output Hfuse ∈ Rn×d by:
λ = Sigmoid(WlHlanguage +WvHattnvision), (5) Hfuse = (1− λ) ·Hlanguage + λ ·Hattnvision, (6)
where Wl and Wv are learnable parameters.
Decoding The fused representation Hfuse is fed to a Transformer decoder to generate the target predictions in a string format. The target predictions consist of a chain of future action plans Yplan and the current action prediction Yaction separated by specific prompts: {Action Plan: Yplan, Action Decision: Yaction}. Concretely, Yplan is a chain of action types to execute in future steps: Yplan = [action_typet, . . . , action_typek]. Yaction contains four components: Yaction = {“action_type”: <action_type>, “touch_point”: <touch_point>, “lift_point”: <lift_point>, “typed_text”: <typed_text>}. These four components will be explained as follows.

heading: 3.3 Coordinate Normalization
text: Recall that a target action consists of four components: action type, touch point, lift point, and typed text. We consider six action types: dualpoint gesture, type, go_back, go_home, enter, and status_complete. A dual-point gesture comprises a touch point and a lift point with [y, x] coordinates.3 The gesture actions ensure a flexible action space and can represent clicks and scrolls at arbitrary locations. For example, a gesture action {“touch_point”: [0.7761, 0.7089], “lift_point”: [0.7761, 0.7089]} means clicking at the coordinate [0.7761, 0.7089], while a gesture action {“touch_point”: [0.1898, 0.4477], “lift_point”: [0.8242, 0.4077]} means scrolling down. A type action means typing a text and the text is placed in the <typed_text> field. The other action types, i.e., go_back, go_home, enter, and status_complete are system actions, whose corresponding <touch_point>, <lift_point> fields are filled with -1, and the <typed_text> is empty.
We observe that high-precision coordinates are not necessary for representing a click or scroll action. Therefore, we apply normalized values of the coordinates, which helps accelerate convergence and mitigate the ambiguity of coordinates. The normalization is applied to click and scroll actions. For click actions, we keep four decimal places. For
3We follow the [y, x] coordinate format in AITW.
scroll actions, we first determine the scroll direction with the touch and lift points. Then, we transform the touch and lift points into fixed directional coordinates as follows: “up”: {[0.8, 0.5], [0.2, 0.5]}, “down”: {[0.2, 0.5], [0.8, 0.5]}, “left”: {[0.5, 0.8], [0.5, 0.2]}, “right”: {[0.5, 0.2], [0.5, 0.8]}, where {[·], [·]} consists of the touch point and lift point in the first [·] and second [·].4 We provide examples of target actions in Appendix B.1.

heading: 4 Experiments

heading: 4.1 Dataset
text: We use the AITW benchmark dataset (Rawles et al., 2023). AITW is a large-scale GUI control benchmark dataset containing natural language instructions, screenshots, and actions. There are 715K episodes spanning 30K unique instructions, covering diverse multi-step tasks such as application operation, web searching, and web shopping, on over 350 Apps and websites. This dataset covers various device types and operation systems in varying screen resolutions to ensure generality. There are five subsets in the benchmark dataset, namely, General, Install, GoogleApps, Single, and WebShopping.
Table 1 presents the data statistics of the AITW dataset. Each subset is split episode-wise into a training, validation, and test set (80/10/10%). More details of the subsets can be found in Appendix A.2.

heading: 4.2 Baselines
text: We adopt three types of baselines, allowing for a comprehensive comparison with our approach. The baselines encompass the in-context earning (ICL) and fine-tuning paradigms. They are based on various backbone models of different sizes.
(i) In-context Learning LLMs. Few-shot PaLM 2, ChatGPT (turbo-3.5) are adopted. Following prior studies (Rawles et al., 2023; Wang et al., 2023a), we feed the LLM a textual description
4The numbers are relative scales and they generalize to different screen sizes.
of the screen and a user instruction. The screen is formatted as an HTML syntax, providing the information of GUI elements derived from OCR detection and icon detection from external tools (Rawles et al., 2023). The model is required to predict an action among pre-defined actions. In addition, we report the results of the multimodal GPT-4V by taking the vision image and action history as the input based on Yan et al. (2023).
(ii) Fine-tuned LLMs. We adopt Llama-2-7B (Touvron et al., 2023) as the baseline and fine-tune it with LoRA. We feed the model with the user instruction and the screen descriptions in HTML syntax (the same as in-context learning LLMs). The model is expected to predict the action in the same output format as in-context learning LLMs.
(iii) Specialized GUI Agent. We adopted the Behavioural Cloning (BC) agent, which reported the state-of-the-art performance in Rawles et al. (2023). BC is a Transformer-based architecture that takes a task instruction, the current screen, and a stacked history of screen observations and actions as input. All the embedded representations are fused to predict the action by a decoder. There are two BC variants, BC-single and BC-history, depending on whether the model takes the screenaction history as input.
More detailed implementation of the baselines can be found in Appendix B.2.

heading: 4.3 Evaluation Measures
text: We compute the screen-wise action matching score as the main evaluation measure, defined as the number of correct actions divided by the episode length (i.e., the number of screenshots). A predicted action is considered correct if the action type and dual-point gesture match the gold ones. As we described in Section 3.3, the gesture actions can represent the click actions and scroll actions at arbitrary locations. Following Rawles et al. (2023), a click action is considered correct if its touch point and lift point fall within a 14% screen distance from the gold gestures or occur within the same detected bounding box with the gold gestures. A scroll action is considered correct if it has the same scroll axis as the gold gesture.
The screen-wise action matching score has been shown to correlate with the task complete score estimated by human evaluations (Rawles et al., 2023) and is appropriate to measure the action success rate for user instructions. Besides the overall matching score, we will also compare the click region
accuracy, scroll direction accuracy, action type accuracy (e.g., clicking, scrolling, typing, etc), and typed text accuracy for a more comprehensive reference (Section 5.1).
The evaluation criteria above apply to the BC baselines and our Auto-GUI. For the LLMs, they can only click on detected GUI elements, rather than clicking at arbitrary locations. Therefore, we consider if the clicked GUI element is matched for click actions instead of comparing dual-point gestures for LLMs.

heading: 4.4 Implementation Details
text: We adopt the encoder-decoder architecture (Raffel et al., 2020) under small (60M), base (200M) and large (700M) settings in our framework. We apply FLAN-Alpaca to initialize our model weights.5 The vision features are obtained by the frozen BLIP-2 encoder (Li et al., 2023) (version: blip2_t5_instruct). We fine-tune the models up to 10 epochs, with a learning rate of 1e-4. The maximum input sequence length is 512. The batch size is 4. Our experiments are run on 8 NVIDIA Tesla V100 32G GPUs. Training the large and base models takes 75 and 25 hours, respectively.
We develop two kinds of approaches to analyze their generalization abilities, namely AutoGUIseparate, and Auto-GUIunified. Specifically, AutoGUIseparate is trained and evaluated independently on each subset. Auto-GUIunified is a unified model trained on the training sets of each subset and evaluated on each test set. As the GoogleApps subset is 10-100 times larger than the other subsets, using all the training data to train a unified model would suffer from the data imbalance issue (Zhang et al., 2022). Therefore, we only use 10% training data of GoogleApps. At the same time, the overall computation cost can also be saved by 80%. We use Auto-GUIunified as the default model for analysis unless otherwise stated.

heading: 4.5 Main Results
text: Table 2 shows the main results. Based on the results, we have the following observations.
(i) Auto-GUIunified achieves the best overall performance compared with all the baselines. Compared with separate (not unified) models, AutoGUIunified shows general effectiveness across various tasks. The results show that a unified multimodal model out of first principles thinking can
5https://github.com/declare-lab/flan-alpaca.
serve as a strong autonomous agent. Compared with previous BC models, Auto-GUIunified has two major advantages. First, Auto-GUIunified is a unified model that can be adapted to different scenarios without the need to train specific models for each task. Second, Auto-GUIunified does not need additional annotations and is more practical in realworld applications. Furthermore, Auto-GUI yields divergent performance across subsets. We provide the explanation in Appendix C.1 to save space.
(ii) Both the chain of actions and coordinate normalization contribute to the overall performance (+5.74% and 4.04%, respectively), as evidenced by the ablation study in Table 3. Additionally, we set the maximum numbers of the previous actions and future actions to 8 and 4, respectively. The choice is made according to our analysis of the
General subset with Auto-GUIseparate (Figure 3). The model under those setups achieves the optimal performance, and neither the input nor output sequence lengths exceed the model limit.
(iii) For the LLMs, using either prompting or fine-tuning techniques does not achieve competitive performance compared with multimodal approaches. The most plausible reason is that they learn from the parsed HTML elements of the screen so that they may suffer from information loss compared with more informative vision features of the screens. Specifically, we find that ChatGPT is quite accurate at predicting the action type but fails at lower-level executions (Appendix 5.2).

heading: 5 Analysis

heading: 5.1 Analysis of Auto-GUI by Category
text: To dive into the capability of Auto-GUI, we calculate the click region accuracy, scroll direction accuracy, action type accuracy, and typed text accuracy. Figure 4 presents the results. We see that Auto-GUI achieves over 90% action type accuracy on average. In contrast, the major challenges lie within the click region and scroll direction predictions. Although the model is able to predict the right action most of the time, it tends to click a

heading: WebShopping
text: wrong place or scroll in a wrong direction. The result reveals a future direction of improving the model’s ability to understand the screen layouts, e.g., using more advanced vision features.

heading: 5.2 Comparison with ICL by Category
text: To understand how the ICL baseline performs on our task and assess the advantage of Auto-GUI, we conduct a category comparison with ChatGPT.
In Table 6, we see that the ICL method (ChatGPT) is quite accurate at predicting the action type (41.72%) but fails at lower-level executions, e.g., clicking positions (8.5%) and scrolling directions (4.0%). The results show that using HTML-based layout information is not enough to accurately execute actions. In contrast, Auto-GUI has the advantage of predicting both action types and performing low-level executions by leveraging multimodal perception and the chain-of-action technique.

heading: 5.3 Generalization Ability
text: As our approach is designed under first principles thinking and does not rely on pre-defined internal APIs, it could be easily generalized to new task
domains. To verify the generality, we evaluate the performance of Auto-GUIseparate on each subset in Figure 5. For example, we train an Auto-GUIseparate model on the training set of General and then test its performance on the tests of each subset.
We see that our approach is able to achieve a decent performance, though the domains vary. This result reveals that the model could capture general knowledge for the GUI control task; thus is applicable to different domains. In addition, the unified model Auto-GUIunified can serve as a potential choice in real-world applications owing to more coverage of training data.

heading: 5.4 Comprehensive Analysis
text: Here, we present a comprehensive analysis of the choice of pre-trained features and model scale. The results are summarized in Table 4.
(i) Pre-trained Features. There are two kinds of pre-trained features used in this work, the vision features and language model weights. For vision features, we compare two popular types, CLIP (Radford et al., 2021) and BLIP-2 (Li et al., 2023). We observe that BLIP-2 achieves relatively better performance. Therefore, we use BLIP-2 by default in Auto-GUI. For pre-trained language model weights, we compare initializing the model with the vanilla T5 (Raffel et al., 2020), FLANT5 (Chung et al., 2022), and FLAN-Alpaca (Taori et al., 2023a) weights under the large size. We see that FLAN-Alpaca achieves the best performance as it has been optimized with Stanford Alpaca synthetic instruction tuning data.
(ii) Model Scale. Compared with the performance gains from our technique components (chain of actions and coordinate normalization) in Table 3, the benefit of scaling parameter size becomes relatively marginal. As we observe that a larger model size does not lead to dramatic improvement in performance, we do not scale the model scale but
focus on the base (220M) and large (770M) models in this work. In addition, our choice is also based on other considerations, including the constriction of GPU memory and computation budget.
According to the analysis above, our work offers insights into the determinants of model efficacy: • Vision features are critical, underscoring the significance of effective perception. • Model scale exhibits diminished importance, indicating that larger model sizes do not necessarily yield dramatic performance enhancements.

heading: 5.5 Computation Cost
text: Table 5 compares the inference speed and GPU memory cost for Auto-GUI and Llama 2. AutoGUI is able to achieve nearly real-time inference (within less than one second for an action prediction) with less than 10GB GPU memory. The inference speed is over 10 times faster than Llama 2. Our work shows the strength of the medium-sized language model in building autonomous agents, which is able to achieve competitive performance with fast inference speed and modest resource cost.

heading: 6 Conclusion
text: This work presents an autonomous GUI agent called Auto-GUI that can interact in a multimodal
GUI environment without environment parsing or application-dependent API access. In addition, we propose a chain-of-action technique that leverages the previously executed actions and future action plans to help the agent decide what action to execute. Experimental results show that AutoGUI achieves superior performance to previous prompting-based and fine-tuning baselines. We show that it is possible to achieve state-of-the-art performance by an end-to-end model without relying on external tools and application-specific APIs to parse the environment and interpret the predicted actions. Besides the strong performance and generality across domains, Auto-GUI is able to infer an action in less than one second.

heading: Limitations
text: We acknowledge two primary limitations in our study. First, we opted not to extend the approach to extremely large models because our work aims to provide a simple yet effective solution for GUI agents. Our findings suggest that scaling may not be fundamentally advantageous in GUI problems. The significance of model scale tends to diminish— increasing the model size does not necessarily result in a substantial performance enhancement. Second, our experiments and analysis were exclusively conducted on AITW, which is the largest-scale and widely recognized benchmark dataset in the research line of autonomous GUI agents, to provide timely and pertinent insights. Given the rapid development of the field, we anticipate future studies to explore the application of our approach on other benchmark datasets as they become available.

heading: A Data Details

heading: A.1 Data Examples
text: We show the data examples from the AITW benchmark dataset (Rawles et al., 2023). Figures 6-9 show the examples in each subset, i.e., General, Install, GoogleApps, Single, and WebShopping. The gold actions for each screen are depicted in the illustrations for reference.

heading: A.2 Data Statistics
text: We use the AITW benchmark dataset (Rawles et al., 2023). AITW is a large-scale benchmark dataset for GUI control, which contains natural language instructions, screenshots, and actions. There are 715K episodes spanning 30K unique instructions, covering diverse multi-step tasks such as application operation, web searching, and web shopping, on over 350 Apps and websites. This dataset covers various device types and operation systems in varying screen resolutions to ensure generality. There are five subsets in the benchmark dataset: General, Install, GoogleApps, Single, and WebShopping.
(i) General contains miscellaneous tasks that need interaction with third-party Apps and websites, as well as question answering.
(ii) Install contains tasks related to installing, uninstalling, logging Apps, and App login support.
(iii) GoogleApps contains tasks about manipulating various Google applications such as Gmail, Calendar, Photos, and Settings.
(iv) Single contains atomic tasks (e.g., “upvote the post”) whose preceding actions have been already completed (e.g., opening Instagram, going to home feed, looking at a post).
(v) WebShopping contains tasks related to online shopping on E-commerce websites, e.g., searching for an item, adding an item to the cart, and viewing the shopping cart.
Table 7 presents the data statistics of the AITW dataset. Each subset is split episode-wise into a training, validation, and test set (80/10/10%).
0 100 200 300 400 500
0
200
400
600
800
1000
0 100 200 300 400 500
0
200
400
600
800
1000
0 100 200 300 400 500
0
200
400
600
800
1000
0 100 200 300 400 500
0
200
400
600
800
1000
0 100 200 300 400 500
0
200
400
600
800
1000
0 100 200 300 400 500
0
200
400
600
800
1000
0 100 200 300 400 500
0
200
400
600
800
1000
0 100 200 300 400 500
0
200
400
600
800
1000
0 100 200 300 400 500
0
200
400
600
800
1000 Set episode status as COMPLETE
Goal: Open a new Chrome private window

heading: Goal: uninstall "Microsoft Authenticator"
text: Figure 7: An example episode from Install.
B Implementation Details

heading: B.1 Coordinate Normalization
text: Recall that a target action consists of four components: action type, touch point, lift point, and typed text. We consider six action types: dualpoint gesture, type, go_back, go_home, enter, and status_complete. A dual-point gesture comprises a touch point and a lift point with [y, x] coordinates. The gesture actions ensure a flexible action space and can represent clicks and scrolls at arbitrary locations. For example, a gesture action {“touch_point”: [0.7761, 0.7089], “lift_point”: [0.7761, 0.7089]} means clicking at the coordinate [0.7761, 0.7089], while a gesture action {“touch_point”: [0.1898, 0.4477], “lift_point”: [0.8242, 0.4077]} means scrolling down. A type action means typing a text and the text is placed in the <typed_text> field. The other ac-
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600
700
0 50 100 150 200 250 300 350 400
0
100
200
300
400
500
600 700 Set episode status as COMPLETE
Goal: turn off javascript in the chrome app
Figure 8: An example episode from GoogleApps.
tion types, i.e., go_back, go_home, enter, and status_complete are system actions, whose corresponding <touch_point>, <lift_point> fields are filled with -1, and the <typed_text> is empty.
We observe that high-precision coordinates are not necessary for representing a click or scroll action. Therefore, we apply normalized values of the coordinates, which helps accelerate convergence and mitigate the ambiguity of coordinates. The normalization is applied to click and scroll actions. For click actions, we keep four decimal places. For scroll actions, we first determine the scroll direction with the touch and lift points. Then, we transform the touch and lift points into fixed directional coordinates as follows: “up”: {[0.8, 0.5], [0.2, 0.5]}, “down”: {[0.2, 0.5], [0.8, 0.5]}, “left”: {[0.5, 0.8], [0.5, 0.2]}, “right”: {[0.5, 0.2], [0.5, 0.8]}, where
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200 1400 Input text "lowe's"
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200 1400 Input text "best rated coffee maker"
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500 600 700
0
200
400
600
800
1000
1200 1400 Set episode status as COMPLETE
Goal: Look up the best rated coffee maker on Lowe's.
Figure 9: An example episode from WebShopping.
{[·], [·]} consists of the touch point and lift point in the first [·] and second [·]. We provide examples of target actions in Table 8.

heading: B.2 Baselines
text: We adopt three types of baselines for comparisons. The baselines encompass the in-context earning (ICL) and fine-tuning paradigms, along with various backbone models of different sizes. This choice of baselines allows for a comprehensive comparison with our proposed approach.
(i) In-context Learning LLMs. Few-shot PaLM 2, ChatGPT (turbo-3.5) are adopted. Following previous studies (Rawles et al., 2023; Wang et al., 2023a), we feed the LLM a textual description of the screen and a user instruction. The textual description of the screen is formatted as an HTML
syntax, providing the information of GUI elements derived from OCR detection and icon detection from external tools (Rawles et al., 2023). The model is required to predict an action among predefined actions. If the action is clicking, the model will be required to provide the index of the clicked GUI element. Alternatively, the model needs to provide the scroll direction if the action is scrolling. In addition, 5-shot CoT prompting is leveraged to improve the performance (Appendix B.3). In addition, we report the results of the multimodal GPT-4V by taking the vision image and action history as the input based on Yan et al. (2023).
(ii) Fine-tuned LLMs. We adopt Llama 2 (Touvron et al., 2023) as the baseline and fine-tune it with LoRA. We feed the model with the user instruction and the screen descriptions in HTML syntax (the same as adopted for in-context learning LLMs). The model is expected to predict the action in the same output format as in-context learning LLMs. As fine-tuning an LLM is expensive, we randomly sample 1% training data to help the LLM adapt to our tasks.
(iii) Specialized GUI Agent. We adopted the Behavioural Cloning (BC) agent, which reported the state-of-the-art performance in Rawles et al. (2023). BC is a Transformer-based architecture that takes a task instruction, the current screen, and a stacked history of screen observations and actions as input. The task instruction and OCRdetected texts are encoded by a pre-trained BERT. The icons are represented by the embeddings for each of the bounding box points. The screen history is modeled by the {x, y} positions of the touch and lift actions. All the embedded representations are fused to predict the action by a decoder. There are two BC variants, BC-single and BC-history,
depending on whether the model takes as input the screen-action history.

heading: B.3 LLM Prompt
text: We use the prompt in Figures 10-11 for PaLM 2- CoT and ChatGPT-CoT owing to its optimal performance reported in Rawles et al. (2023).

heading: C Further Analysis

heading: C.1 Subset Analysis
text: We notice that Auto-GUIunified performs relatively inferior to BC-history on the two App-centered subsets, Install and GoogleApps. It is reasonable because we only use 10% training data of GoogleApps considering the data balance and computation overhead. We observe that the performance does not improve when we use all the training data of GoogleApps, possibly due to the data imbalance issue (Zhang et al., 2022). In contrast, our separate model Auto-GUIseparate can achieve better performance than BC-history, showing that our approach is better than BC-history under the same training setting. As we aim to study a simple and unified approach that achieves generally strong performance, we leave the treatment of the data imbalance issue in future work.

heading: C.2 Ablation Study
text: Table 9 shows the detailed results of the ablation study. We see that both the chain of actions and coordinate normalization contribute to the overall performance (+5.74% and 4.04%, respectively).

heading: C.3 Using Screen Descriptions
text: We are interested in whether Auto-GUI can be further improved when screen annotations are available. Therefore, we incorporate screen descriptions
containing icon and text information, organized in HTML syntax, into our language input Xlanguage. Detailed examples of screen descriptions can be found in the “Screen” block in Appendix B.3.
In Table 10, we see that Auto-GUI can perform better when the annotated screen descriptions are available. The results show that there is still room for performance gains for Auto-GUI. However, as the annotations are not always available in realworld applications, we do not include them by default in our framework.

heading: Future Work
text: As we aim to study a simple and unified approach that achieves generally strong performance, we leave the treatment of the data imbalance issue in future work