CAAP: Context-Aware Action Planning Prompting to Solve Computer Tasks with Front-End UI Only

## Abstract

Abstract Software robots have long been used in Robotic Process Automation (RPA) to automate mundane and repetitive computer tasks. With the advent of Large Language Models (LLMs) and their advanced reasoning capabilities, these agents are now able to handle more complex or previously unseen tasks. However, LLM-based automation techniques in recent literature frequently rely on HTML source code for input or application-specific API calls for actions, limiting their applicability to specific environments. We propose an LLM-based agent that mimics human behavior in solving computer tasks. It perceives its environment solely through screenshot images, which are then converted into text for an LLM to process. By leveraging the reasoning capability of the LLM, we eliminate the need for large-scale human demonstration data typically required for model training. The agent only executes keyboard and mouse operations on Graphical User Interface (GUI), removing the need for pre-provided APIs to function. To further enhance the agent’s performance in this setting, we propose a novel prompting strategy called Context-Aware Action Planning (CAAP) prompting, which enables the agent to thoroughly examine the task context from multiple perspectives. Our agent achieves an average success rate of 94.5% on MiniWoB++ and an average task score of 62.3 on WebShop, outperforming all previous studies of agents that rely solely on screen images. This method demonstrates potential for broader applications, particularly for tasks requiring coordination across multiple applications on desktops or smartphones, marking a significant advancement in the field of automation agents. Codes and models are accessible at https://github.com/caap-agent/caap-agent .

## 1 Introduction

Artificial intelligence (AI) agents are often commercialized as Robotic Process Automation (RPA) tools, designed to streamline business processes with minimal human intervention. Traditional RPA agents use rule-based algorithms to automate structured tasks, making them particularly effective for repetitive and standardized data processing. However, most desktop tasks are not well-suited to this rule-based approach, as they frequently involve handling unexpected situations and exceptions that are too numerous and varied to anticipate and program for in advance. In contrast, agents utilizing deep neural networks can adapt more flexibly to unforeseen situations. Studies have shown that agents trained on extensive data of expert demonstrations can effectively handle unseen tasks by imitating the decision-making styles of human experts (Humphreys et al., ; Shaw et al., ). The emergence of Large Language Models (LLMs) is now further advancing the capabilities of RPA agents. LLMs have demonstrated the potential to support complex decision-making processes with their advanced reasoning capabilities. Their In-Context Learning (ICL) ability allows them to learn meaningful actions for specific tasks with just a few examples, significantly reducing the need for large-scale data collection of expert demonstrations. As a result, research on agents leveraging ICL has been vigorously pursued in recent years (Yao et al., ; Sridhar et al., ; Zheng et al., ; Shinn et al., ; Lin et al., ; Huang et al., ; Liu et al., ). As AI chips are increasingly integrated into computers and smartphones (Intel, ; Samsung, ), and as these AI devices are poised to become the standards in the market, this new trend in the design of AI-driven RPA tools is likely to have a profound impact.

Most prior research on agents for solving computer tasks has primarily relied on HTML or the Document Object Model (DOM) as input sources, or has leveraged well-structured Application Programming Interfaces (APIs) to interact with digital environments (Humphreys et al., ; Liu et al., ; Gur et al., ; Kim et al., ; Sun et al., ; Furuta et al., ). However, the effectiveness of these methods depends on the availability of such shortcuts. They lack general applicability to broader desktop tasks that are not web-based or that require the use of applications with no provided APIs. Often, many applications and web services in the real world do not maintain rigorous development standards.

In this paper, we present an agent model that exclusively uses visual information from the screen as input and keyboard/mouse actions as output. This approach is intuitive, given that many computer-based tasks are designed specifically for human operators. Our proposed model features a modular architecture that distinctly separates the vision model, responsible for perceiving the environment, from the decision-making model, which handles action planning. This clear separation not only enhances the agent’s functionality but also improves its adaptability across various applications. To further support and maximize the effectiveness of our agent design, we have developed a novel prompting technique called Context-Aware Action Planning (CAAP). This technique systematically organizes the contextual information required for action decisions and identifies syntactic structures that trigger the most effective Chain of Thought (CoT) (Wei et al., ). Our agent demonstrates superior task-solving performance on the MiniWoB++ and WebShop benchmark. It surpasses previous studies by utilizing only a minimal amount of human demonstration data, thereby illustrating its efficiency and effectiveness in handling complex computer tasks with limited training cost.

Our contributions are as follows:

•  We present the first LLM-based agent for computer tasks that leverages ICL while emulating human behavior by using front-end User Interface (UI) channels as input-output sources. It processes input through an image-to-text model followed by an LLM, a combination that eliminates the need for extensive human labor in dataset collection before deploying the agent for new tasks. The agent operates exclusively through keyboard and mouse actions, enabling deployment on any GUI-based system without the need for special APIs to be provided.

•  We introduce an effective prompting technique named CAAP, which enhances the ICL capabilities of an LLM-based agent in managing complex desktop tasks. By systematically structuring contextual information and leveraging syntactic patterns that trigger optimal CoT reasoning, CAAP significantly outperforms other agent designs.

## 2 Related work

#### Agents with image inputs

Humphreys et al. () introduced CC-Net, a compound model of ResNet, Transformer, and LSTM that uses image and DOM data as inputs and produces policies for keyboard-mouse actions. It was trained via Supervised Learning (SL) and Reinforcement Learning (RL), covering the largest number of MiniWoB++ tasks to date (104 tasks).
Furuta et al. () proposed WebGUM, which combines Vision Transformer (ViT) and T5 transformers. It was trained using SL to solve both MiniWoB++ and WebShop tasks, utilizing both image and HTML inputs.
Shaw et al. () introduced Pix2Act, the first agent to completely eliminate reliance on DOM or HTML for solving MiniWoB++ and WebShop tasks. The Pix2Act model is built upon the Pix2Struct architecture (Lee et al., ) and uses only image inputs.
While these studies provide valuable insights and serve as important benchmarks, their methods are impractical for real-world applications as they require around a million or more human demonstration data samples, demanding thousands of hours of human labor for training. Cheng et al. () introduced SeeClick, a method based on a Large Multimodal Model (LMM). Aided by the strong reasoning capabilities of the LMM, SeeClick required a much smaller dataset of 2.8K human demonstrations to train for MiniWoB++ tasks. It fell short in overall performance, however, achieving less than a 70% success rate on the 55 tasks it covered. A common feature of all these approaches is the integration of image interpretation and action planning models within a single architecture. This rigid design poses a significant limitation: when new UI elements need to be recognized or additional actions supported for a new task, the entire architecture must be retrained.

#### LLM-based agents with ICL

To reduce dependence on large quantities of human demonstration data, several studies have explored the use of LLMs for action planning. Yao et al. () developed the ReAct model, which evaluates the current state before determining subsequent actions. Kim et al. () introduced the Recursive Criticism and Improvement (RCI) agent, which enhances action planning through iterative self-critique, highlighting the importance of advanced reasoning techniques in agents. Huang et al. () proposed Inner Monologue (IM), a mechanism that integrates real-time environmental states into action planning via an explicit feedback loop. Inspired by IM, Sun et al. () introduced AdaPlanner, which prompts LLMs to generate action plans as Python functions and refine them based on feedback from execution failures. Despite their innovations, all these approaches share a common limitation: their architectures are designed to operate solely with HTML inputs, restricting their applicability to constrained, well-curated web environments.

## 3 Modularized GUI agent for computer tasks

Figure: Figure 1: Architecture of the front-end UI agent and its task-solving workflow.
Refer to caption: x1.png

Our agent consists of three distinct modules: a visual observer that processes and interprets image inputs, a decision-maker responsible for action planning, and an action executor that carries out the planned actions. The agent operates by repeatedly executing a cycle consisting of a series of sequential processes involving these three modules. Each of these modules is further divided into functional submodules, as depicted in Figure 1. This approach offers several advantages, including modular updates, better scalability, and parallelized processing:

#### Modular updates

The modular architecture provides the flexibility to perform isolated updates, minimizing disruption to the overall system and allowing for more efficient model improvements. In contrast, end-to-end monolithic agent designs, where perception and decision-making are tightly integrated within one LMM, require updates to the entire model, which is time-consuming and resource-intensive.

#### Scalability

The agent’s modular structure supports horizontal scalability by allowing the addition of new functional submodules to enhance overall performance. For instance, a reflection submodule (Shinn et al., ; Zhao et al., ) could be added to the decision-maker module, enabling the agent to analyze and learn from previous actions or experience. The visual observer module could be supplemented with multiple vision models, each specialized for different domains or tasks. This allows the system to dynamically select and apply the appropriate model based on the context, significantly enhancing the agent’s adaptability to various environments.

#### Parallelization

The agent’s modular design also supports parallel processing of image models. For example, the screen image can be divided into multiple sections, with multiple instances of visual observers processing different regions concurrently. This parallel processing can be integrated seamlessly without disrupting the performance of the decision-making modules.

### 3.1 Visual observer * * * These authors contributed equally to this work.

The visual observer module employs a two-step approach to transform visual information from a screen into the linguistic domain. Initially, in the detection stage, UI elements are identified, and their spatial locations are extracted. Subsequently, these elements undergo further analysis through advanced object understanding techniques, during which detailed attributes of each element are captured. This processing results in the information being organized into structured text, ready to be handed to the decision-maker as the representation of the current state.

#### UI element detection

For the detection and extraction of location information of all UI elements present in screenshots, we fine-tuned the YOLO model. The YOLO model, known for its rapid and precise detection in one shot, typically extracts object type information along with bounding boxes. However, we utilize only the bounding boxes, leaving the extraction of type information to the subsequent UI element understanding module for greater accuracy.

Figure: Figure 2: The masking methods used by the CAAP agent.
Refer to caption: x2.png

#### UI element understanding

We use the Pix2Struct model (Lee et al., ) to extract detailed attributes from all UI elements identified during the detection stage. We fine-tune the original Pix2Struct model to match the specific output format required for our task. During this fine-tuning process, we employ a different masking technique from Pix2Struct style, as depicted in Figure 2 (a). Our masking technique highlights the target UI element by outlining its region and darkening the surrounding image areas, as demonstrated in Figure 2 (b). This method preserves the pixel-level details of the targeted element while helping the model learn the spatial relationships between the element and its adjacent elements. The structured output from the model is presented as shown in Figure 2 (c), with the range of extractable attributes for each element detailed in Table 7 of Appendix C.

### 3.2 Decision maker

Upon receiving the representation of the current screen, the decision maker module is employed to determine possible actions for completing the task. The decision-making process is facilitated by its two key components: a context collector and a CAAP prompter. The context collector gathers vital contextual information that aids in task resolution. This includes exemplary demonstrations, the ultimate task goal, the history of actions performed so far, and the range of possible actions that the language model can choose from, all formatted as text. The CAAP prompter then uses this information to construct prompts that are sent to the language model, eliciting responses that guide the next steps. The CAAP prompting method, one of our major contributions in this paper, is discussed in detail in Section 4.

### 3.3 Action executor

The actions determined by the decision maker are implemented through a low-level interface to effect changes in the environment by the last module, action executor. It first identifies a series of recommended actions by parsing the response text from the decision maker. Then, it executes these actions using the keyboard and mouse interfaces of the operating system. After each action, it receives an updated screenshot, halting further actions if a change in the environment is detected. The types of actions that can be executed include mouse actions such as clicking, pointing, and scrolling, as well as keyboard actions like typing and using common shortcuts. These are enumerated in Table 8 of Appendix D and implemented as Python functions using the PyAutoGUI package.

## 4 CAAP prompting

Figure: Figure 3: Content design for the CAAP prompting. Figure 8 and 9 in Appendix H show an example of a prompt generated during the MiniWob++ experiments and the LLM’s response to this prompt.

The performance of LLM-based agents is heavily dependent on how the prompts are structured. By focusing on delivering relevant contextual information in the most efficient manner and inducing CoT reasoning during response generation, our prompting technique significantly enhances the decision-making capabilities of LLM agents in computer tasks. CAAP prompting constructs the prompt using the content depicted in Figure 3.

#### Human demonstrations

Providing few-shot exemplars within the prompt to an LLM is a common practice that enhances the advanced reasoning capabilities of LLMs. Our CAAP prompt begins with few-shot examples, which are sequences of actions performed by human demonstrators solving tasks similar to the given objective. In practical scenarios, the Retrieval-Augmented Generation (RAG) method can be employed to extract similar task records from a database. The effectiveness of this demonstration portion of the prompt can be significantly enhanced by incorporating rationale texts behind each action, even when those rationales are not provided by the demonstrators themselves but are autonomously generated by an LLM, as shown in Section 5.3.

#### Surrounding context information

This second part of the CAAP prompt forms the main section, containing all the essential information crucial for the language model to make accurate decisions. It includes a description of the target task, a record of past actions, state descriptions from the visual observer, and the available range of actions for the agent.

#### CoT-inducing instructions

The CoT-inducing instructions section contains phrases that encourage the LLM agent to interpret the surrounding context information independently, thereby inducing the Chain of Thought. Instruction for reviewing action trajectory encourages the agent to review its actions taken so far, enhancing its understanding of the connection between these past actions and the current state of the environment. Instruction for reviewing visual observation prompts the agent to examine the current state’s observations, directing its focus on relevant objects needed to complete the task. Instruction for improving action plan requests the development of a follow-up action plan, considering the self-generated texts so far. Lastly, the Instruction for deciding next actions provides guidance on the response format and requests a list of actions to be taken in the next steps.

#### Extra guidelines

The final section of the CAAP prompt primarily features phrases designed to safeguard the LLM’s output. It incorporates additional directives specifically tailored to reduce common errors made by the particular LLM in use.

## 5 Experiments

### 5.1 MiniWoB++

MiniWoB++ is an extension of the OpenAI MiniWoB benchmark, introduced in 2017, designed to evaluate an agent’s ability to handle a wide range of scenarios that may occur on a computer screen (Liu et al., ; Shi et al., ; ). It consists of over a hundred of web interaction tasks that challenge the agent to devise and execute effective sequences of actions. These tasks include various UI interactions, such as button clicks, form filling, and drag-and-drop operations. We utilize the MiniWoB++ benchmark to thoroughly assess our agent’s effectiveness in navigating and solving diverse tasks within a simulated computer environment.

Previous studies using the MiniWoB++ benchmark often choose subsets of tasks most suitable for demonstrating their methods. We have also selected a total of 73 MiniWoB++ tasks that meet the following criteria:
(1) tasks that consist of UI elements of types recognizable by our visual observer module, as listed in Table 7 of Appendix C
(2) tasks where none of UI elements extend beyond the default screen area of 160x210 pixels
(3) tasks that do not require color recognition, as our visual observer module is not yet trained for this capability.

For each selected task, we run at least 50 repetitions with different seeds to estimate the Success Rate (SR) for that task. Across all task types, we consistently use the identical CAAP prompt template without making any manual adjustments for specific tasks, unlike some other works (Kim et al., ).

#### Models and datasets

YOLO and Pix2Struct models within the visual observer have been fine-tuned to enable accurate recognition of images presented in MiniWoB++ environment. The image dataset for training has been collected through a process of capturing screenshots and annotating the UI elements within. This dataset consists of 1.76K screenshot images, gathered and annotated over a span of 10 hours by two individuals.

For human expert demonstrations, we generate between 0 and 5 demonstrations per task, tailored to the task’s complexity, amassing a total of 97 demonstrations for our experiments. We record the trajectory of each action performed by the demonstrators, alongside screenshots taken at the time of each action. These screenshots are subsequently analyzed to extract UI elements, enabling the identification of the UI component interacted with during mouse-related actions, such as clicks. This demonstration data have been collected by a single individual over a span of approximately two hours.

For the decision-making language model, we utilize the Azure OpenAI gpt-4-0125 model. The Azure OpenAI API features the ‘function calling’ capability, thus our prompt does not directly include the list of action types (as depicted in Figure 3). Instead, this contextual information is conveyed to the language model through arguments in the function calling feature. An example of how the function calling is applied is illustrated in Figure 10 in Appendix I.

#### Experiment result

**Table 1: Comparison with other image-only methods for solving MiniWoB++ problems.**
| Method | Modality | Image datasize | Demo datasize | Reported SR | Reported tasks |
| --- | --- | --- | --- | --- | --- |
| Human | Image | - | - | 93.5% | 104 |
| CC-Net (no DOM) | Image | 0 | 2.4M | 24.0% | 104 |
| Pix2Act | Image | 0 | 1.3M | 96.2% | 59 |
| SeeClick | Image | 0 | 2.8K | 69.4% | 55 |
| CAAP | Image | 1.8K | 0.1K | 94.5% | 73 |

Table 1 compares the performance of our approach with other studies reported on the MiniWoB++ benchmark. Since each study reports results over different sets of tasks (See Figure 11 of Appendix K.), we have included both the SRs reported in the literature and the number of tasks used in each case. First, CC-Net, which covers the largest set of tasks, was trained on 2.4M human demonstration data across 104 tasks. However, when using only image input (excluding DOM), its performance dropped to 24%, indirectly highlighting the challenge of solving computer tasks using image inputs alone. Pix2Act, trained on 1.3M human demonstrations across 59 tasks, achieved an SR of 96.2%, demonstrating the feasibility of agents solving computer tasks with image input only. SeeClick, despite using a relatively small dataset of 2.8K demonstrations to learn click locations on screenshots, achieved an SR of only 69.4% across 55 tasks, showing limited effectiveness. In contrast, our approach, using only 97 demonstration samples, achieved a high task SR of 94.5% across 73 tasks. Although an additional 1.76K samples(^*^**These image samples were annotated via a partially automated process, making the data preparation process relatively efficient.) were required to train the model for image input interpretation due to the modular nature of our approach, our method still demonstrated its effectiveness on a broader range of computer tasks with about 1.86K samples in total. Task-level SRs are detailed in Table 12 of Appendix J.3.

Of the 73 tasks included in our experiments, six tasks—drag-circle, drag-single-shape, find-greatest, generate-number, odd-or-even, and sign-agreement—were excluded from agent’s training, as they were not covered in the other studies. These tasks were deliberately chosen to test our agent’s ability to adapt to new, unseen tasks. Impressively, our agent successfully completed them with an average success rate (SR) of 95.3%. These results highlight the advantages of our modular architecture. Unlike other approaches that rely on end-to-end imitation learning from human demonstrations, where vision recognition and action planning are tightly integrated, our method divides these functions into distinct modules. This separation allows our agent to leverage the knowledge and reasoning power of the LLM, enabling it to handle unseen tasks without requiring additional training, as long as those tasks don’t introduce entirely new types of UI elements that the visual observer module fails to recognize.

### 5.2 WebShop

WebShop (Yao et al., ) is a simulated e-commerce web environment designed to evaluate agents’ performance on tasks that more closely resemble real-world scenarios. To successfully complete WebShop tasks, the agent must understand textual instructions and requirements for the target product and navigate various types of web pages to locate it, out of a pool of 1.18 million real products scraped from amazon.com. The environment provides a total of 12,087 instructions, divided into 10,587 for training, 1,000 for development, and 500 for testing. Among them, we utilized only the 500 test instructions for evaluation purposes.

#### Models and datasets

The web browser for the WebShop environment can be resized freely, and for convenience, we used a fixed resolution of 1276 × 1153 pixels for both data collection and evaluation. Unlike MiniWoB++, the dataset used to train the visual observer for WebShop was generated entirely through an automated process. Given an instruction, a simple helper agent we created extracted a search term, performed a search, and clicked the first item in the results. Screen images were captured throughout this process, and annotation data was collected using Selenium for each image, resulting in a total of 300 annotated samples. The decision-making model, supported action types, and function-calling approach are entirely consistent with those employed in the MiniWoB++ experiments. For the evaluation of the CAAP agent, a single human demonstration is applied universally to all instructions.

#### Experiment result

**Table 2: Comparison with other image-only methods for solving WebShop problems.**
| Method | Modality | Image datasize | Demo datasize | Task score |
| --- | --- | --- | --- | --- |
| Human | Image | - | - | 82.1 |
| Pix2Act | Image | 0 | 1K | 46.7 |
| CAAP | Image | 0.3K | 1 | 62.3 |

There has been limited research on constructing agents that rely solely on image inputs, and Pix2Act is the only method whose performance has been validated on the WebShop benchmark. Table 2 presents a comparison between our approach and Pix2Act on WebShop. (Comparisons with other agent-based methods that rely on additional HTML input, on top of image input, are presented in Section 6 of Appendix B.2.) While Pix2Act employed 1,012 human demonstrations to train its agent model, our approach relied on just a single human demonstration, along with 300 samples collected through a fully automated process for training the visual observer module. Despite the significantly reduced manual data collection effort, our agent achieved a task score of 62.3, a meaningful improvement over Pix2Act’s score of 46.7, with a margin of 15.6. Although this result does not yet reach human-level performance, it demonstrates the potential of our approach to solve real-world computer tasks with minimal human effort in data collection.

### 5.3 Ablation studies

#### CoT inducing instructions for CAAP prompting

Figure: Figure 4: The effects of different CAAP components on MiniWoB++ task performance.
Refer to caption: x5.png

Figure 4 presents experiment results comparing the effects of human demonstrations and CoT inducing instructions in our CAAP prompting method. It is shown that the use of CoT inducing instructions results in a meaningful improvement in performance. Removing CoT inducing instructions from the prompt results in a 2.3% drop compared to the baseline, and a 3.8% drop when the human demo is absent. When only rationales are excluded from human demonstrations, there is a reduction in task-solving SR by 3.5% relative to the fully-prompted baseline. Excluding the whole human demonstrations leads to a decrease of 9.5%. Additionally, the removal of both CoT-inducing instructions and human demonstrations results in an decrease in SR by as much as 13.3%, demonstrating that effective ICL provided by our CAAP prompting scheme significantly enhances the decision-making capacity of the LLM model in use. The per-task result are depicted in Table 10 of Appendix J.1.

#### Comparison of the CAAP and the RCI prompting

**Table 3: Performance comparison of the CAAP and RCI promptings. The RCI agent with CAAP is configured by replacing the prompting method of the original RCI agent with the CAAP approach.**
|  | RCI agent with CAAP | RCI agent (original) |
| --- | --- | --- |
| Average SR | 0.840 | 0.769 |

Since our main experiments combine GUI-based perception and execution with the CAAP prompting technique, we conducted additional experiments to isolate the effect of CAAP prompting as part of an ablation study. Without modifying the input/output channels, we tested how the performance of the RCI agent (Kim et al., ) improves when its prompting mechanism is replaced by our CAAP prompting. We first executed the original RCI code from the authors‘ official GitHub repository without modifications, testing only the 47 tasks listed in the repository by 50 times and taking the average. (^*^**The success rate of the RCI agent we observed deviates from the results reported in the literature, despite our diligent efforts to replicate the experimental setup exactly. Upon inquiry, the RCI authors indicated that they no longer had access to the complete experimental logs used in their publication.) Then, we replaced the prompting method with CAAP and reran the tests. To ensure a fair comparison, we used identical seeds for task instances, the same demo set provided by the repository, and the same LLM for queries.

As described in Table 3, the results demonstrate that applying CAAP prompting to the RCI agent significantly increased the average task-solving SR by 7.1%, from 76.9% to 84.0%. (Task-level results can be seen in Table 11 of Appendix J.2.) The prompting methods of RCI and CAAP exhibit two key differences. First, the RCI approach tends to modify even correct action plans during the iterative critique and revision process. In contrast, the CAAP method, by focusing on thorough context review rather than critique, is able to avoid this issue when formulating action plans. Second, RCI executes actions sequentially based on a fixed initial plan, failing to adapt to changes in the environment that occur during execution. Conversely, the CAAP method is designed to account for state changes resulting from prior actions. This experiment underscores the effectiveness of our CAAP prompting, confirming its potential to enhance the performance of LLMs in computer tasks.

## 6 Conclusion

In this paper, we introduce a modular agent designed to mimic how humans solve tasks on computers. Specifically, we aim to address the limitations of existing models that rely on imitation learning with large quantities of human demonstrations or non-universal input sources such as HTML and DOM. Our agent operates solely on visual inputs and significantly reduces the need for human demonstrations by leveraging the common knowledge and reasoning capabilities built into LLMs. Additionally, we introduce the CAAP prompting mechanism to enhance the decision-making capabilities of LLM-based agents on computer tasks. Evaluations on the MiniWoB++ and WebShop benchmarks demonstrate that our agent outperforms previous methods while using significantly less data.
By adopting a modular approach that separates visual perception, action planning, and action execution, our method allows for easy adaptation to new tasks or model upgrades. Additionally, our agent is designed to operate beyond web environments and across inter-application settings, offering broader applicability and significant advantages in terms of generality.

## Appendix

## Appendix A MiniWoB++ Experiment Details

#### Models for visual observer

We utilize Ultralytics YOLOv8.0.143 (), employing the default hyperparameters with the exception of ’box’, ’cls’, ’dfl’, and ’max_patches’ set to 0.9, 0.05, 0.05, and 512, respectively. For Pix2Struct, we use the base model provided at [https://huggingface.co/google/pix2struct-base](https://huggingface.co/google/pix2struct-base), without any modifications to the default configuration.

#### Fine-tuning dataset for visual observer

Data collection for fine-tuning the visual observer involves a two-step process: screenshot collection and annotation of the collected UI elements. Initially, the target task is manually performed while screenshots are taken at each step. Image comparisons are then conducted to remove duplicate screenshots. From 10 episodes for each of the 73 tasks, a total of 1,768 screenshots were collected. In the second stage, a GUI-based annotation tool, as depicted in Figure 6, is utilized. Each UI element within the screenshots is marked with bounding boxes, and specific attributes are assigned. These attributes are subsequently stored in a JSON format. Data augmentation techniques are also employed to enhance the dataset. These techniques include color adjustments, minor scaling of bounding box sizes, and the random insertion of Gaussian noise. As a result, the total number of annotated data items is tripled compared to the original dataset.

The YOLOv8 model is fine-tuned using the screenshots and their bounding boxes around the UI elements. The Pix2Struct model is fine-tuned using the masked screenshots (Fig. 2(a)) and the corresponding element attributes, excluding the positions of bounding boxes.

#### Human demo dataset

To further enrich the dataset, we augmented each action with a rationale statement. As illustrated in Figure 4, the performance of the CAAP agent significantly improves when rationales for the demonstrated actions are also provided. Instead of relying on demonstrators to articulate their reasoning—a method that is impractical and unscalable for deploying our CAAP agent to a wider range of applications—we employed an LLM (*i.e.* GPT-4) to generate these rationales in a fully automated way. For each action, a prompt comprising both the visual state and the corresponding action was presented to the LLM, as detailed in Figure 7. The rationales were then integrated with the action trajectories to produce comprehensive, high-quality demonstrations. An example is shown in the ‘Expert Demonstration’ section of the prompt in Figure 8.

#### Train and test episode split

To differentiate between the train datasets and the test dataset, we assign distinct seed numbers when generating individual task instances within MiniWoB++. Image annotation data was collected from tasks with seed numbers ranging from 1000 to 2999, while demonstration data was sourced from seeds ranging from 3000 to 3999. Testing was conducted using seeds ranging from 0 to 999.

## Appendix B Comparisons to other approaches

Table 4 summarizes the input types, agent architectures, training methodologies, and whether experimental results on MiniWoB++ and WebShop are reported for our method and other approaches.

**Table 4: Comparison of agent configurations in input types, architectures, training methodologies.**
| Method | Modality | Model | Training | Experiment |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Vision | Decision | Scheme | MiniWoB++ | WebShop |  |  |
| Human | - | - | - | - | - |  |
| WGE | DOM | - | DOM-NET | SL+RL | $\surd$ | - |
| WebN-T5 | HTML | - | WebN-T5-3B | SL | $\surd$ | - |
| RCI | HTML | - | gpt-4 | - | $\surd$ | - |
| AdaPlanner | HTML | - | text-davinci-003 | - | $\surd$ | - |
| WebShop | HTML+Image | ResNet | BERT, BART | SL+RL | - | $\surd$ |
| WebGUM | HTML+Image | ViT-B16 | Flan-T5-XL | SL | $\surd$ | $\surd$ |
| CC-Net | DOM+Image | ResNet | Transformer | SL+RL | $\surd$ | - |
| - | / Image only | - | - |  |  |  |
| Pix2Act | Image | Pix2Struct | Transformer | SL+RL | $\surd$ | $\surd$ |
| SeeClick | Image | ViT | Qwen-VL 9.6B | SL | $\surd$ | - |
| CAAP | Image | YOLOv8,<br>Pix2Struct | gpt-4-0125 | SL | $\surd$ | $\surd$ |

### B.1 Experiment result for MiniWoB++

The data requirements and reported performance in literature are described in Table 5. One notable approach is CC-Net, which reports results across the broadest range of MiniWoB++ tasks. However, this required the collection of 2.4 million demonstrations, demanding approximately 6,300 hours of human labor, and faces challenges in real-world applications due to its high reliance on DOM information. RCI (Kim et al., ) and AdaPlanner (Sun et al., ) achieved average task SRs above 92% with fewer than 100 demonstrations, but their dependence on HTML inputs limits their task coverage. When recalculating SRs, including unsupported or unreported tasks, their performance drops to 50.8% and 49.2%, respectively. WebGUM (Furuta et al., ) expanded task coverage by using both HTML and image inputs. However, despite utilizing 401K demonstrations, its results only showed a 1.9% improvement over RCI, achieving an SR of 52.7%. Pix2Act (Shaw et al., ) was the first method to achieve high task SRs on MiniWoB++ using image inputs exclusively. It leveraged 1.3 million demonstrations, achieving over 96% success on 58 tasks, with an SR of 55.8% across the same set of 100 tasks. In contrast, SeeClick, which also used image inputs but on a smaller demonstration dataset, failed to sufficiently demonstrate problem-solving ability. Our approach represents the most advanced image-only agent, solving 73 tasks with a high SR of 94.5%, using fewer than 1.8K demonstrations. Furthermore, it achieved a 63.3% SR on the 100-task range, 7.5% higher than Pix2Act, with 61 tasks demonstrating an SR of over 80%—five more tasks than Pix2Act. (See Table 12 of Appendix J.3.) This represents a significant improvement achieved with fewer than 100 human demonstrations, resulting from the powerful reasoning capabilities of the LLM and our carefully modularized agent design.

**Table 5: Comparison with other methods that solve MiniWoB++ problems. While the primary configuration of CC-Net utilizes the DOM+Image modality, we have also included the performance of the image-only modality in this table to provide a more in-depth comparison.**
| Method | Modality | Image datasize | Demo datasize | Reported SR | Reported tasks |
| --- | --- | --- | --- | --- | --- |
| Human | - | - | - | 93.5% | 104 |
| WGE | DOM | 0 | $\sim$4.8K | 66.5% | 48 |
| WebN-T5 | HTML | 0 | 12K | 48.4% | 56 |
| RCI | HTML | 0 | $\sim$0.1K | 94.0% | 54 |
| AdaPlanner | HTML | 0 | $\sim$0.1K | 92.9% | 53 |
| WebGUM | HTML+Image | 0 | 401K | 94.2% | 56 |
| CC-Net | DOM+Image | 0 | 2.4M | 93.6% | 104 |
| CC-Net | Image | 0 | 2.4M | 24.0% | 104 |
| Pix2Act | Image | 0 | 1.3M | 96.2% | 59 |
| SeeClick | Image | 0 | 2.8K | 69.4% | 55 |
| CAAP | Image | 1.76K | 0.1K | 94.5% | 73 |

### B.2 Experiment result for WebShop

Table 6 compares our approach with studies reporting performance on the WebShop benchmark. WebGUM was trained using both HTML and image inputs with 1K human demonstrations, achieving an average reward of 66.6, surpassing the performance of the initial model introduced with the WebShop benchmark, which was trained using an SL+RL approach. Pix2Act, designed to operate solely on image inputs, reported an average reward of 46.7 when trained with 1K human demonstrations using a combination of SL and RL techniques.
Our agent, utilizing a vision model training dataset collected through an automated process and just a single human demonstration, achieved a significantly higher average reward of 62.3, outperforming Pix2Act. This result is comparable to methods that leverage both HTML and image inputs, demonstrating that our approach can effectively function across web and non-web environments with minimal human effort in dataset collection.

**Table 6: Comparison with other methods that solve WebShop problems.**
| Method | Modality | Image datasize | Demo datasize | Task score | SR |
| --- | --- | --- | --- | --- | --- |
| Human | - | - | - | 82.1 | 59.6% |
| WebShop | HTML+Image | 0 | 10.6K | 62.4 | 28.7% |
| WebGUM | HTML+Image | 0 | 1K | 67.5 | 45.0% |
| Pix2Act | Image | 0 | 1K | 46.7 | N/A |
| CAAP | Image | 0.3K | 1 | 62.3 | 34.7% |

## Appendix C Element coverage of CAAP agent

Table 7 provides a comprehensive overview of the types of front-end elements handled by the agent, predominantly including well-known types of front-end objects. Beyond the traditional categories, this study introduces the concept of ’tabled text’ to describe texts commonly embedded within table structures, and presents ’draggable text’ to represent text elements of which its text can be partially selected. These additions ensure extensive coverage of both interactive and static elements found in varying types of applications, thereby enhancing the understanding capability of the visual environmental states. Additionally, our agent covers conventional shapes and icons. The shape type includes circle, triangle, and rectangle, distinguished by their subtype values. Moreover, the icon type encompasses subtypes such as "back," "delete," "important," "forward," "reply," "search," and "send," as identified in MiniWob++.

**Table 7: The front-end element coverage of CAAP agent.**
| Type | Subtype | Checked / focused / highlighted | Text | Coordinates |
| --- | --- | --- | --- | --- |
| Text | - | $\surd$ | $\surd$ | $\surd$ |
| Hyperlink | - | $\surd$ | $\surd$ | $\surd$ |
| Button | - | $\surd$ | $\surd$ | $\surd$ |
| Radio | - | $\surd$ | $\surd$ | $\surd$ |
| Checkbox | - | $\surd$ | $\surd$ | $\surd$ |
| Dropdown | - | $\surd$ | $\surd$ | $\surd$ |
| Input field | - | $\surd$ | $\surd$ | $\surd$ |
| Text area | - | $\surd$ | - | $\surd$ |
| Resize handle | - | - | - | $\surd$ |
| Scrollbar | - | - | - | $\surd$ |
| Tabled text | - | $\surd$ | $\surd$ | $\surd$ |
| Draggable text | - | - | $\surd$ | $\surd$ |
| Shape | $\surd$ | - | - | $\surd$ |
| Icon | $\surd$ | $\surd$ | - | $\surd$ |
| Image | - | - | - | $\surd$ |

## Appendix D Action coverage of CAAP agent

Tables 8 and Table 9 detail the action types supported by the CAAP agent and the requisite parameters, respectively. The action executor operates using atom actions based on mouse and keyboard interactions, specifically: mouse down, mouse move, mouse up, and type a key. However, when conveying candidate actions to an LLM, it is more effective to transmit them as high-level actions capable of altering the environment. Consequently, as indicated in Table 8, actions have been delineated into units such as clicking on an element, clicking an empty position, clicking on an element during pressing control key, and typing text.

**Table 8: The action coverage of our agent. ’Element ID’ is the ID number of the screen element to act on. The number is designated during the element detection procedure. ’Text to Type’ is which text to type on. ’Coordinates’ are the X and Y coordinates of the mouse down location.**
| Name | Required | Description |
| --- | --- | --- |
| click_element | element_id | This action moves the mouse pointer to a screen element and |
| performs a left-click to activate that element. |  |  |
| click_new_point | x, y | This action moves the mouse pointer to a screen location and |
| performs a left-click on the location. |  |  |
| control_click_element | element_id | This action click on a screen element while holding down the |
| ’control’ modifier key. It is used to select multiple elements. |  |  |
| type_text | string_to_type | This action makes keyboard typing actions to enter the text into |
| a specific screen element, Clicking on the element is essential to |  |  |
| give it focus, without which typing the text is impossible. |  |  |
| point_element | element_id | This action moves the mouse pointer to on top of an UI element |
| without clicking it. This sometimes activates the element |  |  |
| and reveals hidden menus or scrollbar. |  |  |
| press_control_A | - | This is ’Select All’. All text in the activated text field is highlighted. |
| press_control_C | - | This is ’Copy’. Highlighted text is copied to the clipboard. |
| press_control_V | - | This is ’Paste’. Text stored in the clipboard is pasted into the |
| selected area. |  |  |
| drag_mouse_hold_down | x, y | This action initiates the drag action sequence by clicking and |
| holding down the left mouse button. It is used to move objects on |  |  |
| the screenor to highlight a block of text. This action marks the |  |  |
| starting point of the dragging move. |  |  |
| drag_mouse_move | x, y | This action is the middle of the drag action sequence. It moves the |
| mouse pointer while holding down the left mouse button. When |  |  |
| used to move an object on the screen, the object will be dragged |  |  |
| to the current mouse location. When used to highlight text, it |  |  |
| will highlight up to the current mouse location. |  |  |
| drag_mouse_release | - | This action marks the end of the drag action sequence. It releases |
| the left mouse button, indicating that the drag move is finished. |  |  |
| starting point of the dragging move. |  |  |

**Table 9: Parameter candidates listed in the ‘required’ column of Table 8.**
| Name | Type | Description |
| --- | --- | --- |
| element_id | integer | The id number (ranged from 1 to N) of the screen element to point at. A list of elements |
| will be provided with the corresponding id numbers. |  |  |
| x | integer | The x coordinate of the click location. |
| y | integer | The y coordinate of the click location. |
| string_to_type | string | The text to type. |

## Appendix E Additional related work

#### Vision models for agents

Numerous advanced techniques have been developed for detecting and interpreting UI elements on screens. Faster R-CNN (Ren et al., ) employs a Region Proposal Network (RPN) to extract areas of interest, followed by accurate object detection within these regions. This approach offers high accuracy in complex scenes, though at the cost of slower processing speed. The Single Shot MultiBox Detector (SSD) (Liu et al., ) offers rapid inference by eliminating the RPN and maintains decent accuracy. To improve small object detection and reduce false positives, one can diversify default boxes and feature maps for the SSD. Co-DETR (Zong et al., ), which leverages a Transformer-based architecture, performs detection while learning the relationships between objects. This anchor-free approach simplifies the learning process and effectively understands object interrelations, but it requires significant time and resources for training. YOLO (Redmon et al., ) processes predictions for object classes and bounding boxes across the entire image in one go. YOLO has both high speed and high accuracy compared to most other models. Unlike the previously mentioned technologies that primarily focus on extracting positional and typological information of objects, Pix2Struct (Lee et al., ) specializes in understanding detected objects. Designed around a Vision Transformer (ViT)(Dosovitskiy et al., ) and a T5 transformer(Raffel et al., ), it takes an image as input and outputs structured text, playing a crucial role in extracting and analyzing detailed attributes and embedded text information. Our approach leverages the combination of YOLO and Pix2Struct for visual information processing, aiming to capitalize on the strengths of both high-speed object detection and detailed attribute analysis.

## Appendix F Case study of MiniWoB++ tasks

In this section, we closely examine the failure cases that arise during the execution of MiniWoB++ tasks by the CAAP agent. Figure 5 presents screenshots of selected exemplary failure cases. We have categorized these eight instances into three groups: incorrect directives, observation failures, and decision-making failures.

Figure: Figure 5: Fail cases of the CAAP agent.
Refer to caption: x6.png

#### Incorrect directives

In Figure 5, cases (a) and (b) correspond to failures due to incorrect directives. Case (a) involves a directive that is not a task command, exemplified by the nonsensical sentence "Thanks Bette, I’ll Mauris." Thus, this case is a MiniWoB++ benchmark error. Case (b) concerns a directive with a grammatical error. In the directive "I’d like to send the email from Elka to Ingaborg’s email," a human might recognize the grammatical mistake and deduce that the email should be forwarded to Ingaborg. However, in our failure case, the LLM inputs "Ingaborg’s email." into the field designated for the forward recipient, instead, and attempts to send the email.

#### Observation failures

In Figure 5, cases (c) and (d) exemplify failures due to observation errors. Case (c) involves a failure where the task required identifying an object labeled ’Maighdiln’. However, during the UI element understanding process, the text was misrecognized as ’Maighdin’ and conveyed it to the LLM, which incorrectly determined that the target object was absent, leading to failure. In case (d), the button marked ’ONE’ is obscured behind a button labeled ’TWO’. Although the ’ONE’ text is not visible in the screenshot, the presence of the obscured button can be visually confirmed. A human operator would likely attempt to move the ’TWO’ button to reveal and verify the text on the obscured button.

#### Decision making failures

In Figure 5, cases (e), (f), (g), and (h) illustrate failures resulting from decision-making errors. Case (e) exemplifies a task requiring the identification of the sixth word, ’sed.’ Although the visual observer correctly observed and reported the scene, the LLM incorrectly identified the sixth word. In case (f), the task was to find ’Cassandry’, which was not visible on the screen. The LLM initiated a progressive scroll but prematurely stopped scrolling and failed to make the correct decision to continue. Case (g) involved the LLM determining the coordinates of text-written blocks and predicting their positions to move an object labeled ’Hermione’ to the central left position. The LLM decided on the drag action but incorrectly positioned the endpoint, leading to failure. Case (h) required highlighting a specific paragraph within a document composed of multiple paragraphs. Our visual observer is designed to detect text on a line-by-line basis, making it a challenging task to predict paragraph structure from the coordinates of each line. In this case, our agent failed to accurately predict the paragraph configuration. These four cases represent failures due to incorrect judgments by the LLM, with particular difficulty, as seen in (g) and (h), in interpreting coordinates represented numerically.

Among the failure cases, a tiny portion of the cases result from errors in the benchmark directives. While roughly 40% arise due to observation errors inherent in the agent configuration based on visual observation data, the others are failures that occur within the decision making domain. However, most minor errors in directives or observations tend to be appropriately accounted for during the decision making process by the LLM, often leading to successful outcomes as much as failures. This tendency is expected to intensify as the reasoning capabilities of LLMs advance, thereby improving their task-solving abilities.

## Appendix G Annotation tool

Figure: Figure 6: Our GUI-based annotation tool enables efficient annotation of UI elements within screenshots.
Refer to caption: x14.png

## Appendix H Prompt examples

Figure: Figure 7: A prompt example of generating a rationale for an agent action.

Figure: Figure 8: An example of a CAAP prompt.

Figure: Figure 9: An example of LLM response to the CAAP prompt.

## Appendix I Function calling example

Figure: Figure 10: The example provided illustrates the conversion of action type candidates into the input format supported by OpenAI API for function calling. As demonstrated in this example, information about action types can be conveyed through either of a function calling interface of an LLM service API or a prompt that includes the description of the candidates in plain text.

## Appendix J Task-level result of MiniWoB++ Experiments

### J.1 Per-task results for ablation study on CAAP components

**Table 10: Per-task SRs with respect to varying combinations of prompt components.**
| Task | Full | Demo without | CoT phrases | CoT phrases with | No CoT, |
| --- | --- | --- | --- | --- | --- |
| configurations | CoT phrases | without demo | action-only demo | no demo |  |
| choose-list | 1.000 | 0.980 | 0.940 | 0.920 | 0.980 |
| circle-center | 1.000 | 1.000 | 1.000 | 0.980 | 0.980 |
| click-button | 0.980 | 0.980 | 0.980 | 0.980 | 0.980 |
| click-button-sequence | 0.920 | 0.920 | 0.920 | 0.920 | 0.920 |
| click-checkboxes | 0.980 | 1.000 | 1.000 | 1.000 | 0.960 |
| click-checkboxes-large | 1.000 | 0.960 | 1.000 | 1.000 | 0.860 |
| click-checkboxes-soft | 1.000 | 0.980 | 0.980 | 1.000 | 0.960 |
| click-checkboxes-transfer | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| click-collapsible | 0.920 | 0.140 | 1.000 | 0.960 | 0.340 |
| click-collapsible-2 | 0.900 | 0.900 | 0.920 | 0.920 | 0.780 |
| click-dialog | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| click-dialog-2 | 1.000 | 1.000 | 0.980 | 1.000 | 1.000 |
| click-link | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| click-menu | 0.800 | 0.880 | 0.360 | 0.400 | 0.360 |
| click-menu-2 | 0.920 | 1.000 | 0.560 | 0.660 | 0.780 |
| click-option | 0.960 | 0.960 | 0.960 | 0.980 | 0.940 |
| click-scroll-list | 0.940 | 0.940 | 0.860 | 0.920 | 0.700 |
| click-tab | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| click-tab-2 | 0.960 | 0.980 | 0.980 | 0.960 | 0.980 |
| click-tab-2-easy | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| click-tab-2-hard | 0.980 | 1.000 | 1.000 | 1.000 | 1.000 |
| click-tab-2-medium | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| click-test | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| click-test-2 | 0.920 | 0.920 | 0.920 | 0.920 | 0.920 |
| click-test-transfer | 0.920 | 0.920 | 0.920 | 0.920 | 0.920 |
| click-widget | 1.000 | 0.960 | 0.860 | 0.860 | 0.860 |
| copy-paste | 0.980 | 0.980 | 0.980 | 0.980 | 0.960 |
| copy-paste-2 | 1.000 | 1.000 | 0.820 | 0.980 | 0.780 |
| drag-box | 1.000 | 1.000 | 0.980 | 1.000 | 0.960 |
| drag-circle | 0.980 | 1.000 | 0.960 | 0.980 | 1.000 |
| drag-items | 0.880 | 0.860 | 0.940 | 0.940 | 0.580 |
| drag-items-grid | 0.780 | 0.620 | 0.600 | 0.780 | 0.620 |
| drag-single-shape | 0.940 | 0.980 | 0.980 | 0.940 | 0.980 |
| drag-sort-numbers | 0.720 | 0.480 | 0.560 | 0.620 | 0.160 |
| email-inbox | 0.960 | 0.960 | 0.860 | 0.900 | 0.720 |
| email-inbox-delete | 1.000 | 1.000 | 1.000 | 1.000 | 0.980 |
| email-inbox-forward | 0.960 | 0.980 | 0.640 | 0.960 | 0.620 |
| email-inbox-forward-nl | 0.960 | 1.000 | 0.460 | 0.920 | 0.520 |
| email-inbox-forward-nl-turk | 0.820 | 0.800 | 0.220 | 0.820 | 0.380 |
| email-inbox-important | 1.000 | 1.000 | 0.980 | 1.000 | 0.960 |
| email-inbox-nl-turk | 0.900 | 0.800 | 0.540 | 0.860 | 0.500 |

| Task | Full | Demo without | CoT phrases | CoT phrases with | No CoT, |
| --- | --- | --- | --- | --- | --- |
| configurations | CoT phrases | without demo | action-only demo | no demo |  |
| email-inbox-noscroll | 0.980 | 0.940 | 0.920 | 1.000 | 0.860 |
| email-inbox-reply | 0.960 | 0.980 | 0.940 | 0.900 | 0.960 |
| email-inbox-star-reply | 0.980 | 0.960 | 0.980 | 0.960 | 0.960 |
| enter-password | 1.000 | 1.000 | 1.000 | 1.000 | 0.980 |
| enter-text | 1.000 | 1.000 | 1.000 | 1.000 | 0.980 |
| enter-text-2 | 1.000 | 1.000 | 1.000 | 1.000 | 0.980 |
| enter-text-dynamic | 1.000 | 1.000 | 1.000 | 1.000 | 0.980 |
| find-greatest | 1.000 | 1.000 | 0.420 | 0.860 | 1.000 |
| find-word | 0.820 | 0.620 | 0.740 | 0.800 | 0.380 |
| focus-text | 1.000 | 1.000 | 1.000 | 1.000 | 0.980 |
| focus-text-2 | 1.000 | 1.000 | 1.000 | 1.000 | 0.980 |
| generate-number | 0.860 | 0.780 | 0.840 | 0.880 | 0.680 |
| guess-number | 0.980 | 1.000 | 0.860 | 0.960 | 0.900 |
| highlight-text | 0.920 | 0.960 | 0.680 | 0.900 | 0.940 |
| highlight-text-2 | 0.460 | 0.500 | 0.140 | 0.400 | 0.240 |
| identify-shape | 0.980 | 0.980 | 0.940 | 0.960 | 0.960 |
| login-user | 1.000 | 1.000 | 1.000 | 1.000 | 0.980 |
| login-user-popup | 1.000 | 0.820 | 0.440 | 0.440 | 0.440 |
| multi-layouts | 0.980 | 0.920 | 0.960 | 1.000 | 0.660 |
| multi-orderings | 1.000 | 1.000 | 0.960 | 1.000 | 0.660 |
| odd-or-even | 0.980 | 0.920 | 0.980 | 0.920 | 0.940 |
| read-table | 0.940 | 0.820 | 0.900 | 0.900 | 0.840 |
| read-table-2 | 0.740 | 0.700 | 0.720 | 0.720 | 0.700 |
| resize-textarea | 1.000 | 1.000 | 0.900 | 0.980 | 0.980 |
| scroll-text | 0.620 | 0.780 | 0.200 | 0.420 | 0.020 |
| scroll-text-2 | 1.000 | 0.980 | 0.720 | 0.880 | 0.780 |
| simple-algebra | 1.000 | 0.980 | 1.000 | 1.000 | 0.700 |
| simple-arithmetic | 1.000 | 1.000 | 1.000 | 1.000 | 0.940 |
| sign-agreement | 0.960 | 0.980 | 0.420 | 0.780 | 0.420 |
| unicode-test | 0.900 | 0.920 | 0.900 | 0.880 | 0.880 |
| use-autocomplete | 0.960 | 0.880 | 0.900 | 0.960 | 0.900 |
| use-spinner | 0.980 | 1.000 | 0.900 | 0.920 | 0.700 |
| Average SR | 0.945 | 0.922 | 0.850 | 0.910 | 0.812 |

### J.2 Comparison of the CAAP and RCI promptings

**Table 11: Per-task SRs for RCI agent with CAAP and original RCI agent.**
| Task | RCI | RCI |
| --- | --- | --- |
| agent | agent |  |
| with | (original) |  |
| CAAP |  |  |
| choose-list | 0.80 | 1.00 |
| click-button | 1.00 | 1.00 |
| click-button-sequence | 1.00 | 1.00 |
| click-checkboxes | 1.00 | 1.00 |
| click-checkboxes-large | 1.00 | 1.00 |
| click-checkboxes-soft | 0.98 | 0.84 |
| click-checkboxes-transfer | 1.00 | 1.00 |
| click-collapsible | 1.00 | 1.00 |
| click-collapsible-2 | 0.92 | 0.82 |
| click-color | 1.00 | 1.00 |
| click-dialog | 1.00 | 1.00 |
| click-dialog-2 | 1.00 | 1.00 |
| click-link | 0.98 | 1.00 |
| click-menu | 1.00 | 1.00 |
| click-option | 1.00 | 1.00 |
| click-scroll-list | 1.00 | 1.00 |
| click-shades | 1.00 | 1.00 |
| click-shape | 0.94 | 0.94 |
| click-tab | 1.00 | 1.00 |
| click-tab-2 | 0.92 | 0.94 |
| click-tab-2-hard | 0.98 | 0.96 |
| click-test | 1.00 | 1.00 |
| click-test-2 | 0.98 | 1.00 |
| click-widget | 1.00 | 1.00 |
| count-shape | 0.78 | 0.76 |

### J.3 Comparison of the CAAP agent with SoTAs in per-task result

**Table 12: Comparison with other works in terms of SR per task for MiniWoB++ problems.**
| Task | CAAP | SeeClick | Pix2Act | CC-Net | WebGUM | AdaPlanner | RCI | Human |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| bisect-angle | - | - | 0.960 | 0.970 | - | - | - | 0.920 |
| book-flight | - | - | - | 0.870 | 0.980 | - | - | 0.870 |
| chase-circle | - | - | - | 0.930 | - | - | - | 0.820 |
| choose-date | - | 0.020 | 0.790 | 0.970 | 1.000 | - | - | 0.970 |
| choose-date-easy | - | - | - | 0.990 | 1.000 | - | - | 0.990 |
| choose-date-medium | - | - | - | 0.990 | 1.000 | - | - | 0.980 |
| choose-list | 1.000 | - | - | 0.990 | 1.000 | 1.000 | 1.000 | 0.980 |
| circle-center | 1.000 | - | 0.960 | 0.970 | - | - | - | 0.960 |
| click-button | 0.980 | 0.960 | 0.990 | 1.000 | 1.000 | 1.000 | 1.000 | 0.980 |
| click-button-sequence | 0.920 | 0.860 | 0.990 | 1.000 | 1.000 | 1.000 | 1.000 | 0.940 |
| click-checkboxes | 0.980 | 0.780 | 1.000 | 0.980 | 1.000 | 1.000 | 1.000 | 0.970 |
| click-checkboxes-large | 1.000 | 0.020 | 0.990 | 0.710 | 0.990 | 1.000 | 0.940 | 0.870 |
| click-checkboxes-soft | 1.000 | 0.220 | 0.610 | 0.950 | 1.000 | 0.800 | 0.960 | 0.730 |
| click-checkboxes-transfer | 1.000 | 0.700 | 1.000 | 0.990 | 1.000 | 0.980 | 1.000 | 0.980 |
| click-collapsible | 0.920 | 1.000 | 0.940 | 1.000 | 1.000 | 1.000 | 1.000 | 0.990 |
| click-collapsible-2 | 0.900 | 0.480 | 0.970 | 0.980 | 0.950 | 0.840 | 1.000 | 0.970 |
| click-color | - | 1.000 | 0.990 | 1.000 | 1.000 | 1.000 | 1.000 | 0.970 |
| click-dialog | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| click-dialog-2 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 0.990 |
| click-link | 1.000 | 0.900 | 0.980 | 0.990 | 1.000 | 0.980 | 1.000 | 0.990 |
| click-menu | 0.800 | - | - | 0.940 | 0.970 | 0.780 | 1.000 | 0.970 |
| click-menu-2 | 0.920 | - | - | 0.830 | - | - | - | 0.980 |
| click-option | 0.960 | 1.000 | 1.000 | 0.990 | 1.000 | 1.000 | 1.000 | 0.990 |
| click-pie | - | 0.800 | 0.990 | 0.970 | 0.990 | - | - | 0.980 |
| click-scroll-list | 0.940 | - | - | 0.600 | 1.000 | 1.000 | 1.000 | 0.910 |
| click-shades | - | 0.020 | 0.990 | 1.000 | 1.000 | 1.000 | 1.000 | 0.910 |
| click-shape | - | 0.520 | 0.940 | 0.950 | 0.940 | 0.750 | 0.980 | 0.880 |
| click-tab | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 0.990 |
| click-tab-2 | 0.960 | 0.600 | 0.980 | 0.980 | 0.990 | 0.850 | 1.000 | 0.970 |
| click-tab-2-easy | 1.000 | - | 0.990 | 0.990 | - | - | - | 0.990 |
| click-tab-2-hard | 0.980 | 0.420 | 0.970 | 0.980 | 0.950 | 0.780 | 0.980 | 0.960 |
| click-tab-2-medium | 1.000 | - | 1.000 | 0.990 | - | - | - | 0.970 |
| click-test | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| click-test-2 | 0.920 | 0.940 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 0.990 |
| click-test-transfer | 0.920 | - | 1.000 | 1.000 | - | - | - | 0.990 |
| click-widget | 1.000 | 0.580 | 1.000 | 1.000 | 1.000 | 1.000 | 0.980 | 0.830 |
| copy-paste | 0.980 | 0.800 | - | 0.790 | - | - | - | 0.940 |
| copy-paste-2 | 1.000 | 0.800 | - | 0.630 | - | - | - | 0.940 |
| count-shape | - | 0.280 | 0.700 | 0.850 | 0.680 | 0.500 | 0.400 | 0.820 |
| count-sides | - | - | 1.000 | 1.000 | - | - | - | 0.980 |
| drag-box | 1.000 | - | 0.990 | 1.000 | - | - | - | 0.990 |
| drag-circle | 0.980 | - | - | - | - | - | - | - |

| Task | CAAP | SeeClick | Pix2Act | CC-Net | WebGUM | AdaPlanner | RCI | Human |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| drag-cube | - | - | - | 0.790 | - | - | - | 0.990 |
| drag-item | - | - | 1.000 | 1.000 | - | - | - | 0.980 |
| drag-items | 0.880 | - | 1.000 | 0.990 | - | - | - | 0.930 |
| drag-items-grid | 0.780 | - | 0.890 | 0.980 | - | - | - | 0.870 |
| drag-shapes | - | - | 0.980 | 0.990 | - | - | - | 0.960 |
| drag-single-shape | 0.940 | - | - | - | - | - | - | - |
| drag-sort-numbers | 0.720 | - | 0.950 | 0.970 | - | - | - | 0.920 |
| email-inbox | 0.960 | 0.800 | - | 1.000 | 1.000 | 0.980 | 0.980 | 0.960 |
| email-inbox-delete | 1.000 | - | 1.000 | 1.000 | - | - | - | 0.990 |
| email-inbox-forward | 0.960 | - | - | 1.000 | - | - | - | 0.960 |
| email-inbox-forward-nl | 0.960 | 0.740 | - | 1.000 | 1.000 | 1.000 | 1.000 | 0.910 |
| email-inbox-forward-nl-turk | 0.820 | 0.560 | - | 1.000 | 1.000 | 1.000 | 0.940 | 0.880 |
| email-inbox-important | 1.000 | - | 1.000 | 1.000 | - | - | - | 0.990 |
| email-inbox-nl-turk | 0.900 | 0.680 | - | 1.000 | 1.000 | 0.900 | 0.980 | 0.930 |
| email-inbox-noscroll | 0.980 | - | - | 1.000 | 0.000 | - | - | 0.960 |
| email-inbox-reply | 0.960 | - | - | 1.000 | - | - | - | 0.910 |
| email-inbox-star-reply | 0.980 | - | - | 1.000 | - | - | - | 0.950 |
| enter-date | - | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 0.960 | 0.970 |
| enter-password | 1.000 | 1.000 | - | 1.000 | 1.000 | 0.980 | 1.000 | 0.960 |
| enter-text | 1.000 | 1.000 | - | 1.000 | 1.000 | 0.980 | 1.000 | 0.980 |
| enter-text-2 | 1.000 | - | 0.970 | 0.980 | - | - | - | 0.910 |
| enter-text-dynamic | 1.000 | 1.000 | - | 1.000 | 1.000 | 0.960 | 1.000 | 0.970 |
| enter-time | - | - | 1.000 | 0.970 | 1.000 | 0.960 | 1.000 | 0.980 |
| find-greatest | 1.000 | - | - | - | - | - | - | - |
| find-midpoint | - | - | 0.960 | 0.970 | - | - | - | 0.940 |
| find-word | 0.820 | 0.100 | - | 0.880 | - | - | - | 0.960 |
| focus-text | 1.000 | 1.000 | - | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| focus-text-2 | 1.000 | 0.960 | - | 1.000 | 1.000 | 0.940 | 1.000 | 0.990 |
| generate-number | 0.860 | - | - | - | - | - | - | - |
| grid-coordinate | - | 0.520 | 0.920 | 1.000 | 1.000 | 1.000 | 1.000 | 0.870 |
| guess-number | 0.980 | 1.000 | - | 1.000 | 0.430 | 0.880 | 0.200 | 0.990 |
| highlight-text | 0.920 | - | - | 1.000 | - | - | - | 0.970 |
| highlight-text-2 | 0.460 | - | - | 1.000 | - | - | - | 0.970 |
| identify-shape | 0.980 | 0.680 | 1.000 | 1.000 | 1.000 | 0.960 | 1.000 | 0.980 |
| login-user | 1.000 | 1.000 | - | 1.000 | 1.000 | 1.000 | 1.000 | 0.960 |
| login-user-popup | 1.000 | 0.980 | - | 1.000 | 1.000 | 0.980 | 0.680 | 0.940 |
| moving-items | - | - | - | 0.880 | - | - | - | 0.180 |
| multi-layouts | 0.980 | 0.720 | - | 1.000 | 1.000 | 0.840 | 0.960 | 0.950 |
| multi-orderings | 1.000 | 0.860 | - | 1.000 | 1.000 | 1.000 | 1.000 | 0.960 |
| navigate-tree | - | 0.820 | 0.990 | 0.990 | 1.000 | 0.820 | 1.000 | 0.980 |
| number-checkboxes | - | - | 0.840 | 0.990 | - | - | - | 0.960 |
| odd-or-even | 0.980 | - | - | - | - | - | - | - |
| read-table | 0.940 | 0.720 | - | 0.970 | - | - | - | 0.970 |
| read-table-2 | 0.740 | - | - | 0.940 | - | - | - | 0.950 |
| resize-textarea | 1.000 | - | 0.990 | 1.000 | - | - | - | 0.940 |
| right-angle | - | - | 0.970 | 0.980 | - | - | - | 0.870 |

| Task | CAAP | SeeClick | Pix2Act | CC-Net | WebGUM | AdaPlanner | RCI | Human |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| scroll-text | 0.620 | - | - | 0.960 | - | - | - | 0.970 |
| scroll-text-2 | 1.000 | - | - | 1.000 | - | - | - | 0.970 |
| search-engine | - | 0.840 | - | 1.000 | 0.960 | 1.000 | 1.000 | 0.970 |
| simon-says | - | - | - | 0.000 | - | - | - | 0.620 |
| simple-algebra | 1.000 | 0.380 | 1.000 | 0.750 | - | 0.820 | 1.000 | 0.860 |
| simple-arithmetic | 1.000 | 0.780 | 1.000 | 0.860 | - | - | - | 0.960 |
| sign-agreement | 0.960 | - | - | - | - | - | - | - |
| social-media | - | - | - | 0.900 | 1.000 | 0.820 | 0.980 | 0.960 |
| social-media-all | - | - | - | 0.750 | 0.520 | 1.000 | 1.000 | 0.890 |
| social-media-some | - | - | - | 0.850 | 0.730 | 0.900 | 0.960 | 0.910 |
| terminal | - | - | - | 0.000 | - | 0.980 | 1.000 | 0.880 |
| text-editor | - | - | - | 0.980 | - | - | - | 0.880 |
| text-transform | - | 0.460 | 0.920 | 0.600 | - | - | 0.800 | 0.860 |
| tic-tac-toe | - | 0.580 | 0.830 | 0.830 | 0.560 | 0.480 | 0.560 | 0.710 |
| unicode-test | 0.900 | 0.000 | 1.000 | 1.000 | - | - | - | 0.990 |
| use-autocomplete | 0.960 | 0.820 | 0.990 | 1.000 | 0.980 | 0.880 | 0.580 | 0.980 |
| use-colorwheel | - | - | 0.970 | 0.980 | - | - | - | 0.900 |
| use-colorwheel-2 | - | - | 0.950 | 0.950 | - | - | - | 0.940 |
| use-slider | - | 0.320 | 0.920 | 0.910 | - | - | - | 0.980 |
| use-slider-2 | - | - | 1.000 | 0.950 | - | - | - | 0.970 |
| use-spinner | 0.980 | 0.160 | - | 1.000 | 0.110 | 0.900 | 0.960 | 0.980 |
| visual-addition | - | - | 1.000 | 0.990 | - | - | - | 0.970 |
| Average SR | 0.945 | 0.694 | 0.962 | 0.936 | 0.925 | 0.929 | 0.940 | 0.935 |

## Appendix K Task coverage of the agents that rely solely on screen images for MiniWoB++

Figure: Figure 11: Task coverage of the agents with image-only inputs for MiniWoB++
Refer to caption: x15.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: What is the reason for converting screenshots to text for LLM and not using a VLM as there is other visual information that isn't captured?
What is the reason for the choice of MiniWoB++?
Why fine-tune the visual observer (YOLO + Pix2Struct) for this particular dataset when you could have auxiliary losses for VLM? 
Why have an intermediate representation and not have a more end-to-end approach? 
Why is having a VLM be an action planner preferable to other longer-horizon policy learning methods that use semantics as part of their input observation?
soundness: 3
strengths: This method makes a strong push for an agent that interacts with computer tasks using only screenshot images and standard input devices (keyboard and mouse), which more closely mimics human interaction than methods that use HTML or APIs. The overall paper is fairly clear and there is a good suite of baseline experiments and decent ablations. The originality of investigating prompting structure for agentic interactions with standrad input devices is good and is an important direction for future research.
summary: This paper introduces CAAP, which is a novel framework for automating complex computer tasks using LLMs by interacting solely through the front-end UI. Unlike previous approaches that rely on HTML source code or application-specific APIs, CAAP perceives its environment through screenshot images, which are then converted into text using a visual observer module (YOLO and Pix2Struct). This text is processed by an LLM with CAAP, which includes human demonstrations, surrounding context information, CoT-inducing instructions, and lastly some extra guidelines. The agent executes keyboard and mouse operations to interact with the GUI, eliminating the need for pre-defined APIs or large-scale human demonstration data. Experiments on the MiniWoB++ and WebShop benchmarks demonstrate that CAAP outperforms existing methods.
weaknesses: The main novelty seems to be the CAAP prompting because converting the screenshots to text is done with fine-tuned YOLO and Pix2Struct models. Although Figure 4 is good at ablating different components of CAAP, it would be beneficial to include more analysis of these components and what traits help the prompting. Since the novelty is predicated on this prompting strategy, I believe a more thorough answer to the question of the best-prompting strategy for agents interfacing with just screenshots is interesting and needed. It would also help if Figures 1 and 2 would be more clear using the captions. Also, it would be good to discuss the limitations of the method.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Is the proposed model specific to certain UI elements in miniwob++ and webshop benchmarks or are they generally applicable?
1. How does the model handle dynamic UI elements (pop-ups), screen resolution, and other visual features when they change position or appearance?
1.  What was the process for selecting the "single human demonstration" used in WebShop instructions? How representative is it? How does it relate to "300 samples collected through a fully automated process"?
soundness: 3
strengths: 1. Modular framework: focus on visual observation, decision making and action execution aspects to solve computer tasks. The core components make it suitable for future extensions, maintainability and scalability.
1. Extensive discussion on prompt: for in-context learning from few-shot examples, context info, CoT, extra guidelines. The impact of this prompting scheme is shown as ablation with existing work RCI.
1. Open-source code and models
summary: The paper proposes a LLM-agent based framework for computer use that uses only screenshot images as input, converting them to text for LLM processing, operating through basic keyboard and mouse actions on GUI and reducing HTML code and specific APIs. The proposed automation solution CAAP achieves 94.5% success rate on MiniWoB++ benchmark and 62.3 average task score on WebShop as compared to image-only based agents.
weaknesses: 1. Architectural Complexity and overhead: seems more complex than end-to-end solutions due to modular design. This also creates specific APIs requirement to connect different modules, fine-tuning for vision models for screenshot specific recognition. It is unclear on what are the implications of these on robustness of the proposed framework.
1. Limited real world evaluation: while the proposed approach is tested on miniwob++ and webshop, it'll be useful to demonstrate it on real world tasks and analyze failure cases.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - What other applications could CAAP be applied to?
- On line 343, it is stated that some tasks were excluded from the agent’s training. How are other tasks used for training? Aren’t these demonstrations just used at inference as part of the prompt?
- What is the advantage of using a semi-masked screenshot rather than a fully masked or cropped screenshot of the element?
soundness: 2
strengths: - The paper addresses an interesting problem of solving computer tasks using LLMs and visual inputs.
- The engineering effort of integrating different modules into a unified framework is well-executed and appreciated.
- The provided implementation and experimental setup details are a valuable part of the paper.
summary: This paper introduces a modular agent to solve computer tasks using screenshot images (that are converted into text), Large Language Models,  and a Graphical User Interface. UI elements are first detected and formatted into text using fine-tuned models. Then, a large language model decides appropriate actions using a specific prompting technique. Finally, the actions are executed using the keyboard and mouse interfaces. As such, the model doesn't rely on HTML or the Document Object Model as input sources. The proposed prompting technique, CAAP, structures the available context and introduces chain-of-thought reasoning. Experiments on MiniWoB++ and WebShop benchmarks show good performance with a limited number of expert demonstrations.
weaknesses: - It is not clear why the “UI element understanding” module needs to understand the spatial relationships between the element and its adjacent elements (line 173). It seems that only the element's position is extracted. This should be clarified in the paper.
- The proposed prompting technique is described only superficially. Examples and more detailed discussions about each part of the prompt should be included in the main paper.
- The proposed prompting technique seems to be quite tailored towards a specific application and tested for a single language model. Therefore, it comes off as a hack rather than a novel contribution. Overall, the paper has limited scientific contribution. I would suggest the authors to focus more on investigating the influence the proposed technique has on In-Context Learning overall and highlighting their contribution in this way.
- The paper would benefit from a discussion about using VLMs, rather than a separate vision module and an LLM. Could a VLM be used to complete tasks in full or to replace the vision module, eliminating the need for fine-tuning different vision models?
- The comparison to different methods evaluated on different numbers of tasks doesn’t hold a lot of scientific value. Maybe the metrics only for the tasks all methods were evaluated on could be reported?
- The success rate range in Figure 4 (x-axis) overemphasises the performance of the proposed method when compared to ablation variants. Adjusting the range of choosing a different format for the graph could fix this issue.

Meta-review: The paper proposes a modular LLM-based agent for solving GUI-based tasks using screenshots. The work addresses an important problem and demonstrates impressive engineering efforts. The proposed method achieves good performance on two benchmarks. 

However, reviewers raised significant concerns:

- While the modular design and the prompting method are interesting, the reviewers found the contributions insufficiently explained or lacking broader generalization. The CAAP prompting was perceived as tailored toward the specific application and not rigorously validated across diverse situations.

- The focus on MiniWoB++ and WebShop benchmarks without real-world scenarios undermines claims of generalizability. The lack of empirical results demonstrating scalability and applicability to various environments was a common critique.

- Reviewers noted unclear explanations of architectural designs and figures. More detailed discussions on prompting strategies and limitations were requested.

While the paper is promising, it needs to further improve the rigor and clarity. Therefore, I recommend rejection.

Decision: Reject