Autonomous Evaluation and Refinement of Digital Agents
Authors: Jiayi Pan
Yichi Zhang
Nicholas Tomlin
Yifei Zhou
Sergey Levine
Alane Suhr
Venue: COLM (2024)
Keywords: Language Agent, Autonomous Refinement, Automatic Evaluation
TL;DR: We study the design and use of automated evaluation models to both evaluate and autonomously refine the performance of digital agents, with our experiments confirming their effectiveness.

Abstract: We show that domain-general automatic evaluators can significantly improve the performance of agents for web navigation and device control. 
We experiment with multiple evaluation models that trade off between inference cost, modularity of design, and accuracy. 
We validate the performance of these models in several popular benchmarks for digital agents, finding between 74.4 and 92.9% agreement with oracle evaluation metrics.
Finally, we use these evaluators to improve the performance of existing agents via fine-tuning and inference-time guidance.
Without any additional supervision, we improve state-of-the-art performance by 29% on the popular benchmark WebArena, and achieve around 75% relative improvement in device control settings.
We release our code and data at 
 [https://github.com/Berkeley-NLP/Agent-Eval-Refine](https://github.com/Berkeley-NLP/Agent-Eval-Refine)

Published as a conference paper at COLM 2024
Autonomous Evaluation and Refinement of Digital Agents
Jiayi Pan Yichi Zhang² Nicholas Tomlin Yifei Zhou Sergey Levine' Alane Suhr!
UC Berkeley Uniiversity of Michigan
Abstract
We show that domain-general automatic evaluators can significantly im- prove the performance of agents for web navigation and device control. We experiment with multiple evaluation models that trade off between inference cost, modularity of design, and accuracy. We validate the perfor- mance of these models in several popular benchmarks for digital agents, finding between 74.4 and 92.9% agreement with oracle evaluation metrics. Finally, we use these evaluators to improve the performance of existing agents via fine-tuning and inference-time guidance. Without any addi- tional supervision, we improve state-of-the-ar performance by 29% on the popular benchmark WebArena, and achieve around 75% relative im- provement in device control settings. We release our code and data https://gi http:///rrrellleell///eeeee---e Ref
1 Introduction
Given an instruction, e.g, "Tell me the cost of my latest canceled order," " an automated digital agent would be expected to first navigate to user's profile page, then to list of their previous orders, identify the most recent order that has been canceled, and return its total amount to the user. Such agents offer the long-term potential of making digital devices more accessible, while also simplifying tedious or mundane tasks. However, in the short term, even state-of-the-art agents still make mistakes on simple tasks. Evaluating such agents and characterizing their failure modes is not only important for understanding and improving the models, but also critical for safely deploying them real world. In this paper, demonstrate the opportunities and efficacy of using automated evaluation models to both characterize and improve agent performance, without requiring access to any extra supervision, such expert demonstrations evaluation functions.
We propose to automatically evaluate user instructions and arbitrary agent trajectories with domain-general neural models. We explore two main variants of this approach (Figure I left): first, a modular caption-then-reason approach where a vision-language model (VLM) first captions the screenshots, and a language model (LM) is used to reason about if agent succeeds based on textual information, and second, an end-to-end approach where we
prompt an advanced VLM like GPT--V (Achiam et al. 2023) directly evaluate trajectory. T'esse two different approaches offer trade-ofts in pertormance, cost, and transparency.
We first evaluate our proposed approach on its ability to match oracle evaluation metrics using WebArena (Zhou et al. 2024) and Android-in -itrridd----te-Will (AitW; Rawles et al. 2023), achieving accuracies up to 82.1 and 92.9% respectively. We then show how these evaluation models can be used to refine existing agents through inference-time guidance or during training, without access to any hand-designed evaluation functions or additional demon- stration data (Figure [ right). When integrated as the reward function in Reflexion (Shinn et al. 2023), the evaluator enhances the rest-performing GPT-4 WebArena agent's success rate by up to 29% of relative improvement. Additionally, iOS device control settings. For iOS, there is no existing benchmark environment or training data. When using our evaluation models filter sampled trajectories to be used in behavior cloning, we see relative improvements of around 75%.
evaluate in both Android and
Email:: jiayiann@berkeley.edu

---

Published as a conference paper at COLM 2024
Policy Evaluation
Autonomous Refinement
Inference-tinge time Guidance (Reflexion)
User Instruction
Tell me the cost intest conceted onder
Agent Trajectory
Instruction
States and actions
Agent Trajectory
Reflect
Evoluator
Retry
XFAILURE
$174.99
Filtered Behavior Cloning
Modular
End-to-End Evaluator
Instructions
Evaluator
VLM based Captioner
Evaluator
Trajectories
VLM-basse Reasoner
Reasoner
Fiitered State-Action Pairs
Behavlor Cloning
Evalucrtion
Fine--tning
X FAILURE
Figure 1: Method overview: A model-based evaluator provides evaluation of a digital agent's trajectory (left). can be used as the reward function for Reflexion (Shinn et al.) 2023)
filtered behavior cloning to enhance model performance (right).
Related Work
Building automated digital agents that map from user instructions to executable actions has been long-standing goal in the NLP and Al communities (Allen et al. 2007 Branavan et al., 2009: 2010). Recent advances in NLP and multimodal machine learning have supported the development more capable agents, and many recent benchmarks and approaches cover instruction-conditionnel tasks such web navigation and device control.
Digital Agents Early modeling of language-conditioned autonomous agents focused on approaches that include semantic parsing (Allen et al. 2007 Xu et al., 2021 Li al. 2020).
reinforcement learning (Branavan an et al. 2009 | 2010),, and imitation learning (Humphrey)
et al. 2022). The strength of pretrained language and language-an -nnguaag-ann-vvsionn modeling has
renewed interest in building language-conditiones digital agents (Zhang & Zhang 2023 ?;
Zhou et al. 2024 Deng et al. 2023 Wang et al. 2023a Gur al. 2024). For For example, baseline
approaches to WebArena (Zhou et al. 2024) use few-shot prompting with language-only
models, representing the environment state and action space with its document object
model (DOM). More recent works in building these agents have moved from language-only modeling to vision-language modeling, representing the environment state space as its rendered pixel representation instead of relying DOM. Another line of work has applied inference- time techniques to improve model's performance, for example with inference- time exploration (Zhang et al. (2023), intermediate plan revision (Zhang et al., 2024) and error correction (Wang et al. 2024), and self-critique (Wu et al. 2024) on GPT- or GPT-4V. Concurrent to our work, OS-Copilot (Wu et al., 2024) proposes self-critique component to autonomously refine Mac device control agents, implementing the critic as a LM that reasons about proposed tool implementations and error messages. In contrast our work, this critic does not evaluate actual agent behavior in the execution environment or used in model training.
Autonomous Refinement and Evaluation
Recently, there has been renewed interest in
methods for improving policies at training (Ouyang et al. 2022 Bai et al. 2022 Lee et al.
2023 Abdulhai et al. 2023) inference (Shinn et al. 2023 Yao et al. 2023
Wu et 2024
2

---

Published as a conference paper at COLM 2024
time without human supervision. Approaches like Reflexion (Shinn et al. 2023) assume access to an external evaluation function, leveraging it as the supervision signal to guide policy improvement at inference time. In contrast, we study applications of inference- time autonomous refinement without requiring access to the external evaluation function, and show that using our proposed domain-general evaluation models improves agent success rate by 29%. Meanwhile, methods which bootstrap policies with supervised training and refine them with reinforcement learning have been widely adopted; our proposed evaluators enable this paradigm in open, realistic digital agent scenarios, providing relative improvement in performance of over 70%. Concurrent to our work, WebVoyager (He al., 2024) also explores using GPT-4V as an automated proxy for human evaluation of web agents, though is not the primary focus of their work, and neither performs in-depth analysis of the quality of its judgments, nor explores its applicability to improving agents. More broadly, model-based metrics have been highly successful across various domains, such as evaluating image or text quality (Heusel et al., 2017: Zheng et al., 2024), and assessing success of a robot or human activity (Du et al. 2023). Part of our contribution is to provide strong empirical evidence to extend this success to the domain of real-world digital agents.
Digital Agent Benchmarks Recently-proposed benchmarks that study digital agents fall roughly into two categories: simulation-based benchmark and demonstration-based one. Simulation-based benchmarks include environment simulators that offer the ability to execute arbitrary agent trajectories. Early simulation environments such as WoB (Shi al., 2017 Liu al. 2018), WebShop (Yao et al. 2022), and others (Branavan et al. 2009) are limited in their domain coverage, realism, or generalizability of their evaluation tunctions. Recently proposed simulation environments like Env (Toyama et al. 2021), WebArena (Zhou al. 2024) 2002) and VisualWeb (Koh et al. 2024), though far from perfect, have offered improvement across these dimensions. However, designing simulators, curating tasks, and handcrafting evaluation functions fundamentally limits their ability to mirror task and environment diversity of real environments.
In parallel, the community has focused on demonstration-basse benchmarks that do not include executable simulation environment, including PIXELHELP (Li et al.. 2020), MoTIF (Burns et al., 2022), Mind2Web (Deng et al. 2023), and AitW (Rawles et al. 2023). Notably, Mind2Web and AitW contain over 2K and 715K human trajectories respectively on a wide range of web navigation and device control tasks. Though primarily used for model training (Rawles et al.] 2023 Deng et al. 2023 ?; Zhang & Zhang (023), these datasets are also used for evaluating digital agents through reference-based metrics like action matching score. In this setting, an agent is given the prefix of human demonstration and evaluated on its prediction of the next action to take. However, this metric requires human demonstrations and does not directly reflect agent's performance in real-world because does not account for consequences of an agent's sequential decision process, alternative actions that diverge from the demonstration.
We propose third approach in which arbitrary instructions and agent trajectories are directly evaluated by a model.
3 Domain-General Evaluators
We develop multiple domain-general automatic evaluators for digital agents. Given a user instruction x and an initial environment state So, an agent generates and executes a sequence of actions a = (4o, A1,..., resulting in a sequence of state visits s = (So, $1, $2, ,,,,1.,,) In this work, we assume and in text form, such as <Type: 'Hello' '> and "Check the weather", and each state s is represented as a screenshot image. Given x, a, and as inp model produces scalar evaluation F trajectory:
Tn) corresponding to each step of the
F = evaluate (,,
The evaluator can provide either trajectory-level or per-step evaluations. For trajectory-level evaluation, to
Tn-1 = 0, with rn for successful trajectories and rn = 0 otherwise. For per-step evaluation, we classify each step into three types, r; = 1 indicates task success after action a; r; = p 2 0 indicates progress toward the goal, and r; d < 0 is assigned to
3

---

Published as a conference paper at COLM 2024
actions that do not contribute to the objective. We query the model once for trajectory-level
evaluation and и times for per-step evaluation, reducing the model's 's task into a binary or ternary classification problem at each step.
problee
We explore two methods for constructing the model:
1. An end-to-end approach that maps directly from instructions and screenshots to evaluation via a pre-trained VLM.
2. A modular approach which first transcribes the observed screenshots into text descriptions using a VLM, and then uses a LM to map the descriptions, actions, and user instruction onto an evaluation.
Both methods have tradeoffs: in the first, we can apply advanced VLMs like GPT-4V. However, this approach is relatively expensive and relies on API calls to proprietary models. In the second, can compose open-weight models to achieve slightly weaker performance, but with added benefits of explainability via modularity and low-cost local deployment.
3.1 End-to-End Approach
We directly provide an instruction-tuned VLM with x, a, and 5. We prompt to first produce a text-based reasoning process (Wei et al. 2022), then output its evaluation result. In our experiments, use the proprietary vision- model GPT-4V (Achiam et al. 2023
3.2 Modular Caption-then -easonn Approach
Many existing approaches for joint reasoning about language and vision disentangle percep- tion and reasoning. In these approaches, a VLM is first applied to visual input to generate danguage-based description; then, a text-only model (e.g., LM) takes as input this de- scription and the user instruction to produce a response by reasoning only about linguistic inputs. Existing work applying this approach has mostly focused on joint reasoning about natural images and text, e.g., for visual question answering (Guo et al. 2023 You et al., 2023 Wang al. 2023b). We take similar approach here, where we first description of the agent's 's observations given as 3, then feed these descriptions, along with actions a and the user's 's instruction x to an LM to produce a final evaluation
VLM to prod uce
Captioner One drawback to this modular approach the potential for information loss, where the image description may not include all the details necessary for task success (Wang) et al.) 2023b). In our case, this could include missing or misrepresenting details about the screenshot, and indeed, we find that current open-weight VL.Ms struggle to produce detailed screenshot descriptions out of the box. In contrast, the most advanced, yet proprietary, VLMs can produce very detailed descriptions with adequate prompting.
To improve a captioner's 's ability to provide detailed, well-formatted descriptions, we collect dataset of screenshots paired with descriptions, and use it to fine-tune an open-weight VLM. We first acquire screenshots from a variety web and device control domains, then use GPT-4V to provide an initial detailed description for each screenshot. We manually filter out fix apparent errors in GPT-4's output, resulting total of 1,263 data points We use this data to fine-tune the QLen-VL-chat (Bai et al. 2023) model. During both finetuning and at inference time, we provide text recognition results from EasyOCR as an additional input to the VLM reduce hallucination.
At inference time, we use our finetuned captioner model to acquire a description for each step in the agent trajectory. Critically, we do not provide this model access the original user instruction, as we find this exacerbates model hallucinations; e.g., describing webpage attributes which would be relevant the task, but are not actually present in the screenshot.
Reasoner Finally, we provide the actions, generated descriptions, and the original user instruction to language-only instruction-tuner model. We experiment with prompting two
I Prompt templates and additional details are provided in Appendix A.1
?Data collection process, hyper-parameters, and output examples are detailed in Appendix A.2 'able In Appendix A.2 contains details of data sources and sizes.
<nttps:/ thwbb www.tthu....me..eeeeeeeeeeeeeoee wttp///aade/////ayyyoooooooooa/o
4

---

Published as a conference paper at COLM 2024
LMs, Mixtral [iiang et al. 2024) and GPT-4, to produce text-based thought and reasoning process as well as the final evaluation.
4 Experiments and Results
Our goal is to show how domain-general evaluation models can support the autonomous evaluation and refinement of digital agents, without requiring access to human demonstra- tions or oracle evaluation metrics. To this end, we first evaluate how these models perform as autonomous evaluators by comparing their judgments to benchmark-provided metrics and human judgements (Section 4.1. We then illustrate how these evaluation models, while imperfect, can serve as discriminators in autonomous refinement settings through both inference-time policy refinement (Shinn et al. 2023) and filtered behavior cloning (filtered BC; Chen et al. 2020 2021: Emmons et al. 2022 to support significant improvements in agent performance (Section
Our rationale behind experiment design is to cover broad range of domains and chal- lenges. We use WebArena for both evaluation and inference-time refinement, its built-in evaluation functions facilitate direct comparison. Android-in-the-Win for evaluation since it is widely used for training and evaluating Android agents, and is typically evaluated using a reference-based metric instead of task success. Next, we refine a model through filtered behavior cloning on iOS, where data scarcity poses significant challenge to supervised methods. We further validate the effectiveness of filtered behavior cloning with a similar, larger experiment on Android.
(AitW) is chosen
Environments WebArena (Zhou et al. 2024) is offline web emulation environment and dataset that supports execution of arbitrary policies. WebArena comprises 812 human- written task instructions across various domains, including shopping, maps, and content management systems. Each instruction is paired with handwritten test case that verifies agent success, e.g., by checking the status of specific webpage element against reference. We refer to this set of test cases as WebArena's 's oracle evaluator.
Android-in-the-Wil Wild (AitW; Rawles et al. 2023) is a large-scale dataset for Android device control containing 715,142 human demonstrations of 30,378 unique instructions. In our experiments, we focus on a subset of 120 tasks randomly sampled from the AitW test set Unlike WebArena, AitW does not include an emulation environment for agent execution. Instead, the suggested evaluation metric is based on action matching: given sequence of actions representing the prefix of a human demonstration, the agent is evaluated on its ability to predict the next action in the demonstration. While we compare against this reference-basee metric in our experiments, we focus on end-to-end task-level success rate and implement an Android emulator to support execution of arbitrary trajectories We refer to human judgements trajectory success the oracle evaluation.
Despite significant interest in developing digital agents, progress in the domain of iOS device control has been modest, with the exception of Yan et al. (2023), who collect small unreleased dataset of human demonstrations in this domain. We curate a set of 132 tasks in the iOS domain, taking inspiration from tasks included in AitW. We experiment with using our proposed evaluation models to facilitate domain transfer, with the goal of applying the strongest model on AitW, CogAgent (?), to iOS. We develop a Python interface to the iOS emulator on macOS, and design its action space to align with the Android-in-the-Wee to facilitate domain transfer. 6
Evaluation Models
We evaluate three evaluation model variants:
GPT-4V: End-to-end approach (Section 3.1 using GPT-4V.
QWen-VL-chat: End-to-end approach (Section 3.1 using QWen-VL-chat. This serves as oeentt model's baseline performance.
5We subsample from the original test set of 1.4k tasks to facilitate acquiring human judgments of trajectories. See Appendix(A. AA4 for details on list of evaluated tasks and details task sampling. Details on our emulators are available in A Appendices and A.6
and
5

---

Published as a conference paper at COLM 2024
GPT-4V
QWen-VL-chat
Captioner Mixtral Captioner + GPT-4
WebArena (%)
80.6
68.0 70.2
74.4
82.1 89.8
Android (%)
90.6
92.9
Table 1: Comparison of evaluators accuracy against oracle evaluator or human judge in WebArena and Android.
Captioner + Mixtral: Modular approach (Section 3.2 using finetuned QWen-VL- chat (Bai et al. 2023) to generate a trajectory description, and Mixtral (Jiang al.
2024 to provide the final evaluation.
Captioner + GPT-4: Modular approach (Section 3.2) using a finetuned QWen-VL- chat to generate a trajectory description, and GP to provide the final evaluation.
In most experiments, the evaluation model produces trajectory-level evaluation, and takes as input only the last frame Sx+1 in the trajectory, along with the instruction x and action sequence ä. Preliminary experiments suggested that model performance does not improve with information about previous states, likely due to limitations of existing models in processing long contexts. In the iOS experiments, the evaluation model takes input the entire trajectory and à and the instruction x, and produces a per-step evaluation.
Agent Policies We experiment with evaluating and refining the current state-of-the-art digital agents. In WebArena, this is GPT-4-based agent described by Zhou et al. (2024). For each task, GPT-4 is provided the user's instruction and the current DOM representation of the webpage derived from its HTML accessibility tree. GPTT-4 is prompted to generate an action grounded in the DOM, e.g,, clicking button with a specific element ID. The task success rate of this agent is 14.4% reported in Zhou et al. 2024 whereas reproduction, as shown in Figure 3 indicates a success rate of 15.6%.
The strongest agent
the AitW benchmark is CogAgent (?), followed by Auto- UT (ange, base) (Zhang & Zhang 2023) These agents are implemented as neural vision- language models that map observations, represented as images, and instructions to exe- cutable actions. We also experiment with the human demonstrations provided in AitW
4.1 Automatic Evaluation
WebArena For each WebArena task and corresponding trajectory sampled from the GPT- 4-based policy (Zhou al. 2024), we acquire task-completion judgments for each of the three evaluation systems described above. [able[] the overall accuracy of the evalua- tor's 's predictions [8] The end-to-end approach with GPT-4V achieves 80.6% accuracy, while Captioner + Mixtral, which uses only open-weight models, matches the oracle's evaluations for 74.4% of tasks, and replacing Mixtral with GPTT44 achieves the highest accuracy at 82. Additionally, we see that using QWen-VL-chat VL-chat end-to-end significantly underperforms the modular, wpen-weight Captioner + Mixtral approach, confirming the effectiveness of our modular design choice.
For the 120 sampled test tasks in AitW, we evaluate trajectories sampled from four policies: CogAgent (?), Auto-Ul(large, base} (Zhang & Zhang 2023 and
Android-in-the-Will
human experts Rawless et al., 2023 We acquire human judgments trajectory success, as well as judgments from the three evaluation model variants 9 Figure 2 shows the performance of all four agents as evaluated by humans and the three evaluator variants.
The human demonstrations use the original AitW emulator, which was not released by the authors; thus, these results are not directly comparable with the automated policies, which use the emulator we implement. However, the focus of our experiments is not to directly compare policies, but to compare evaluators across variety of policies, tasks, and domains.
8 Figure 4 in Appendix|A.. A.3 includes the confusion matrices these predictions. Experimental setup details for AitW are provided in Appendix A.4

---

Published as a conference paper at COLM 2024
64.2
Human
60
GPT-4V
55.0
% Successful Trajectories
Captioner + GPT-4
52.5
50
Captioner + Mixtral
45.6
40
30
20
16.7
14.2
14.2
10
9.2
7.5
8.3
5.0
1.7 0.8 0.0
AutoUl-base (67.5%)
AutoUl-large (70.0%)
CogAgent (70.7%)
Human Reference (100%)
Policy Model (Action Matching Score)
Figure 2: Evaluating models in Android-in-the-Wiile with different evaluation methods.
We use human judgments of trajectory success oracle reference and compare it with judgments from our evaluation models and AitW's standard action matching score.
Below each agent label we also include each policy's partial action match score (Li et al. 2020 which is the standard reported metric for agents on AitW 10
Unsurprisingly, we find that the human reference trajectories achieve the highest perfor- mance as evaluated by all success metrics. However, our analysis reveals that about 36% of the human demonstrations we annotate are actually unsuccessful, with common errors including early stopping, completing the wrong task, and making mistakes with respect to the parameters of the task. The difficulty of collecting high-quality demonstration data at scale further demands automated evaluation methods that can either act as a quality filter or provide more direct evaluation than action matching score.
Among the three neural network policies, CogAgent achieves the highest success rates, followed by Auto-Ulbase while the performance of Auto-Ullarge is close to zero according to all evaluators. When comparing conclusions that can be drawn from the two styles of metrics task success and action matching there three clear differences: first, that success rate lags far behind single-step action prediction; second, that relative performance of models changes depending on the metric used; and third, that using reference- based metric on erroneous references could result in inflated impressions of model performance. In particular, while Auto-Ulaarg appears outperform Auto-Ulbase according the action matching metric, it is clearly inferior in terms of overall task success rate. Quantitatively, all three evaluators achieve a Kendall correlation of 100% with the human judges, while the action matching score only obtains 66.7%. This highlights a fundamental drawback single-step metric like action matching: it does not reflect error propagation or distribution shift in the sequential prediction process of an arbitrary policy, which can be captured by whole-trajectory success metrics.
Measuring whole-trajectory success for the complex tasks that digital agents complete has typically required either human evaluation of individual trajectories, or manual creation of individual test cases, as in WebArena. We analyze the potential for automating this process using three proposed evaluators. Table TableeSsowe shows the accuracy of each evaluator variant aggregated over trajectories from all four policies Overall, FOverilssJOveral,,,,, metrics correlate very strongly with human judgment: the Captioner + Mixtral variant shows the highest agreement with human judgment at 92.9% accuracy; replacing Mixtral with GPTT4 leads to a performance drop to 89.8%; and the end-to-end approach of GPT-4V
find that automated
Action matching scores are averaged across the subsets of AitW we sample from, reported in and Zhang& Zhang (2023).

---

Published as a conference paper at COLM 2024
Oracle Evaluator
GPT-4V
26
Captioner GPT-4
24.8%
Captioner Mixtral
24
Success Rate
21.9%
22
20.4%
20.2%
19.
18.6%
19.0%
18.1%
18
15.6%
18.0%
17 7.9%
18.1%
16
Baseline w/o Reflexion
1
2
Trial Number
Figure 3: Results of applying Reflexion for up to rounds using different evaluation systems on the WebArena benchmarks. Here, the oracle evaluator denotes performance using WebArena's built-in evaluation functions the reward function; this provides upper- bound of improvement using Reflexion.
achieves 90.6% accuracy. Again, the low accuracy with QWen-VL-chat confirms our modular approach design choice.
4.2 Autonomous Refinement
We demonstrate how our proposed evaluation models can serve as a reward signal guide an existing web agent at inference time, using the Reflexion technique (Shinn et al.) 2023) as example. In Reflexion, and an external evaluator is used to judge whether its attempt was successful or not. is judged as unsuccessful, the agent will be prompted to reflect the failure and retry. We experiment with improving the current state-of-the-art GPT-4-based WebArena agent [1]]
Reflexion on WebArena
agent first attempts task,
Figure includes the agent's baseline performance, and performance using up to three rounds of Reflexion with the oracle evaluator (which serves as upper bound) and our three evaluation systems as external supervision. We see the improvement our evaluators provide scales favorable with evaluator capability, with Captioner Mixtral improves agent's relative success rate by 16% and GPT-4V based evaluator by 29%. All system variants, including the low-cost and locally-hosted variant Captioner Mixtral, significantly enhance agent's performance while requiring no access to hand-designed evaluation functions.
Our preliminary study suggests that false negative evaluations have more detrimental impact on agent's performance compared to false positives. If our evaluator predicts an execution is incorrect, but was actually successful, this forces the agent to retry a successful execution, which nearly always leads subsequent failure. In contrast, false positives only lose out on the opportunity to retry, which creates an upper bound of performance for the agent, but does not degrade its performance. Improving the robustness of inference-time algorithms under noisy supervision is an interesting future direction to explore.
Filtered Behavior Cloning for Device Control We demonstrate how our evaluator can guide the refinement of policy using filtered behavior cloning (filtered BC), without additional supervision. We experiment on both iOS and Android environments.
On iOS, we use CogAgent (?) as the policy model for the experiment. It is primarily instruction-tuned with demonstrations from web navigation and Android device control, and incorporates very limited, manually collected iOS dataset for training. For data collection and testing purposes, we design 132 common tasks on iOS, with 80 tasks for training and 52 for testing. Given scaling limitations of emulation, including low speeds
11 Reflexion prompts are detailed in Appendix A.5
8

---

Published as a conference paper at COLM 2024
and restriction to emulation macOS, we only experiment with iOS built-in apps and with the Captioner + Mixtral evaluator.
We first sample 737 trajectories from CogAgent, conditioned the 80 training tasks. We use our evaluator to provide per-step evaluations to these trajectories, then apply filtered BC for fine-tuning using this data. Unlike standard fine-tuning, this method filters out data points with rewards below a specified threshold. We set this threshold at 2 pł iee we retain only state-action pairs that positively influence the success of a trajectory (Section 3 Additionally, we assess CogAgent's unmodified performance on iOS and explore a self-training approach by finetuning without data filtering as baselines for comparison.
Table 2 contains results for the 52 test tasks. iOS device control is a challenging task, with the baseline agent completing only 8 out of 52 tasks, yielding 15% success rate. Self-training improves over the baseline by 3 tasks. Filtered BC with our evaluator significantly improves the policy model's performance from to 14 successes, marking a 75% relative improvement. We also conduct a preliminary analysis of our per-step evaluator's 's accuracy. Human annotators agree with our step-wise evaluator on 43 of 50 state-action pairs randomly sampled from the iOS experiments.
However, our iOS experiments require physical Apple computers for emulation, which are slow and not scalable, leading to natural limitations in scope of our evaluation. To address this, we design a similar but larger experiment on Android. Unlike iOS, Android allows us to use scalable emulators on clusters, enabling efficient repetition of experiments. We use AutoUI-base policy for the experiment, selecting 560 tasks for training and 96 for evaluation. The base policy (AutoUl-base) completes 15 tasks, improving to 18.9 1 1.0 (averaged over training runs) with self-training, Filtered BC with our step-wise evaluator still significantly improves, succeeding 26 + 0.8 tasks,r esulting in 73% relative improvement.
Policy
# Success (Total: 52)
Policy
Success (Total: 96)
CogAgent
8
Auto-UI-base
15
+ Self-training
11
Self-training
18.9 + 1.0
+ Filtered BC (Ours)
14
Filtered BC (Ours)
26.0 + 0.8
(a) iOS Experiment
(b) Android Experiment
Table Comparison of base policy and refined policies via self- self-training and filtered behavior cloning, including the number of successful tasks in our test set.
4.3 Error Analysis
We randomly sample 20 successful and 30 erroneous evaluations for each evaluation model in WebArena and manually annotate the sources of failure 12 We categorize errors into three primary types, providing percentage estimates rounded to the nearest 5%.
1. Critical information lost from captions in the modular approach (10%); errors in screenshot understanding for the end-to-end GPT-4V approach (5%).
2. Errors in the reasoning process, observed in 50% of cases for GPT-4V/GPT-4basel methods and 70% for Mixtral-Capptioer.
3. Ambiguities in task specification and success criteria, observed in 30% of cases for GPT--V BAADV//GTT----aa
methods and 10% for Mixtral-Captioner.
We note that in our error categorization, a model must overcome errors in preceding categories to be assessed under the subsequent one. Consequently, Mixtral-Captioner lower rate of Type 3 errors is mostly attributed to its higher frequency of Type 1 and 2 errors.
Additionally, we find the model provides the correct final evaluation, but incorrect reasoning, in about 10% of correct evaluations.
12 Figures through in the Appendix for visual representations of these evaluations.

---

Published as a conference paper at COLM 2024
5 Conclusion
In this study, we design automatic methods to both evaluate and refine the performance of digital agents. We first describe model that provides either trajectory-level or per-step evaluation of agent's performance. Subsequently, we propose two approaches to implement the model: an end-to-end approach using pre-trained vision-language model, and modular aoption-then-reaseee approach using VLM and pre-trained language model together. These methods offer trade-offs between performance, cost, and modularity.
Using WebArena and serridd-i--the-Wiee as testbeds, we first validate the effectiveness of these evaluators against oracle evaluation metrics, and highlight their advantage over standard reference-based metrics AitW. We then show how the evaluators can be used to refine existing agents through both inference-ime guidance and filtered BC. When integrated as the reward function in Reflexion, method for inference-time refinement, our evaluators enhance the best-perfore rest-performing agent's success rate by up to 29%. Additionally, it boosts the performance of strong device control policy in domain transfer task by 75% via filtered behavior cloning, all without any extra supervision. Our findings show the potential of model-based automated evaluators for both evaluating and improving digital agents, which is especially critical in developing real-world agents where ground truth evaluation functions or human supervision are not always available.
Limitations and Future Work
While our research demonstrates the potential of model-based evaluators evaluating and improving digital agents, also identify several areas for future exploration. First, current evaluators are still far from perfect, and any enhancement in their performance, e-g, from better representations of the action space stronger base models, will likely directly translate to improved outcomes. Second, in this work, we focused on Reflexion and filtered behavior cloning. Future works can explore scaling up the experiments and developing better training and inference-ting algorithms that are robust and efficient under noisy supervision. Finally, in this work we only make use of the evaluator's 's binary or ternary judgment, and discard the language-based explanation it generates. Future work can explore how leverage this information, for example, to further enhance policies through language supervision or to provide scalable oversight of agent behavior.
Ethics Statement
Most currently available digital agents are research artifacts. As the performance of these agents improve and they are increasingly deployed in the real world, they may pose security risks to their users. For example, web agent with unconstrained access browser might be able to gain access a user's passwords, financial information, or social media messages. Better understanding the potential failure modes of these models in real-world use cases is critical to ensuring their safe deployment. We view our work as a first step in this direction: by developing domain-general evaluators, we hope to facilitate better understanding of models (and their risks) outside of simulated environments like WebArena. At the same time, human evaluation and oversight of these future systems will also be important for mitigating potential harms; although our work in this paper focuses autonomous evaluation, we hope it will supplement, rather than supplant, human efforts.
Acknowledgments
We thank the Berkeley NLP group, especially Ruiqi Zhong, Andre He, Charlie Snell, Cather- ine Chen, Sanjay Subramanian, and Zineng Tang, well as Allen Nie for feedback and discussions and Shuyan Zhou for assistance in setting up the WebArena experiments. This work was partially supported by A12 Young Investigator Grant. NT is supported by the DARPA SemaFor program.
10

---

Published as a conference paper at COLM 2024
References
Marwa Abdulhai, Isadora White, Charles Burton Snell, Charles Sun, Joey Hong, Yuex- iang Zhai, Kelvin Xu, and Sergey Levine. LMRL Gym: Benchmarks for multi-turn reinforcement learning with language models. ArXiv, abs/2311.18222, 2023. URL
attps://apl..smaatiiccs www.smmmmnncholllaat wtt.cccoooooooooogorooooossoo wwt.ooooogs...ooo..... 2655066111
OpenAl: Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapire Christo- pher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloun- dou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Adeola Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M'ely, Ashvin Nair, Reichhir Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O'Keefe, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perel- man, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin D. Sokolowsky, Yang Song, Natalie Stau- dacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer' on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 technical report. 2023. URL h/tps: semanticscholar hww.hoooll...oo/ooooooooooo..ooe 2575335281
Johannes
11

---

Published as a conference paper at COLM 2024
James Allen, Nathanael Chambers, George Ferguson, Lucian Galescu, Hyuckchul Jung, Mary Swift, and William Taysom. PLOW: a collaborative task learning agent. In AAAI, 2007.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen- VL: A frontier large vision-language model with versatile abilities. arxiv, abs/2308. 12966, 2023. |Rthttpss://api wewansicscooooaaaa............ org/CCrpusII 263875678
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jarrie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhagg, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI feedback. arXiv, ///.//200....666 2022.
S.R.K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay, Reinforcement learnin for mapping instructions to actions. In ACL-AFNLP, 2009, URL https///c//allllthoooooyy 001/P09-1010
S.R.K. Branavan, Luke Zettlemoyer, and Regina Barzilay. Reading between the lines: Learning to map high-level instructions to commands. In ACL, 2010. URL https:// aclanthology.org 1ttph/oooooy.og///-111111 org/P10-1129
Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, and Bryan A. Plummer. A dataset for interactive vision-language navigation with unknown command feasibility. In ECCV, 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurlPS, 2021. URL http:tts://opeerrreeeeeeeetteoee wwwp:eewwwwiiieteffruuuro
id=a7APmM4B9d
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. BAII: Best-action imitation learning for batch deep reinforcement learning. In NeurlPS, 2020. URL http::////pooeednnnneeerippppppp 4556FF700115444410eeeeee66000200
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards generalist agent for the web. In NeurIPS Datasets and Benchmarks Track, 2023. URL URL.httts::/oopnreviie www..eeeoooruiiccciiooooooo
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, and Serkan Cabi. Vision-language models as success detectors. arviv, 2023.
Brad Dwyer. Website screenshots dataset, 2020. URL https://public..roffl..... seaectdddeectttoo//wssst-eeessee screenshots
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. RvS: What is essential for offline RL via supervised learning? In ICLR, 2022. URL https://opeerreview het/fformm/id-5774AAIpp.
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In CVPR, 2023.
12

---

Published as a conference paper at COLM 2024
Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. real-world webagent with planning, long context under- standing, and program id=9J@trunvga
programsyythhesir synthesis. In ICLR, 2024. URL https:://openrreiieeeeeett//
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. ArXiv, abs/2401..1990 139999 2024. URL /tthttts:://ai semanti www.tttiiccchcc.cccoccmcomoca..t wttphholaarrrroooouuuoouueese
267211622
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge to local nash equilibrium. NeurIPS, 30, 2017.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. URL
Peter C. Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timo- thy P. Lillicrap. A data-driven approach for learning to control computers. In ICML, 2022. URL /ttpss//s// semaatticsccolar..ogg/rrrrr
246867455
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. ArXio, abs/2400.00gg 24401.0408,, 2024. URL
wwtp:/opo//oooroo.... Stt.aaatiicccc.......orrrrrroopa 266844877
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/ abs/1442.6980, 2014. URL https:://api.semattiie
wttp://rrr..g//oooous....o. 6628106
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. ACL, 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF: Scaling reinforcement learning from human feedback with ai feedback. ar arXiv, abs/2309.0026. 2023.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile UI action sequences. In ACL, 2020. URL https:///cclaathoologg
org/2020.ac11 acl-main.. 729
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, and Percy Liang. Reinforcement learning on web interfaces using workflow-guides exploration. In ICLR, 2018. URL https
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022. URL https s/api.ssmanticschol.. wtwp:/aar..rr/ooousoiiisi/ 246426909
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P Lillicrap. AndroidInTheWild: A large-scale dataset for android device control. In NewrIPS Datasets and Benchmarks Track, 2023. URL
13

---

Published as a conference paper at COLM 2024
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of Bits: An open-domain platform for web-based agents. In ICML, 2017. URL https //proceedings.. mlr. www.pp/sss/ee/eee otw.:eeddinns.mmm..rrres//////// html
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In NeurlPS, 2023. URL httt:s::/aap..emmatiiciiiii
Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Za- farali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precupp AndroidEnv: A re- inforcement learning platform for android. arXiv, abs 13221, 2021. URL tts
Bryan Wang, Gang Li, and Yang Li. Enabling conversationnal interaation with mobile ui using 1145/3544548, 3580895 3558888
large language models. In CHI, 2023a.
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent: Autonomous multi-modal mobile device agent with visual perception. Xrv, abs/ 2401.16181 2024.
Ziyue Wang, Chi Chen, Peng Li, and Yang Liu. Filling the image information gap for VQA: Prompting large language models to proactively ask questions. 2023b. URL https://aclantthology wrg./y...../........ wwt.:/....ninns.....e... fiidinggnnnnnnl.. 189
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurlPS, 2022.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. OS-Copilot: Towards generalist computer agents with self-improvement. ar Are, 2402.0075.. 2024.
Nancy Xu, Sam Masling, Michael Du, Giovanni Campagna, Larry Heck, James Landay, and Monica Lam. Grounding open-domain instructions to automate web support tasks. In NAACL-HLT, 2021. URL /tpp: orgp://g...gg//1//1// www.:r///.......la........
An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Qinghong Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. GPT-4V in Wonderland: Large multimodal models for zero-shot smartphone GUT navigation. arXiv, abss2331.077 2023. URL httpp://apt semanticscholar org/ CorpusID:2651499900
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: To- wards scalable real-world web interaction with grounded language agents. In NeurlPS, 2022. URL httph/ttss//ppcccingss.eee
wwwp:nneerrssccpppeeeee files/paper//002//ille// 0e2.ac/ce.....44444..23434430000
pdd]
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R. Narasimhan. Tree of thoughts: Deliberate problem solving with large language
models. In NeurIPS, 2023. URL
Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad Ayyubi, Kai-Wei Chang, and Shih-Fu Chang. IdealGPT: Iteratively decomposing vision and language reasoning via large language models. In Findings of EMNLP, 2023. URL mrndingg-emnlp..
httts:://aclanthollgg org/2023
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. UFO: Ul-focused agent for Windows OS interaction. arxiv, abs/ htss/4402.7933) 2024.
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. AppAgent: Multimodal agents as smartphone users. arxiv, abs/2212.13771, 2023.
14

---

Published as a conference paper at COLM 2024
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv, abs/ 2309.12309.1436, 2023. URL https://app www.nticsccooarr.orgggggo//////e 262053313
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. NeuriPS, 36, 2024.
Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In ICLR, 2024. URL
15

---

Published as a conference paper at COLM 2024
A
Experiment Details
In this section, we provide details about our experiments. Please refer to our code at for the official reference.
A.1 End-to-End Approach
We use gpt-4-1106-vision--ees resizing in "high-resolution". We use temperature of 0 and keep other parameters at default. The prompt templates for each environment are provided in Figures 12 and 13
through the OpenAl API and feed the image without
A.2 Modular Caption-then-Reason Approach
we constructed our dataset primarily through random subsampling from source datasets. However, for the iOS domain, due to limited online resources, manually capture 50 extra screenshots in-house.
Collecting screenshots As described in Table 3
Action representation We represent actions as strings, e.g., Type 'Hello'' This method leads to information loss when processing actions like clicks for pixel-based policies, as the coordinates [x, y] become meaningless when the image is represented by its textual description. We leave the task of more adequately transforming pixel-localized actions into textual forms for future work.
Collecting screenshot descriptions After obtaining the screenshots, we query GPT-4V (specifically,
through the API) to get dense caption demonstra- tions. We manually fix or filter out ones with apparent errors. We use temperature of 0 and keep the other parameters their default settings. The prompt template is provided in Figure 15
Finetuning Qwen-Vi VL captioner The prompt template to query the finetuned Qwen-VL- chat captioner is provided in T6 We fine-tuned the model over 3 epochs with a batch size of 72 and adamw optimizer (Kingma & Ba 2014), employing cosine scheduler for learning rate adjustments starting from le-5, a weight decay of 0.1, and warmup ratio of 0.01. As shown in the prompt template, during both finetuning and at inference time, we provide text recognition results from the EasyOCR engine as an additional input to the model to reduce hallucination. We provide randomly-sampled model output examples in Figure
Quering the reasoner After obtaining the descriptions of the screenshots and actions, we query the LM, either
1 or gpt-4-turbo-pres -previee..... We use a temperature of and keep the other parameters at their default settings. We provide the prompts to query the trajectory-level evaluator on Web and Android, and step-wise evaluator on iOS Figures 17 18 and respectively.
A.3 Evaluation on WebArena
We directly use the GPT4-0613 CoT v2 trajectories released by WebArena for evaluation. Confusion matrices for our evaluators' predictions compared to the oracle evaluator are shown in Figure 4 (left).
A.4 Evaluation on Android
Emulator We Android Studio's built-in emulator to simulate Pixel 4 with API version 33, and we develop a Python API for agent execution based on the appium package. We opt not use AndroidEnv for Android emulation it lacks support for typing actions.
16

---

Published as a conference paper at COLM 2024
Tasks The 120 evaluation tasks are evenly and randomly sampled from the General, WebShopping, and GoogleApps subsets of the Android-in-the- -iild test set (40 each) as shown in Listing Note that we have excluded the Install and Single subsets. The install tasks require credit card information and are not safe to evaluate, while single-step tasks fall outside our focus on trajectory-level tasks.
Evaluation We use temperature of O, for all policies during evaluation. The confusion matrices comparing our evaluators with human judgments are presented in Figure 4 (right).
A.5 Refinement on WebArena
Reflexion We implement the Reflexion agent following the original paper (Shinn et al., 2023). The algorithm involves three key components; an Actor, an Evaluator, and a Self- Reflection module. The Actor generates thoughts and actions in text based on the state observations it receives, where the actions are parsed into executable commands to transform the environment. The Evaluator assesses the quality of the outcome produced by the Actor. It computes a reward score based on how well the generated trajectory align with the goal. the evaluator assesses the task to be failed, the Self-Reflection model generate verbal reflections, which is stored in the actor's 's memory, facilitating improved decision-making in subsequent trials.
model will be evoked to
Impelementation Details We use the DOM tree representation from the WebArena simu- lator as the environment observation. The LLM we use for Actor and the Refleceflntt is the GPT-4-preview-1 1106 model, and the prompts for these are shown in Listing and respectively. For the evaluator, we experiment with all the three variants proposed as well as the oracle evaluator from WebArena which is used for performance evaluation. Note that we use the webpage snapshot images instead of the DOM tree as the input to our evaluator.
A.6 Refinement on iOS
Emulator We use XCode's built-in emulator to emulate an iPhone 13 device running iOS 16 and develop a Python API for the agent based on the faeeffcceoo//ii /idb packok We align its action space with that of the Android-in-the-Wiin schema. Notably, since swiping up on the homescreen in Android means opening the AppDrawer, which can roughly be translated to swiping left or right on the homescreen, we bridge this domain gap by translating swiping up to swiping left or right 50% of the time during data collection and to the right 100% of the time during evaluation.
Tasks As shown in Listing 2 we design 132 task instructions that covers typical iOS device control tasks Apple's official apps, where 80 are used for data collection and training the agent, and 52 are hold out for testing. While we aimed to minimize the distribution difference between our tasks and that from the AitW dataset, the inherent differences between the platforms necessitated the inclusion of iOS-specific instructions, such as "Disable Siri's access Photos in Settings."
Filtered BC Details We finetune the CogAgent model with its official code and apply LoRA (Hu et al.) 2022), setting the parameters as follows: lora rank to 50, employing a cosine learning rate schedule, with warmup proportion of 0.2, learning rate (Ir) of 0.00001, batch size of 4 and train for 3000 steps. We use a releatively high temparature of 1.5 and topk=100 during data collection to improve diversity and greedy decoding (temperature=0) for all policies during evaluation.
A.7 Refinement on Android
We reuse the Android emulator as described in Appendix A.4 and sample the training and evaluation tasks from AitW General split's corresponding part. We use AutoUI-base the policy. We set the temperate for during sampling and uses a learning rate of 3e - 3 and a batch size of 128 for the training.
17

---

Published as a conference paper at COLM 2024
Source
#
Domain
WebScreenshot (wwyer 2020
128
Web
Mind2Web (Deng al. 2023
429
Web
AitW (train set] [Rawles les et al. 2023
596
Android
GPT-4V in Wonderland Yan et al.
2023
60
iOS
In-house
50
iOS
Total
1,263
Table 3: Sources of the screenshots dataset, including the number of screenshots paired with detailed descriptions and the original domain of the screenshots.
Captioner GPF4
Captioner Mixtral
Figure 4: Confusion matrices for different evaluation systems against oracle evaluator human judge on WebArena and Android.
O
Star the top one most stared repos Gitlab.
click
[The A11Y Project/allyyrojject.omm
go back
click
[The Projeett/allyproeect.co
stop
Response: The AuY Project/anxypnjeee.comms
already starred.
Thoughts: The user's intent wus to star the top one most sturred repo in GitLab, but the attempted star a specific repository that was already starred. There is no indication that repository question perform any action ensure it interacting with the correct repository based the user's 's intent. Therefore, the task is not completed successfully.
top one most starred, and bot did not
FAILURE
Figure 5; The evaluator detects a hard-to-fine issue caused by a key missing step in the execution (i.e. ordering).
18

---

Published as a conference paper at COLM 2024
O
Post "Igim" for the merge request related to semantic HTML post autypoojcc project.
click [The AllY Project/alxyprojett.con
click [Merge requests']
click [Add draft of what Semantic HTML post]
scroll [down]
type [1gtn] [Comment]
click at [Comment]
type [1gtn] at [Comment]
click at [Comment]
type [1gtm] at [Comment]
stop
Response: Early stop: Same typing action for 3 times.
Q
The image shows that the comment "Igtm" has been posted twice by the "Byte Blaze," which indicates that the bot successfilly posted the comment intended. However, the bot stopped early due to the same typing action being repeated three times, which is likely sufeguard against spamming. Since the user's intent wus to post "lytm" and it is vistble in the merge request comments, the task is successful despite the early stop.
Thoughts:
FAILURE
Figure 6: The evaluator correctly judge a task execution successful despite the task ends with an exception.
O type
Tell me who has made most contributions (commits) to the Pytorch GAN project.
[Pytorch GAN]
[Search GitLab]
click
[Erik Linder-Norén PyTorch-GAN]
click [193 Commits]
eTToni
stop
Response: Cannot determine.
Q
The image shows the commits page of the PyToorch-GAN project on GitLab, but does not provide summary or a list that ranks contributors by the number of commits. The stopped with "Cannot determine" response, which indicates it could not find specific information requested by the 'Contributors' section to find the information about who has made the most contributions. Therefore, the task is not completed successfully.
The bot should have navigated the
FAILURE
Figure 7: The evaluator can also provide insight for possible corrective actions of failure executions.
19

---

Published as a conference paper at COLM 2024
O
Check the walmart in pittsburgh can reached in one hour by car from 5600 5600 avenue.
goto
hhttps://naps.. gongle..oooooo
type
'5686 Fifth Avenue, Pittsburgh, PA
[Search Google Mарs]
click
(Search nearby 5680 Fifth Ave]
type
[Nelnart] [Search nearby Fifth Ave]
click [Search]
[Get directions to walmart Supercenter]
type
(5680 Fifth Avenue, Pittsburgh, PA]
[Choose starting point]
stop
Response: Yes, the Walmart Pirtsburgh he reached minutes by from 560u Avenue, which
bour
Thoughts: The action history indicates that the agent performed seurch for directions from "5600 Fifth Avenue, Pittsburgh, PA" a Walmart Pittsburgh. However, the final state of the webpage shows that the estimated travel time by car actually hour that stated 16-mirute
43 minutes", which contradicts the bot's response the travel time. This discrepancy suggests that the bot's response is incorrect, and the task of verifying the Walmart can be reached within one hour by car has failed.
FAILURE
Figure 8: Captioning error where driving time is not captured, and the public transportation time is incorrectly captioned as driving time. The error is corrected if GPT4V is used.
O
Tell me the total cost of my latest cancelled order?
click [My Account]
click [My Orders]
stop
Response: $231.54-
My Crders
The user's intent to know the total cost of their latest cuncelled order. From the action history, us see that bot successfully navigated to "My Account" and "My Orders" pages. The final state the webpage shous list of orders with details. The latest cancelled order has price of $231.54, which matches the bot's response. Therefore, seems that the bot has successfully completed the task.
SUCCESS
Figure 9: Mixtral failed to identify that the price not from cancelled order from cluttered webpage. The error is corrected if GPT4/GPT4V is used.
20

---

Published as a conference paper at COLM 2024
0
Help me Fork ChatGPT.
type [ChatGPT]
at [Search GitLab]
click [Convex Eggtart Chatgpt]
click [Fork]
click [Fork project]
stop
Response: Forking ChatGPT is in progress.
Thoughts: The action history indicates that the agent has performed the necessary steps to initiate forking of the ChatGPT repository on GitLab. The final state of the webpage shows the agent is on "Fork project" form, which consistent forking process. The bot's response
the confirms that the forking process has been initiated. There is no indication of an error incomplete action, so assume the task was successful unless there is evidence to the contrary.
SUCCESS
Figure 10: All the evaluation models fail in identifying the failure because they miss to the error message which indicates the failure of forking.
O
How much spent on food-reltted shopping during March 2023?
1: click [My Account]
2: click [My Orders]
3: click [View Order]
4:
click [My Orders]
5: click [View Order]
Urders
6: click [My Orders]
stop
Response: $83.31.
Q
Thoughts: The bot's response only includes single order total, but the user's intent was know the total amount spent on food-related shopping for entire month of March 2023. The image shows multiple orders March, and the bot should have summed all completed and not canceled orders to provide accurate total. Therefore, the task has foiled.
X
FAILURE
Figure 11: Though our approach successfully identifies that the task has failed, the reasoning is incorrect. The error is caused by not filtering the orders for food-related only, instead of not summing up the total price.
21

---

Published as a conference paper at COLM 2024
Systen Prompt
navigation agent. agent designed help user navigate a website to complete task. Given the user's intent, the agent's action history, the final state webpage, and agent's response the user, your geal is
You expert evaluating performance
whether agent execution
successful or not,
There are three types of tasks: 1. Information senking: The user wants obtain certain information from the webpage, such the information of product, reviews, info, comparison of nар must contain the information the user wants, or explicitly state that the information is not available. Otherwise. e.g-
The bot's response
tonsidered failure. Besides, be careful about the sufficieno of the agent's actions. For example, when asked to list the top-searched itens shop, agent should order ordering action is nissing, the task likely fail. 2. Site navigation: The user wants to navigate to specific page. Carefully examine the bot's action history and the final state of the webpage to deternine whether the bot successfully completes the modify the content of webpage or configuration Carefully exanine the bot's action history and the final state of webpage
encounters exception
respond
error content,
itens by the number searches, and then return the top itens.
need consider the
response. Content nodification:
successfully completes task. *INPORTANT Fornat your response into two lines shown below:
response.
Thoughts: <your thoughts and reasoning process> Status: "success" or "failure"
User Proupt
User Intent: intent
History: {last actions]
The last snapshot of the web page is shown in the inage. response user: (response response else "N/A").
Figure 12: Prompts to query the VLM Evaluator in WebArena.
Systen Prompt
You expert evaluating the performance huran user navigate the device to complete a task. Given the user's intent, and the final state of the screen, your goal decide
android navigation agent. agent designed help
agent successfully completed
not.
*IMPORTANTA Format your response into two lines shown below:
Thoughts: <your thoughts and reasoning process?* Status: "success" or "failure"
User Proupt
User Intent: {intant)
Action History:
actionnctions)
The last snapshot of the screen is shown in the inage.
Bot response to the user: (response if response else "N/A").
Figure 13: Prompts to query the VLM Evaluator in Android
22

---

Published as a conference paper at COLM 2024
M
Dashboard
-2079
Liferine
AvenogoOdder
(a) WebArena Example
#12.
Agent-Eval- Eval-Refine
Organization Card
bilt Mirropor
yet
Refinement of Digital Agentss
project
Digita Atenttr
And Welk Nicelle Tanils Tlass Sermecladda
Poceed the thas
Project Page
Pape: 10D0
(b) Random Real Website
Contacts
KB
Kate Bell
Prokkeer
Entracs Wenpoa biilies 2004
1990 2053
Suggetad you
Horten cel lecullmod
Fime Shee de Ailingre
25
(c) iOS Example
(d) Android Example
Figure 14: Example outputs from the captioner model across different environments. Exam- ples are selected at random without cherry-picking
23

---

Published as a conference paper at COLM 2024
User Prompt are edvanced captioner. describe response should be hierarchical and in Markdown Fornat. Don't it do paraphrase. Don't wrap your response in code
in
anything.
Figure 15: Prompts to query GPT-4V for collecting dense captions
User Prompt
Please describe the screenshot above in details.
Result:
focr_result]
Figure 16: Prompts to query the fine-tuned captioner for dense captioning.
Systen Proupt
You are an expert in evaluating the performance of a web navigation agent. The agent is designed to help a huran navigate website corplete Given of the webpage, and the agent's response to the user, your geal is to decide whether the agent's execution is successful or not.
intent, the agent's history, the final
There are three types of tasks:
Information sneking: The wants obtain certain information from the wrbpage, such the information of product, reviews, nap info, comparison of maр routes, etc. The bot's response must contain the information the user wants, explicitly state and respond with the error content, the task is considered a failure. Besides, be careful about the sufficiency top-searched
information
available. Otherwise, e.g.
encounters exception
agent's actions. the itens by the number of searches, and then return the top itens. If the ordering action is missing, the task is likely to fail.
example, when
shop, agent
navigate specific page. Carefully examine bot's the final state of the webpage to determine whether the bot successfully completes the task. No need to consider respanse.
Site navigation: The
history and
3. Content modification: The user wants to modify the content of webpage configuration. Carefully examine the whether successfully completes
bot's action history
webpage task. No need consider the bot's response.
IIMPORTANT*
Fornat your response into two lines as shown below:
Yyour thoughts and reasoning process>" Status: "success" or "failure"
Thoughts:
User Proupt
Intent: {intent]
History:
[last.actions]
The detailed final state of the webpage:
<>aad
(sap)
Figure 17: Prompts to query the LM reasoner in WebArena
24

---

Published as a conference paper at COLM 2024
Systen Prompt
You expert evaluating performance android navigation egent. agent designed help hunan user navigate the device to complete task. Given the user's intent, and the state of the screen, goal whether agent successfully completed
or not.
+IMPORTANT* Fornat your response into two lines shown below:
Thoughts: <your thoughts and reasoning process?" Status: "success" or "failure"
User
User Intent: {intent]
Action Histary:
(last,actions)
The detailed final state of the screen: "nd
[cap]
Figure 18: Prompts to query the LM Reasoner for trajectory-level evaluation in Android
25

---

Published as a conference paper at COLM 2024
Systen Prompt
You Trajectory Evaluator-
within graphical (GUI) classify its behavior into ane of four categories based on its progress towards specified goal. The categories are:
observe
"towards-the-geal" - The bot moving closer to achieving the goal. "not-sure" - It's unclear if the bot's actions are helping reach the goal. "goal-reached" The
successfully completed the geal. 4. soay-ffrr-tteggall The bot's actions are diverting it from the goal.
Please fornst response as follows:
Thoughts: [Explain reasoning here]
Response: "towards-the-gaa" "not-sure", "geal-reached", or "away-fran-tha-pee
Here are some example responses:
Exarple
Thoughts: The goal is to 'set an alarm at 8:00 an." Initially, the bat is on the hone screen. After a top action, navigates alarm apo, indicating progress towards goal.
Response: "Lowards-the thee-llat
Example
Thoughts:
The goal is to "buy the latest iPhone on Arazon. bot starts at the checkout page on Anazon, After tap action, the screen shows successful purchase, signifying that the goal has been reached.
Response: "goal-reached"
Example 3:
Thoughts: goal to ne
York." hot begins London's weather page. pressing 'home", it returns the hone screen, moving away from the goal.
Response: "eney-fron-the
Example
Thoughts: goal 'buy
Starbucks app. begins Anazon app. pressing 'back," it noves to the home screen, which is prerequisite for opening the Starbucks opp.
Response: "towards-the therdd-thheggal?
Exarple
Thoughts:
The is "open YouTube." The bot begins the hone After swipe, it appears renain the same page, suggesting no progress towards the goal.
Response: "nat-sure"
Note:
You should be extra-careful when assigning "goal-reaches" or "towards-the the--lll""""" labels. If unsure, please select "not-sure" instead.
Proupt
Goal: [intent)
Original State:
.-"nd
(surrent suureett.state state)
State after action: "faction)";
condd
(next_state)
Figure 19: Prompts query the LM Reasoner in iOS for per-step evaluation.
Listing 1: Evaluation tasks sampled from
test set
--- General
Open
Play the new Katy Perry video on YouTube
nenu at Papa Murphy's? s?
How nuch does bedroon apartnent rent for in Philadelphie?
What's the price of the Galaxy phone on eBay?
in How do I get to the nearest Target? app
Search for flights from NYC to Mexico city
what is the capital of China? What's dellar exchange rate against the South Korean Won? What is the speed of train?
capital of Norway?
What's 's the news in Peru?
do get
26

---

Published as a conference paper at COLM 2024
What's good restaurant Miani?
How much does a 3 bedroon apartnent rent for in Houston?
What's the weather like in thicago?
Set an alarn for 6pm
What is the capital of Mexico?
What's the What's is a good restaurant in Chicago?
Canbodia?
capital of Japan?
Search for flights from Barcelona to Mexico city What's 's What's the news India?
menu at Olive Garden?
What is the cupi of Canadia?
Search for
nail polish
Open new throne tab What's
latest news planetary science?
How nuch does 2 bedroon apartnent rent for in Boston?
Search for hotels in Las Vegas
Where can
nice beach towel?
What's s the price of the LG TV?
Search for
iigheighter
What's the latest video fron GameXplain? Search for
sushi restaurants Maps
's the weather like in Johannesburg?
Search for hotels in London
Install the Google Heb Shopping --- Search for
65"
Best Buy
Add "sony triple a" to the cart on amazon Search for "nacbook pro Look the best ganing headphones on Best Add bose quietconfort Add nacbook
walnart.com select first entry, add
to the cart on costco.con
the Add "dell xps" to the cart on bestbuy, then select checkaut. Clear the shopping
newegs, con. Search for "razer naga" newegg.com con, select the first entry, and < select checkout.
+ to the cart.
entry, add the cart, then Search for "rayovac triple s" on walnart, select the first entry, and add it to the cart, Clear the cart newegg. Search for duracell triple newegg, select the first entry, add to the cart,
View shopping cart ebay. Search "usb-a" on ebay,
mi then select checkout.
Add bose quletconfort Search for jbl flip on abay.com select the first entry, add it to the cart, then select checkout. on newegg. con, select
the
anazon. con
cart newegg. con. Add Clear all itens from cart costco. Search for bose quietconfort 35 costco, select the first entry, and add
- it to the cart.
Snarch for the best books of all time Goodreads
Clear the cart on con. Search for "logitech g502* on con,et.com, select the first entry, and add it to the
Clear the cart costco. con. Search for usb-c to usb-a an costco. con, select the first entry, and add it to the
- cart.
Search for duracell triple newegg.con, select the first entry, and add it the cart.
Show the shopping cart on anazon. Add dell alierware to the cart on amazon, then select checkout. the cart.
Search for
select the first entry, and Show the shopping cart on ebay. con. Search for "razer blackwidow" on ebay.com, select the first entry, add it to select
predator ebay
the cart, Add razer nari to the cart on target, then select checkout.
bose soundlink the cart anazon.com
Empty the shopping cart newegs.com Add charge 4 to the cart newegg.com, con, then select checkout.
Clear the shopping cart on costco. Clear all itens from
bestbuy.com con. Search for usb-a bestbuy, con, select first entry, and add it anazon.com, con, the first entry, add it
< the cart.
the shopping cart anazon.com. con. Search for razer - the cart, then select checkout.
Add nacbook Search for "corsair k78" target.con, select the first entry, add it the cart, then select checkout. View the shopping cart on bestbuy. Search for "lagitech g pro" on besthuy, select the first entry, add it to the cart, then select checkout.
cart on target
View the shopping cart on bestbuy.com Search for panasonie triple on bestbuy. select the first entry, add the cart, then select checkout.
Clear the shopping cart on target.con. Add nacbook air to the cart on target.con, then select checkout. Search for razer blade bestbuy.ce con, select the first entry, and add it to the cart.
Clear the cart target. Add "usb-c to usb-a" the cart on target
Search for the best books on Goodreads
Search for
printer allexpress
Search for duracell triple on costco.com select the first entry, add it to the cart, then select checkaut. on besthuy.coo ccm
ig tragerr
Clear all itens fron cart on walmart.con con. Add macbook pro to the cart con, then select checkout.
Google Apps
have events tonorrow?
toggle data saver in the chrone app off' airplane node
27

---

Published as a conference paper at COLM 2024
Open calendar show first
Go to location settings create a new albun in the google photos
Open calendar and show the fourth week of next month What's on my calender today?
check android version
add label to a message in the gnail app
Open calendar turn on the 24-hour format for clock
ne
empty trash in google photos
turn showing notifications the lock screen turn on bluetooth scan
toggle notification dots set the tiner
Open Wikipedia
open chrone and create booknark for the current page change the upload size google photos Show ne popular videos Youtube to settings
change the clock display analog
Search for Italian restaurants on Maps poprups turn vacation reply on the gnail turn on wifi
Search for pizza restaurants Maps Check the weather
toggle sleep mode
turn notification dots off trenslation
app
all nails in gnail
to privacy settings Is it going rain today?
toggle wifi
golng
this weekend?
Open calendar and show ne the second week of next month creations saved
google photos
Open internet settings
Listing 2: Data collection and evaluation tasks for iOS experiments
Train Tasks
Open the Reminder App
the Calendar
Open the Map App
Open Safari
Open
App
Open Messages
Open Healthh App
Files
Open Shortcuts App
App
Open Watch App
Open General Page in Settings App
Find the About Page in Settings App
Find the ios version of this device in Settings
Find the Serial Number this device Settings Disable Auto-Correction for Keyboard in Settings Disable Disable Slide to Type for Keyboard in Settings Disable Caps Change the Temperature Degree Celsius in Settings/General Change Safari Search Engine to Yahoo in Settings
Punctuation Keyboard Settings
Keyboard Settings
Change Safari Search Engine
Settings
Change Safari Search Engine to Baidu in Settings
Change
Engine Sagau Settings
Change the preferred language of Safari to thinese in Settings
Change the preferred language Safari Japanese in Settings Change the preferred language of Safari Russian in Sectings Change the preferred language of Safari to Thai in Settings Change preferred type of
of
Driving Settings
Change the preferred type of travel of Maps to Transit in Settings Change preferred type travel Maрs Cycling Settings Disable location access of Mарs in Settings Disable Siri's access Maps Settings
Disable Siri's
Health Settings
Disable Siri's access to News Settings
Create reninder
Birthday Gift Alice"
Create reminder of "Dinner with Peter" reminder "I
28

---

Published as a conference paper at COLM 2024
reninder of "Christmex Eve"
Send nessage to Kate Bell say "How are you doing?" Send text nessage using Messages app to the contact saying "Happy New Year!"
Use the Rentnderrs
set reninder for "Water the plants" for tonorrow.
Use the Reninders app to set reminder for "Wash Clothes" for tonerrow. Show the latest
took the Photos
Show me the photos in the albun "Racent" ne photos in
"Tavorite"
Use Haps to find the nearest Mclanald's.
Use Haps to find the nearest grocery store.
Use Haps to find the nearest gas station.
Use Haps to find the nearest netro station. Create event called "dinner" calendar for today
Create an event called "dinner" on calendar for tonorrow
Create
called "lunch" colendar
today
Create event called "Neeting with Josy" on calendar for today 8pm Create event called "Heeting with Anne" on calendar for tonorrow 2pm Create event called "Neeting with Simon" calendar for today Bam Use the Contacts app to find David Taylor's phone number. Hell phone nunber.
Use the Contacts app find
Use the Contacts app to find Daniel's phone number.
Contacts app find
Haro's birthday.
Use the Contacts to find David Taylor's birthday.
Use the Contacts app to find David Taylor's address.
Use the Contacts
find Kate Bell address.
Use the Contacts app to find Baniel's address.
Use the Contacts app find John Appleseed's address. Create a new Contact called Bill. Contact
Create
phone number 8888-8988.
Create new Contact called Kelly with phone number 1234-5678.
Use the Calendar app to add new all-day event called "On Vacation" at tonorrow. called "On Vacation" for yesterday. Open Files and create a new folder named "Work Documents" at "On My iPhone". article.
Use the Calendar app add
all-day
Open the News app and open top-nost
Open Safari and show ne HIT's wikipedia page.
Open Safari
Stanford's wikipedia page.
Open Safari and show CMU's wikipedia page.
Open Safari and show ne Apple's wikipedia page.
Open Safari and show Micrasft's wikipedia page.
Open Safari and show ne Anazon's 's wikipedia page.
--- Eval Tasks -**
Open Settings
Open News
Open the upp Watch
the Settings and open settings for Safari
Find the nearest gas station on Map
reminder of "take dinner"
Create a reminder of "gracery"
Create a reminder of "Finish hanewark" Send
Kate with "Hi"
Send message to John Appleseed with "Haw are you daing"
Find the nearest gas
with Maps
Use Maps to f'ind the nearest Target Haps find nearest CVS
Use Contacts to find John Appleseed's 's birthday Use Contacts
Bell phone number
Find Anna Haro's phone number with the Contacts app Find the app Freeforn and open
Find the app Shortcuts and open
Open Settings and open About page Find Appleseed's phone number contacts Find Kate Bell 's phone number with contacts
Show me image in Photos
Show me all images in Photos Open naps for me add reminder add reminder to do grecery
dinner
store on naps send message to Kate Bell with the nessage
nearest
send message John with nessage app find the nearest netro station Use Contacts to check out John Appleseed's hone address the Settings open Accessibility
Find the Model Number of this device
Disable Character Preview Keyboard Settings Change Safari Search Engine to DuckDuckta in Settings
Change the preferred language of Safari to Turkish in Sectings Change the preferred Disable Siri's access to Photos in Settings reminder of "Christnex gift for Kitty"
of travel of Maрs to Malking Settings
29

---

Published as a conference paper at COLM 2024
Use
app Send nessage to John say "How are doing?"
reminder "Finish honework" for tonorrow.
Show me all images have Photos
Show me the photos in the albun "Favorite"
Use Haps to find the nearest Burger King.
Create event called "dinner" calendar for tonorrow Create an event called "Heeting" on calendar for today 1pm Contacts app Use the Contacts to find Kate Bell birthday.
Apoleseed's phone
Use Contacts app to find Anna Haro's address.
Create Contact called Simon with phone number 1234-5678.
Use the Calendar app to add new all-day event called "On Vacation" for today. Open Safari and show Berkeley's wikipedia page.
Listing Prompts for Reflexion Agent Take Actions
proupt =
"intro"; menYou autoronous intelligent agent
navi navigating gating browser. S -based tasks. These tasks will be accomplished through the use of specific actions can issue.
given web
Here's the infornation you'll have:
The user"s objective: This is the task you're trying to complete.
current page's accessibility tree: This simplified representation webpage, providing key - information.
current page's URL: This is page you're currently navigating.
The tabs: These are the tabs have The previous action:
This is the action you just perforned. It may be helpful track your progress.
The actions can perforn fall into several categories:
Page Operation Actions:
"click [id]": This action clicks on an elenent with specific on the webpage.
"type [id] [content] poress_enter_ee al Use this to the content into the field with id. default, = the "Enter" key is pressed after typing unless press_enter_after is set to e. element with id.
"hover [id] Hover
'press [key_comb]'; Simulates the pressing of a key combination on the keyboard (e.a., Ctrl+v).
[directionsdown up]
the page up
Tab Management Actions:
browser tab. 'tab_ facus [tab_ stahh_index]' i: Swits the browser's facus to a specific tab using its index. "close_tab's Close the currently active tab.
'new_tab": Open
Navigation
"goto [url]": Navigate specific URL.
"go_back", Navigate to the previously viewed page. 'go_forward"; Navigate the page (if previous 'ga_back' action performed).
Completion
"stop [answer] ':
Issue this action when you believe the task is complete. If the objective is to find a text- is inpossible to complete,
answer, provide the answer in the bracket. If you believe + provide the answer as "N/K" in the bracket.
Honepage:
you want to visit other websites, check out the homepage at httt:://hooppgg.coo con. It has list of websites you
http:// httg:://hhepppee..o con/password.httm lists all the account nane and password for the websites. You can use them to
< log in
websites.
To be successful, it is very important follow the following rules:
should
action that is valid the current observation
2. You should only issue one action at a tine.
should follow the examples
step
4. Generate the action in the correct format. Start with "In sumary, the next action I will perform is" phrase,
+ followed by action inside thinking process before giving the action encouraged. Issue stop action when you think you have achieved the abjertive. Don't generate anything after stop."""
For example, "In sumaary, the next action will perforn is ""ccick
[1234] *.*. Including
"examples":
"""GBEEVATION:
[1744] 1ink 'IP CB782AAABA 648 Inkjet Fax Machine (Renewed)
[1749] StaticTekt '5279.49'
[1757] button "Add Cart"
[1760] button 'Add Wish List"
[1761] button 'Add Compare'
URL: http:///onestonareet.coom coe//offcee-prduutss/ffff eleettroniss
0BJECTIVE:
price Inkjet Machine
MEMORY: None
PREVICUS ACTION: None
30

---

Published as a conference paper at COLM 2024
"Let's think step-by-step. page list information HP Inkjet Machine, product + identified in the objective. Its price is $279.49. 49. I think have achieved the objective. will issue stop action with the answer, In summary, the next action will perforn "ssso 15279.493"""1
"""GEERVVATON:
[164] textbox 'Search' focused: True required: False
[171] "Go"
[174] link "Find directions between two points'
[212] heading 'Search Results'
[216] button "Close"
URL: http:://openstrreetmp..orr
OBJECTIIE:
Show the
CMU
MEMORY: 1 first searched "OMJ" and then searched for "restaurants", It looks like the search results are expected,
restaurants ansther city. think b should go back to the previous and search for "restaurants near OMI"
from
search results
PREVIOUS ACTION: None""",
"Let's think step-by-step. This
search box whose I0 [164]. According to the noninatin rule of openstreetnap, can search for the restaurants near a location by Vrestaurants near can submit typing by pressing the Enter afterwards. sumnary, the action will perform type [164] [restaurants near (MU) [1]**"*,
+
"template": ""OBSERVATION:
(observation)
UAL: (url)
OBJECTIVE: (objective)
HEMORY: (menory)
PREVIOUS ACTION: (previous_action)
"neta_data"; {
"observation": "acessibilltyy.rres tree".
"action_type", "id_.ccessibilityyteee"
"keywords": ['url", "ohjective", "observation', "previous_action", "nemory"], prrapt.constructoo": "feflextorrrommttentttrtaa
"answerpphrase": phrase":: "In summary the next action will perforn is", "action_splitter; *....
"answer
Listing 4: Prompts for Reflexion Agent to Reflect
proppt = "intro": """You c -based tasks. These tasks will be accomplished through the use of specific actions you can issue.
igget agent tasked with gatingg unb browser. You will be web
Heer''s s the information you'll have: user's objective:
the task you're trying complete. The web page's accessibility tree: This is simplified representation of the webpage, providing key information. The web page's 's URL: This is the page you're 'i'ree currently navigating. The open tabs: These the you have open.
actions you can perform
several categories:
Page Operation Actions:
"click [id]": This action clicks on element with specific id on the webpage.
"type [id] [contant] arrrss_mntr_aaterspap "Enter" key pressed after typing unless press_enter_aftte is set "hover [id]': Hover over an element with id.
Use this to type the content into the field with id. By default,
'press [key_conb]'s
pressing key combination on keyboard (e.g., Ctrley).
'scrall [directionsdom//ne up] Scrall the page or down.
Tab Management Actions:
new._tbb' Open a new, empty browser tab. 'tau_focus [tab_index] Switch browser's focus specific using its index.
"close_tab's :: Close the currently active tab.
URL Navigation Actions:
"goto [url]"; Navigate specific URL. bal.back' Navigate the previously viewed page.
go_roowwrdd Navigate to the next page (if previous 'go_back' action was performed).
Completion Action:
'stop [answer]
objective find <i based answer, provide the answer in the bracket. If you believe the task is impossible to complete, provide the answer as "N/A" in the bracket.
you believe
complete.
Now you are trying to evaluate your performance on past task. You will be given the objective of the task, the actions you issued,
history interaction including observations you
4 the task. You will also be given the nemory of your previaus attempts. Your geal is to think about the strategy path you atterpt complete task. Try to sumarrize the reason why you failed
31

---

Published as a conference paper at COLM 2024
concise, new plan accounts your mistake and helpful - when you are solving the same task. Try to think differently from the previous atterpts. Try to fecus key aspect and plan concise.
complete task,
"examples": L
I"OOJECTIVE: Compare the time for walking and driving route from AMC Waterfront to Carnegie Me Mellon University
OBSERVATION ACTION HISTORY:
OBSERVATION 8:
Tab (current): Dashboard Magento Admin
[1] RootMebArea 'Dashboard Magento Admin' focused: True
'Hagento Admin Panel'
[201] ] inn
'Magento Adnin Panel
[85] menubar
orientation: Norizontal
[87] 1ink
"Nue584 DASHBOARD
[90] link
"lur58b SALES'
[96] link
"Aun5e8 CATALDG" DG'
[192] 1ink
"Xue683 CUSTOMERS'
MARKETING"
[114] link
"sue602 CONTENT'
[120] 1ink "tue6da REPORTS'
[138] link "Sue6êd STORES'
[144] link "Aue618 SYSTEM
[158] link 'lue612
PARTNERS EXTENSIONS'
[821] button
"Systen Messages:
[983] link 'indexers are invalid'
[904] StaticText Make sure your
[985] link 'Magento jab
[906] Staticlext is running."
heading 'Dashboard"
[242] 1ink 'Nue6ee adnin'
[244] '\ue607"
[913] textbox 'Yue50c' required: False
[48] main
[219] StaticText 'Scope:"
[258] button 'All Store Views' hasPopup: menu
1ink
[226] button 'Reload Data'
[917] HeaderrssHeonanddurkk
[919] StaticText 'Advanced Reporting"
[928] StaticText "Gain new insights and take camman of your business' performance, using our dynamic product,
reports tailored your custoner data." [921] link 'Go to Advanced Reporting \ue644'
order, and
disabled. enable chart, click
[925] link
'here"
[1154]
[1854] StaticText '$8.00"
[1155] StaticText 'Tax"
[1156] Staticlext 'Shipping'
[1157] StaticText 'Quantity'
[1868] StaticText '8'
[57] tablist multiselectable: False orientation: horizontal.
changed. contains data, 4 before saving, Loading... Bestsellers' expanded: True selected: True controls:
[59]
+
has been changed. This tab contains invalid Please resolve < this before saving. Loading... Bestsellers'
[67] link 'The information in
information
changed.
contains invalid
resolve
+ before saving, Loading... Most Viewed Products' expanded: false selected: False controls: ui-id-1
has been changed. This tab contains invalid data. Please resolve + this before saving. Loading... Most Viewed Products"
[69] 'The information in
[63] tab 'The information in this tab has been changed. This tab contains invalid data, Please resolve this expanded: False selected: False controls: ui-id-2
before saving. Loading...
[713 1ink TTe information in this tab has been changed. This tab contains invalid data. Please resolve Custoners'
saving, Loading... [6S] tab 'The information in this tab has been changed. This tab contains invalid data. Please resolve this
before saving. Loading... Custoners' expanded: False selected: False controls: ui-id-3
[73] 1ink 'The information in this tab has been changed. This tab contains invalid data. Please resolve
< this before saving, Loading... Customers'
[79] tabpanel
information in < this before saving. Louding... Bestsellers'
changed. tab contains invalid Please resolve
[1088] table
[1158] row
[1159] columheader 'Product' required: False
[1160] columbeader 'Price" required: False
[1161] columheader 'Quantity' required: False
[1162] row http::/ccaale..777.edda///////od
32

---

Published as a conference paper at COLM 2024
[1167] gridcell 'Sprite Ball required: False
[1168] gridcell '$27.86 227.80' required: False
[1169] gridcell required: False
[938] staticText "Lifetine Sales'
[933] StaticText '30.00"
[937] StaticText 'Average Order'
[944] Staticlext 'Last Orders'
[945]
[979] row
[980] colunrheader "Custoner' required: False
[981] colunrheader: 'Itens' required: False
[982] columreeader 'Total' required:
False
[883]
[988] gridcell 'Sarah Miller' required: False
[989] gridcell '5' required:
[998] gridcell '8194.48' required: Falsse
[984] row
[991] gridcell "Grace Nguyen' required: False
[992] gridcell '4' required: False
[993] gridcell '$198.88 required: False
ACTION stop [N/K]
STATUS: FAILED
REFLECTIONS
FROM PREVIOUS ATTEMPTS: none""",
think the task inpossible complete, thus the stop action. task is not 4 completed successfully, which neans an wrong. I think I should go to the V"REPORTV" tab and do best-selling products tine."
GOJECTIVE: List reviewers, if exist, who mention about fingerprint resistant
OBSERVATION 8:
sf secplaa-----crree--rrttttteeeeee -etpppatttteeeeeeeeeeeeeeeeeeeee tempeeed-glasss glaas--ccree--pootcttr--aa < -Bthtaardnes--crystall -llaetl--lee-----buubllle trsterrrrrrensslllaaatoosssrrrt tttt
http://loccahhsst777788 sancan-- galayyy--
Tab (current): [3 Pack] Sansung Galaxy 56 Screen Protector, Nearpow (Tenpered Glass] Screen Protector with [91] + Hardness] [Crystal Clear] [Easy Bubble-Free Installation] [Scratch Resist]
[1] RootMebArea '[3 Pack] Samsung Galaxy s6 Screan Protector, Nearpow [Tenpered Glass] Screen Protector with [SH
Hardness] [Crystal Clear] (Easy Bubble-Free Installation] (Scratch Resist] focused:
[1314] 1ink 'My Account"
[1312] 'My
[1316] 1ink "Sign Out'
[1319] StaticText "Welcone, Erna Lapez!'
[1220] link 'Skip Content"
[1229] link "store logo'
[1322]
sove_sstp_narrke.c
[2246] "lue611 My Cart'
[1323] link
[1508] conbebox "\ue515 Search" autocomplete: both hasPopup: listbox required: False expanded: False
[2249] link 'Advanced Search'
[1511] button 'Search' disabled: True
[1096] tablist multiselectable: False orientation: horizontal
[1898] tabpane1
[48] menu
[791] orientation: vertical
Beauty Personal Care! hasPopup: nenu
[856] menuiten "Aue622 Sports Outtloors' hasPapup:
[866] menuiten 'ue6222 Clothing, Shoes & Jewel hasPopup: menu
[888] menuiten 'Aue622
Kitchen' hasPopup:
[917] menuiten "Nue622 office Products' hasPopup: menu
[925] menuiten
Inprovement' hasPopup: menu
[930] menuiten ten "Aue622 Health & Household' hasPopup: nenu
[936] menuiten "ue6622 Patio, Lawn Garden" hasPopup: menu
[941] menuiten 'ue6622 Electronics' hasPopup: menu
[1002] nenuitan "Nuezz2 Cell Phones Accessories' hasPopup: menu
[1817] nenuiten "Nue622 Video
hasPopup:
[1838] nenuiten "Nue622 Grocery & Gourmet Food' hasPapup: menu
[1253] link
[1256] Staticlext '[3 Pack] Sansung Galaxy S6 Screen Protector, Nearpow [Tenpered Glass] Screen Protector with
< [96 Hardness] [Crystal Clear] [Easy Bubble-Free Installation] [Scratch Resist]"
[5] nain
[1257] heading '[3 Pack] Sansung Galaxy s6 Screen Protector, Nearpow [Terpered Glass] Screen Protector with Hardness] [Crystal clear] [Easy Butble-Free Installation] [Scratch Resist]"
[11] generic "Availability""
[13] StaticText
[1331] Staticlext 'SKU"
[1467] StaticText 'B81G31IY48'
[1264] LayoutTable
[1469] StaticText 'Rating:"
generic '78x'
33

---

Published as a conference paper at COLM 2024
[2221] StaticText 'X
[2224] StaticText '100"
[1335] '12'xa0 Reviews
[1336] 1ink "Add Your Review"
[1338] StaticText '$7.99'
[1279] LayoutTable
[1483] Staticlext 'Qty'
[1484] spinbutton "Qty' required: Folse voluemin: valuetext:
[1885] button 'ddd to Cart'
[1281] link Add Wish List'
[1282] 1ink 'Add to Compare'
[1287] link "Skip to the end of the inages gallery'
[1117] button "Previous"
[1119] generic 'Inage'
[2252] ing 'Inage'
[1118] button 'Next'
ACTION click [1335] where [1335] is [1335] link '12/xa8 Reviews
OBSERVATION 1:
<- http://localhost:777//33
sansung
Tab (current): [3 Pack] Sansung Galaxy 56 Screen Protector, Nearpow (Tempered Glass] Screen Protector with [9H
+ Hardness] [Crystal Clear] [Easy Bubble-Free Installation] [Scratch Resist]
[1] RootMebArea '[3 Pack] Sansung Galaxy 56 Screen Protector, Nearpow [Tempered Glass] Screen Protector with [SH
Hardness] [Crystal Clear] [Easy Bubble-Free Installation] [Scratch Resist] focused:
[5] main
[1349] StaticText
beginning
inages gallery'
[1186] tablist " nultiselectable: False orlentation: horizontal
[1187] tab 'Details' expanded: False selected: False controls: description
[1350] 1ink 'Details"
[1110] tab 'Reviews (12)' expanded: True selected: True controls: revinws
[1352] "Reviews (12)'
[2365] tabpanel 'evvees (12)'
[2460] StaticText "Custoner Reviews'
[2555] StaticText "Best screen protectors I've used!"
[2519] LayoutTable
[2556] LayoutTableRov
[2699] LayoutTableCell 'Rating'
[2789] generic '1085'
[2559] StaticText 'It is super clear and fingerprint resistant, It was kind of hard trying to get it on, was great! Bubbles went away
the sticky side, but barely tell they there. They also give tons of extra tools to help clean the screen and get dust particles off of the screen before you put it on. think it was just ne being clunsy with all the dust particles gateing inside the screan."
get
snall hairs
[2562] StaticText "Review by
[2564] StaticText 'Rachel'
[2567] StaticText "Posted on
[2568] ".
[2781] StaticText '4/18/23'
[2569] StaticText "Good screen protector for the money and good custaner service'
[2522] Layoutlable
[2578] LayoutTableRon
[2782] LayoutTableCell 'Rating'
[2783] generic '88%"
tricky to apply. few tines without incident. The last 4 had it ny phone for about nonths and had drapped product.
[2573] Staticlext
drop shattarnd the protactor but thankfully did what was supposed to do and protacted my phone screen. the package had snall chip + way through. I enailed the company and they were very quick to respond and sent new one free of charge. very
which caused product only give
have hairline crack all the
rating because it difficult to get out the bubbles. have very tiny specks that would just not cone out.
very satisfied
[2576] StaticText "Review by
[2578] StaticText 'chris"
[2581] Staticlext 'Posted on
[2582] ..
[2784] StaticText '4/18/23'
[2583] StaticText "Bubbles
days"
[2525] LayoutTable
[2584] LayoutTableRow
[2785] LayoutTableCell 'Rating"
[2706] generic '8%
with only snall bubble. like 4 it normal to have few smull bubbles. The description says that the snall bubbles will disappear small tiny bubble went away. Ugh need add this to review. The glue does not last forever. It started to cone off about a nonth after put it on. The bad thing when it does start to come off, it's easy to take off the protectant."
[2587] StaticText "OK, my first impression was,
a couple days.
there. never
ACTION
34

---

Published as a conference paper at COLM 2024
[Rachel]
STATUS: FALLED
REFLECTIONS FROM PREVIOUS ATTEMPTS: none""",
"I find the review from Rachel, which is the to the objective. issue the action with the by answer. However, the task is not completed successfully. this might be because I missed other reviews should
nention good fingerprint
time."
"tepplate:: ""OJJCTTIV:: (objective)
OBSERVATION ACTION HISTORY:
(trajectory)
(status)
REFLECTIONS FROM PREVIOUS ATTENPTS: (nemory)"*",
"meta_data"; {
"abservation"; "accessibility_teee
'action_type"; "fd_sceessiillity.treee
"keywords": ["objective", "trajectory", "status", "nenory"J,
"prempt_cons"" corppt.costtuctor.. "ellleciooeerssttorratttnsssssss
bonswer_phaase"
"sctionspliiter": *****
35


Reviews:
[{"review_id": "l990fAEdRI", "reviewer": "Reviewer_saqr", "summary": "This paper empirically studies model's automatic evaluation of trajectories on digital agents, such as agents operating on the web and Android. The authors focus on two methods to construct the evaluator, one is an end-to-end approach that directly uses a pretrained VLM, another is a modular approach that first transcribes the screenshot to captions with a VLM, and then feeds all text to a LM. Both methods demonstrate high accuracies against oracle evaluator and human judge.  The authors then utilize such automatic evaluators to improve the policy agent, through reflexion and filtered behavioral cloning respectively. In both settings, the evaluation is able to improve the tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We respond to your points below and will incorporate them into our paper.\n\n> step-level evaluation is only applied for the iOS agent refinement\n\nWe focus on using step-level feedback for training via filtered BC, not as an evaluation metric. We didn’t apply this learning approach to WebArena due to the lack of a strong open-weight model in this domain.\n\n> how accurate such step-level evaluation is\n\nComprehensive evaluation of step-wise metrics requires significantly more hand annotation than overall trajectory success. However, human annotators agreed with our step-wise evaluator for 43 of 50 state-action pairs randomly sampled from the iOS experiments. \n\n> how significant [filtered BC] result is\n\nPlease see our response to Reviewer zFha.\n\n> end-to-end methods only evaluate GPT-4V\n\nPreliminary results showed no existing open VLM can simultaneously perform strong reasoning and GUI understanding. This motivated our caption-then-reason approach, which decouples the two capabilities into separate models.\n\nHowever, to address this concern, we evaluated QWen-VL-chat (the base model for our Captioner and strongest open VLM at the time) as an end-to-end evaluator. It significantly underperforms the open-weight Captioner + Mixtral (from 74.4 to 68.0% accuracy in WebArena; 92.9 to 70.2% in AitW). We didn’t proceed with full agent refinement experiments here given its worse performance.\n\n>  where the final state is difficult to classify\n\nOur method supports feeding intermediate states as input to the evaluator, though this wasn’t helpful in preliminary experiments. We are optimistic that as VLMs perform better long-context reasoning, evaluators will also perform well in cases where the final state alone is hard to classify; e.g. when booking a flight and the confirmation page lacks all flight details.\n\n> experimenting with VisualWebArena?\n\nWe use WebArena, which natively supports both text or pixel-based observation space.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Empirically studying whether current models are able to evaluate digital agents’ trajectory success answers a previously unknown question and is useful practically as well.  \n2. The authors successfully utilize the model evaluators to improve the performance on two difficult tasks, with different approaches.   \n3. This paper studies both end-to-end and modular approaches, across both proprietary and open-source models, which can provide practical guidance for different readers.", "reasons_to_reject": "As an empirical paper, I expect the experiments to be conducted in a more comprehensive manner with more details, and in general the empirical results are relatively sparse.  Some examples:   \n(1) while the authors talk about both trajectory-level and step-level evaluators, step-level evaluation is only applied for the iOS agent refinement, and there is no study how accurate such step-level evaluation is. From Table 2, I guess step-level evaluation may not be very accurate given small improvement over self-training;   \n(2) end-to-end methods only evaluate GPT-4V, while I would like to see the performance of more open VLMs;   \n(3) In Table 2, filtered BC only finishes 3 more tasks than self-training in a test of 52 tasks. I am not sure how significant this result is.", "questions_to_authors": "1. The high accuracy in Table 1 is nice, but I am feeling this is because the special properties of the two environments – for example, the final screenshot is typically enough to classify whether it satisfies the goal. I wonder whether the authors think such high accuracies are general, and whether there are environments where the final state is difficult to classify, I would like to see more discussions on this aspect as potential limitations of the paper.  \n2. Webarena by design is a text-only task originally, I see the authors are using GPT4-V on that, are the authors experimenting with VisualWebArena or am i missing sth?\n\n```\nAfter rebuttal, I bump up my score to 6 given the clarifications and added results to increase the significance of the previous results.\n```", "extra_scores": {}}, {"review_id": "u1xKHMzbko", "reviewer": "Reviewer_zFha", "summary": "This paper designs two lines of automatic evaluation for digital agents: (1) Using VLMs to caption the trajectory and then prompting the LLM to reason if the trajectory is correct. (2) Using an end-to-end VLM to evaluate the trajectory (GPT-4V in the paper). These methods show a high correlation with ground truth evaluation metrics, and the paper demonstrates that the agents can be improved by using reflective prompting or filtered BC.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We respond to your points below and will incorporate them into our paper.\n\n> evaluation of filtered BC in the iOS domain is not very persuasive\n\nScaling experiment in real-world tasks is challenging, shown in recent works ([Du. 2023](https://proceedings.mlr.press/v232/du23b.html) and [Zheng. 2024](https://arxiv.org/abs/2401.01614) use 20 and 92 evaluation tasks respectively). The iOS experiment is limited by requiring physical Apple machines and slow inference speeds of CogAgent, taking 4 physical days for the whole experiment, thus we chose an evaluation set of 52 tasks.\n\nHowever, to address this concern, we apply filtered BC to AitW (unlike iOS, we can use Android emulators on clusters). We use 560 tasks for training and 96 for evaluation. The base policy (AutoUI-base) completes 15 tasks, improving to 18.9 ± 1.0 (averaged over 3 training runs) with self-training. Filtered BC with our step-wise evaluator still significantly improves, succeeding at 26 ± 0.8 tasks.\n\n> seems common in existing literature\n\nThank you for highlighting these relevant works; we will add a discussion of them into our paper. We clarify that we don't claim novelty in proposing LM-as-judge or in applying these inference and training techniques.\n\nOur core contribution is to provide strong and comprehensive evidence on the effectiveness of domain-general evaluators in the realistic digital agent domain. Current evaluations of digital agents, conducted in hand-crafted simulators or through reference-based metrics, suffer from labor intensity and limited generalizability. We characterize the pros and cons, and explore the opportunities of using autonomous evaluators. Additionally, we apply exploration-driven learning techniques like filtered-BC to train language agents, in contrast to existing methods that rely on either training with expert demonstrations or inference-time algorithms.\n\nWe distinguish ourselves to [1, 2] in\n- Digital agents domain is more open-ended and complex than pick-and-place or insert-remove tasks [1] or plans generated by another agent [2], allowing for arbitrary user instructions across broad domains.\n- Unlike [1], we don’t need any human demonstrations or success labels to train our evaluator, which also mitigates performance degradation in out-of-distribution cases.\n- We experiment with using our evaluators to evaluate policies of varying quality, in contrast to just using the evaluators in a training/refinement loop as in [2].", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow.\n- The framework is interesting and holds promise for digital agent development. The paper has the potential to provide insights for the community.\n- The experiments involving filtered BC are interesting and address the data scarcity issue in some agent learning settings well.\n- The paper curates data and tasks in the iOS domain, which may benefit future studies on digital agents.", "reasons_to_reject": "- Using LLMs to evaluate the trajectory of agent policies and improving the agent seems common in existing literature. For example, [1] also uses LLM evaluators to conduct filtered BC, and [2] uses reflection.\n- The evaluation of filtered BC in the iOS domain is not very persuasive because Table 2 only shows the comparison of the number of successful tasks for different methods. It is unclear if solving 14 out of 52 tasks represents a real improvement or just randomness.\n\n[1] Du et al., Vision-Language Models as Success Detectors.\n\n[2] Wang et al., JARVIS-1: Open-world Multi-task Agents with Memory-Augmented Multimodal Language Models.", "questions_to_authors": "I think the methods in this paper are promising for digital agent development. But the weaknesses on originality and evaluation may limit the contribution of this paper. I would like to raise my assessment for the paper in the followup phase if the authors could well address the issues and make a rounded discussion.", "extra_scores": {}}, {"review_id": "u3QTPd1Xrg", "reviewer": "Reviewer_tJw6", "summary": "This paper tackles an important problem of agentic research: how to automate the evaluation of domain-general agents? Current agent benchmarks rely heavily on hard-coded rules for evaluation, which is expensive to construct and hard to scale. This paper proposes a multimodal approach to automate this process. They first train a VLM-based Captioner to generate a textual description of the agent's observation (i.e., the screenshot). Then the LLM-based Reasoner will take this description along with the actions, and the original user\ninstruction to judge whether the execution is successful or not. \n\nGiven such an automatic evaluator, they further show that it can be used to improve the performance of existing agents by providing inference-time guidance for self-refinement.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We respond to your points below and will incorporate them into our paper.\n\n> key to Captioner's success is…1,272 manually curated caption annotations...slow and expensive\n\nWe’d like to correct a possible misunderstanding. **Collecting these annotations is actually cheap and efficient**. The cost for querying GPT-4V for initial annotation is about $40, and the manual coarse filtering process took only around 2 hours.  \n\nSecondly, this caption dataset is only used to teach the model to generate captions with appropriate granularity and correct markdown format. Motivated by recent studies on fine-tuning language models ([Zhou. 2023](https://arxiv.org/abs/2305.11206), [Gudibande. 2023](https://arxiv.org/abs/2305.15717)), we suspect Captioner’s performance is bottlenecked by base model’s GUI understanding capability learnt during pre-training, rather than the size of the fine-tuning dataset (i.e., adding more fine-tuning data has diminishing returns). With open VLM’s capabilities continuously improving, we are optimistic that they should be able to perform the captioning task in zero- or few-shot in the future, similar to proprietary models such as GPT-4V.\n\n> screenshots cannot capture the change of state\n\nDigital environments are designed to be user-friendly, displaying crucial information directly on the screen, such as the consequence of an action. For example, `mv file_a folder_b` will return an error message in the terminal if it fails and otherwise nothing. Thus, trajectory (screenshots and actions) should still be adequate for evaluation in well-designed environments. Part of the motivation behind using the agent trajectory as input is to provide models with the same amount of evidence as a human judge.\n\nHowever, we agree there will be cases where there’s insufficient evidence in trajectory to determine the agent's success. One promising way to address this is to train or incentivize agents to show evidence of success, e.g., by rewarding the agent to `ls .` and `ls folder_b` to show the contents of the two folders after the task has been completed. \n\n\n> biggest challenge in automating agent evaluation\n\nWe think the biggest challenge is in evaluating agents actually deployed in real-world systems at scale. Our work moves towards this goal of going beyond manually-crafted environments by providing reproducible and cost-effective evaluations that align closely with oracle judges. We are happy to discuss other challenges as well!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is well-motivated and clearly presented.\n- The problem being tackled is important and impactful.\n- The proposed approach, although quite simple, is effective on two evaluated benchmarks.", "reasons_to_reject": "- The key to Captioner's success is the collection of 1,272 manually curated caption annotations, whose collection is slow and expensive. Is there any way to automate this process such that it can be scaled up?\n- The proposed approach, despite being effective on WebArena and AitW, might not work well in domains where screenshots cannot capture the change of state. For instance, if an agent executes \"mv file_a foler_b\" in the terminal. The action will be executed completely in the backend and nothing will be shown unless both folders are explicitly open and displayed on the screen.", "questions_to_authors": "What would you perceive as the biggest challenge in automating agent evaluation?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper shows that domain-general LLMs are sufficiently good reward models to be able to improve web agents. It has certain weaknesses. It is primarily an empirical paper, and the range and scale of the empirical evaluations seems limited compared to other domains (in part because of the difficulty of scaling evaluations in this domain). And the methods studied are similar to or the same as existing methods. Nevertheless, this is a good study and a well-written paper in an area of intense current interest. It makes real progress in showing that domain-general evaluators (LLMs or a good VLM, really only GPT-4V currently) can be quite effective for general tasks in the web agent domain, for either inference-time guidance or behavior cloning, obviating the need for hand-designed evaluation functions. The transfer learning extension to iOS, while smaller in scale than would be ideal, nevertheless seems a very useful contribution to have.
Decision: Accept