Autonomous Evaluation and Refinement of Digital Agents

Authors: jiayi  pan
yichi  zhang
nicholas  tomlin
yifei  zhou
sergey  levine
alane  suhr
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 02:09:29

Abstract: We show that domain-general automatic evaluators can significantly improve the performance of agents for web navigation and device control. 
We experiment with multiple evaluation models that trade off between inference cost, modularity of design, and accuracy. 
We validate the performance of these models in several popular benchmarks for digital agents, finding between 74.4 and 92.9% agreement with oracle evaluation metrics.
Finally, we use these evaluators to improve the performance of existing agents via fine-tuning and inference-time guidance.
Without any additional supervision, we improve state-of-the-art performance by 29% on the popular benchmark WebArena, and achieve around 75% relative improvement in device control settings.
We release our code and data at 
 [https://github.com/Berkeley-NLP/Agent-Eval-Refine](https://github.com/Berkeley-NLP/Agent-Eval-Refine)

Main paper: 
 introduction: 
 Given an instruction, e.g., "Tell me the cost of my latest canceled order," an automated digital agent would be expected to first navigate to a user's profile page, then to a list of their previous orders, identify the most recent order that has been canceled, and return its total amount to the user. Such agents offer the long-term potential of making digital devices more accessible, while also simplifying tedious or mundane tasks. However, in the short term, even state-of-the-art agents still make mistakes on simple tasks. Evaluating such agents and characterizing their failure modes is not only important for understanding and improving the models, but also critical for safely deploying them in real world. In this paper, we demonstrate the opportunities and efficacy of using automated evaluation models to both characterize and improve agent performance, without requiring access to any extra supervision, such as expert demonstrations or evaluation functions.We propose to automatically evaluate user instructions and arbitrary agent trajectories with domain-general neural models. We explore two main variants of this approach (Figure 1 , left): first, a modular caption-then-reason approach where a vision-language model (VLM) first captions the screenshots, and a language model (LM) is used to reason about if an agent succeeds based on textual information; and second, an end-to-end approach where we prompt an advanced VLM like GPT-4V (Achiam et al., 2023) to directly evaluate a trajectory. These two different approaches offer trade-offs in performance, cost, and transparency.We first evaluate our proposed approach on its ability to match oracle evaluation metrics using WebArena (Zhou et al., 2024) and Android-in-the-Wild (AitW; Rawles et al., 2023) , achieving accuracies up to 82.1 and 92.9% respectively. We then show how these evaluation models can be used to refine existing agents through inference-time guidance or during training, without access to any hand-designed evaluation functions or additional demonstration data (Figure 1 , right). When integrated as the reward function in Reflexion (Shinn et al., 2023) , the evaluator enhances the best-performing GPT-4 WebArena agent's success rate by up to 29% of relative improvement. Additionally, we evaluate in both Android and iOS device control settings. For iOS, there is no existing benchmark environment or training data. When using our evaluation models to filter sampled trajectories to be used in behavior cloning, we see relative improvements of around 75%. time without human supervision. Approaches like Reflexion (Shinn et al., 2023 ) assume access to an external evaluation function, leveraging it as the supervision signal to guide policy improvement at inference time. In contrast, we study applications of inferencetime autonomous refinement without requiring access to the external evaluation function, and show that using our proposed domain-general evaluation models improves agent success rate by 29%. Meanwhile, methods which bootstrap policies with supervised training and refine them with reinforcement learning have been widely adopted; our proposed evaluators enable this paradigm in open, realistic digital agent scenarios, providing a relative improvement in performance of over 70%. Concurrent to our work, WebVoyager (He et al., 2024 ) also explores using GPT-4V as an automated proxy for human evaluation of web agents, though is not the primary focus of their work, and neither performs in-depth analysis of the quality of its judgments, nor explores its applicability to improving agents. More broadly, model-based metrics have been highly successful across various domains, such as evaluating image or text quality (Heusel et al., 2017; Zheng et al., 2024) , and assessing success of a robot or human activity (Du et al., 2023) . Part of our contribution is to provide strong empirical evidence to extend this success to the domain of real-world digital agents.Recently-proposed benchmarks that study digital agents fall roughly into two categories: simulation-based benchmark and demonstration-based one. Simulation-based benchmarks include environment simulators that offer the ability to execute arbitrary agent trajectories. Early simulation environments such as WoB (Shi et al., 2017; Liu et al., 2018) , WebShop (Yao et al., 2022) , and others (Branavan et al., 2009) are limited in their domain coverage, realism, or generalizability of their evaluation functions. Recently proposed simulation environments like AndroidEnv (Toyama et al., 2021) , WebArena (Zhou et al., 2024) and VisualWebArena (Koh et al., 2024), though far from perfect, have offered improvement across these dimensions. However, designing simulators, curating tasks, and handcrafting evaluation functions fundamentally limits their ability to mirror task and environment diversity of real environments.In parallel, the community has focused on demonstration-based benchmarks that do not include an executable simulation environment, including PIXELHELP (Li et al., 2020) , MoTIF (Burns et al., 2022) , Mind2Web (Deng et al., 2023) , and AitW (Rawles et al., 2023) . Notably, Mind2Web and AitW contain over 2K and 715K human trajectories respectively on a wide range of web navigation and device control tasks. Though primarily used for model training (Rawles et al., 2023; Deng et al., 2023 ; ?; Zhang & Zhang, 2023) , these datasets are also used for evaluating digital agents through reference-based metrics like action matching score. In this setting, an agent is given the prefix of a human demonstration and evaluated on its prediction of the next action to take. However, this metric requires human demonstrations and does not directly reflect agent's performance in real-world because it does not account for consequences of an agent's sequential decision process, alternative actions that diverge from the demonstration.We propose a third approach in which arbitrary instructions and agent trajectories are directly evaluated by a model. 
domain-general evaluators: 
 We develop multiple domain-general automatic evaluators for digital agents. Given a user instruction x and an initial environment state s 0 , an agent generates and executes a sequence of actions a = ⟨a 0 , a 1 , . . . , a n ⟩, resulting in a sequence of state visits s = ⟨s 0 , s 1 , s 2 , . . . , s n+1 ⟩.In this work, we assume a and x are in text form, such as <Type:''Hello''> and "Check the weather", and each state s is represented as a screenshot image. Given x, a, and s as input,the model produces a scalar evaluation r = ⟨r 0 , r 1 , . . . , r n ⟩ corresponding to each step of the trajectory: r = evaluate(x, a, s) . The evaluator can provide either trajectory-level or per-step evaluations. For trajectory-level evaluation, r 0 = • • • = r n-1 = 0, with r n = 1 for successful trajectories and r n = 0 otherwise. For per-step evaluation, we classify each step into three types, r i = 1 indicates task success after action a i , r i = p ≥ 0 indicates progress toward the goal, and r i = d < 0 is assigned to actions that do not contribute to the objective. We query the model once for trajectory-level evaluation and n times for per-step evaluation, reducing the model's task into a binary or ternary classification problem at each step.We explore two methods for constructing the model:1. An end-to-end approach that maps directly from instructions and screenshots to an evaluation via a pre-trained VLM. 2. A modular approach which first transcribes the observed screenshots into text descriptions using a VLM, and then uses a LM to map the descriptions, actions, and user instruction onto an evaluation. Both methods have tradeoffs: in the first, we can apply advanced VLMs like GPT-4V. However, this approach is relatively expensive and relies on API calls to proprietary models. In the second, we can compose open-weight models to achieve slightly weaker performance, but with added benefits of explainability via modularity and low-cost local deployment.We directly provide an instruction-tuned VLM with x, a, and s. We prompt it to first produce a text-based reasoning process (Wei et al., 2022) , then output its evaluation result. In our experiments, we use the proprietary vision-language model GPT-4V (Achiam et al., 2023).1 Many existing approaches for joint reasoning about language and vision disentangle perception and reasoning. In these approaches, a VLM is first applied to visual input to generate a language-based description; then, a text-only model (e.g., a LM) takes as input this description and the user instruction to produce a response by reasoning only about linguistic inputs. Existing work applying this approach has mostly focused on joint reasoning about natural images and text, e.g., for visual question answering (Guo et al., 2023; You et al., 2023; Wang et al., 2023b) . We take a similar approach here, where we first use a VLM to produce a description of the agent's observations given as s , then feed these descriptions, along with actions ā and the user's instruction x to an LM to produce a final evaluation.2 Captioner One drawback to this modular approach is the potential for information loss, where the image description may not include all the details necessary for task success (Wang et al., 2023b) . In our case, this could include missing or misrepresenting details about the screenshot, and indeed, we find that current open-weight VLMs struggle to produce detailed screenshot descriptions out of the box. In contrast, the most advanced, yet proprietary, VLMs can produce very detailed descriptions with adequate prompting.To improve a captioner's ability to provide detailed, well-formatted descriptions, we collect a dataset of screenshots paired with descriptions, and use it to fine-tune an open-weight VLM. We first acquire screenshots from a variety web and device control domains, then use GPT-4V to provide an initial detailed description for each screenshot. We manually filter out or fix apparent errors in GPT-4V's output, resulting a total of 1,263 data points. 3 We use this data to fine-tune the QWen-VL-chat (Bai et al., 2023) model. During both finetuning and at inference time, we provide text recognition results from EasyOCR4 as an additional input to the VLM to reduce hallucination.At inference time, we use our finetuned captioner model to acquire a description for each step in the agent trajectory. Critically, we do not provide this model access to the original user instruction, as we find this exacerbates model hallucinations; e.g., describing webpage attributes which would be relevant to the task, but are not actually present in the screenshot.Reasoner Finally, we provide the actions, generated descriptions, and the original user instruction to a language-only instruction-tuned model. We experiment with prompting two LMs, Mixtral (Jiang et al., 2024) and GPT-4, to produce a text-based thought and reasoning process as well as the final evaluation. 
experiments and results: 
 Our goal is to show how domain-general evaluation models can support the autonomous evaluation and refinement of digital agents, without requiring access to human demonstrations or oracle evaluation metrics. To this end, we first evaluate how these models perform as autonomous evaluators by comparing their judgments to benchmark-provided metrics and human judgements (Section 4.1). We then illustrate how these evaluation models, while imperfect, can serve as discriminators in autonomous refinement settings through both inference-time policy refinement (Shinn et al., 2023) and filtered behavior cloning (filtered BC; Chen et al., 2020; 2021; Emmons et al., 2022) to support significant improvements in agent performance (Section 4.2).Our rationale behind experiment design is to cover a broad range of domains and challenges. We use WebArena for both evaluation and inference-time refinement, as its built-in evaluation functions facilitate direct comparison. Android-in-the-Wild (AitW) is chosen for evaluation since it is widely used for training and evaluating Android agents, and is typically evaluated using a reference-based metric instead of task success. Next, we refine a model through filtered behavior cloning on iOS, where data scarcity poses a significant challenge to supervised methods. We further validate the effectiveness of filtered behavior cloning with a similar, larger experiment on Android.Environments WebArena (Zhou et al., 2024 ) is an offline web emulation environment and dataset that supports execution of arbitrary policies. WebArena comprises 812 humanwritten task instructions across various domains, including shopping, maps, and content management systems. Each instruction is paired with a handwritten test case that verifies agent success, e.g., by checking the status of a specific webpage element against a reference. We refer to this set of test cases as WebArena's oracle evaluator.Android-in-the-Wild (AitW; Rawles et al., 2023 ) is a large-scale dataset for Android device control containing 715,142 human demonstrations of 30,378 unique instructions. In our experiments, we focus on a subset of 120 tasks randomly sampled from the AitW test set. 5Unlike WebArena, AitW does not include an emulation environment for agent execution. Instead, the suggested evaluation metric is based on action matching: given a sequence of actions representing the prefix of a human demonstration, the agent is evaluated on its ability to predict the next action in the demonstration. While we compare against this reference-based metric in our experiments, we focus on end-to-end task-level success rate and implement an Android emulator to support execution of arbitrary trajectories. 6 We refer to human judgements on trajectory success as the oracle evaluation. Despite significant interest in developing digital agents, progress in the domain of iOS device control has been modest, with the exception of Yan et al. (2023) , who collect a small unreleased dataset of human demonstrations in this domain. We curate a set of 132 tasks in the iOS domain, taking inspiration from tasks included in AitW. We experiment with using our proposed evaluation models to facilitate domain transfer, with the goal of applying the strongest model on AitW, CogAgent (?), to iOS. We develop a Python interface to the iOS emulator on macOS, and design its action space to align with the Android-in-the-Wild to facilitate domain transfer. 6We evaluate three evaluation model variants:• GPT-4V: End-to-end approach (Section 3.1) using GPT-4V.• QWen-VL-chat: End-to-end approach (Section 3.1) using QWen-VL-chat. This serves as open-weight model's baseline performance. • Captioner + Mixtral: Modular approach (Section 3.2) using a finetuned QWen-VLchat (Bai et al., 2023) to generate a trajectory description, and Mixtral (Jiang et al., 2024) to provide the final evaluation. • Captioner + GPT-4: Modular approach (Section 3.2) using a finetuned QWen-VLchat to generate a trajectory description, and GPT-4 to provide the final evaluation.In most experiments, the evaluation model produces a trajectory-level evaluation, and takes as input only the last frame s n+1 in the trajectory, along with the instruction x and action sequence ā. Preliminary experiments suggested that model performance does not improve with information about previous states, likely due to limitations of existing models in processing long contexts. In the iOS experiments, the evaluation model takes as input the entire trajectory s and ā and the instruction x, and produces a per-step evaluation.We experiment with evaluating and refining the current state-of-the-art digital agents. In WebArena, this is a GPT-4-based agent described by Zhou et al. (2024) .For each task, GPT-4 is provided the user's instruction and the current DOM representation of the webpage derived from its HTML accessibility tree. GPT-4 is prompted to generate an action grounded in the DOM, e.g., clicking a button with a specific element ID. The task success rate of this agent is 14.4% as reported in Zhou et al. ( 2024), whereas our reproduction, as shown in Figure 3 , indicates a success rate of 15.6%.The strongest agent on the AitW benchmark is CogAgent (?), followed by Auto-UI {large, base} (Zhang & Zhang, 2023) . These agents are implemented as neural visionlanguage models that map observations, represented as images, and instructions to executable actions. We also experiment with the human demonstrations provided in AitW.7 WebArena For each WebArena task and corresponding trajectory sampled from the GPT-4-based policy (Zhou et al., 2024), we acquire task-completion judgments for each of the three evaluation systems described above. Table 1 shows the overall accuracy of the evaluator's predictions. 8 The end-to-end approach with GPT-4V achieves 80.6% accuracy, while Captioner + Mixtral, which uses only open-weight models, matches the oracle's evaluations for 74.4% of tasks, and replacing Mixtral with GPT-4 achieves the highest accuracy at 82.1%. Additionally, we see that using QWen-VL-chat end-to-end significantly underperforms the modular, open-weight Captioner + Mixtral approach, confirming the effectiveness of our modular design choice.Android-in-the-Wild For the 120 sampled test tasks in AitW, we evaluate trajectories sampled from four policies: CogAgent (?), Auto-UI {large, base} (Zhang & Zhang, 2023) , and human experts (Rawles et al., 2023) . 7 We acquire human judgments of trajectory success, as well as judgments from the three evaluation model variants. 9 Figure 2 shows the performance of all four agents as evaluated by humans and the three evaluator variants. We use human judgments of trajectory success as oracle reference and compare it with judgments from our evaluation models and AitW's standard action matching score.Below each agent label we also include each policy's partial action match score (Li et al., 2020) , which is the standard reported metric for agents on AitW. 10Unsurprisingly, we find that the human reference trajectories achieve the highest performance as evaluated by all success metrics. However, our analysis reveals that about 36% of the human demonstrations we annotate are actually unsuccessful, with common errors including early stopping, completing the wrong task, and making mistakes with respect to the parameters of the task. The difficulty of collecting high-quality demonstration data at scale further demands automated evaluation methods that can either act as a quality filter or provide more direct evaluation than action matching score.Among the three neural network policies, CogAgent achieves the highest success rates, followed by Auto-UI base , while the performance of Auto-UI large is close to zero according to all evaluators. When comparing conclusions that can be drawn from the two styles of metrics -task success and action matching -there are three clear differences: first, that success rate lags far behind single-step action prediction; second, that relative performance of models changes depending on the metric used; and third, that using a reference-based metric on erroneous references could result in inflated impressions of model performance.In particular, while Auto-UI large appears to outperform Auto-UI base according to the action matching metric, it is clearly inferior in terms of overall task success rate. Quantitatively, all three evaluators achieve a Kendall correlation of 100% with the human judges, while the action matching score only obtains 66.7%. This highlights a fundamental drawback in a single-step metric like action matching: it does not reflect error propagation or distribution shift in the sequential prediction process of an arbitrary policy, which can be captured by whole-trajectory success metrics.Measuring whole-trajectory success for the complex tasks that digital agents complete has typically required either human evaluation of individual trajectories, or manual creation of individual test cases, as in WebArena. We analyze the potential for automating this process using our three proposed evaluators. Table 1 shows the accuracy of each evaluator variant aggregated over trajectories from all four policies. 8 Overall, we find that our automated metrics correlate very strongly with human judgment: the Captioner + Mixtral variant shows the highest agreement with human judgment at 92.9% accuracy; replacing Mixtral with GPT-4 leads to a performance drop to 89.8%; and the end-to-end approach of GPT-4V Here, the oracle evaluator denotes performance using WebArena's built-in evaluation functions as the reward function; this provides an upperbound of improvement using Reflexion.achieves 90.6% accuracy. Again, the low accuracy with QWen-VL-chat confirms our modular approach design choice.Reflexion on WebArena We demonstrate how our proposed evaluation models can serve as a reward signal to guide an existing web agent at inference time, using the Reflexion technique (Shinn et al., 2023) as an example. In Reflexion, an agent first attempts a task, and an external evaluator is used to judge whether its attempt was successful or not. If it is judged as unsuccessful, the agent will be prompted to reflect on the failure and retry. We experiment with improving the current state-of-the-art GPT-4-based WebArena agent. 11Figure 3 includes the agent's baseline performance, and performance using up to three rounds of Reflexion with the oracle evaluator (which serves as an upper bound) and our three evaluation systems as external supervision. We see the improvement our evaluators provide scales favorable with evaluator capability, with Captioner + Mixtral improves agent's relative success rate by 16% and GPT-4V based evaluator by 29%. All system variants, including the low-cost and locally-hosted variant Captioner + Mixtral, significantly enhance agent's performance while requiring no access to hand-designed evaluation functions.Our preliminary study suggests that false negative evaluations have a more detrimental impact on agent's performance compared to false positives. If our evaluator predicts an execution is incorrect, but it was actually successful, this forces the agent to retry a successful execution, which nearly always leads a subsequent failure. In contrast, false positives only lose out on the opportunity to retry, which creates an upper bound of performance for the agent, but does not degrade its performance. Improving the robustness of inference-time algorithms under noisy supervision is an interesting future direction to explore.We demonstrate how our evaluator can guide the refinement of a policy using filtered behavior cloning (filtered BC), without additional supervision. We experiment on both iOS and Android environments.On iOS, we use CogAgent (?) as the policy model for the experiment. It is primarily instruction-tuned with demonstrations from web navigation and Android device control, and incorporates a very limited, manually collected iOS dataset for training. For data collection and testing purposes, we design 132 common tasks on iOS, with 80 tasks for training and 52 for testing. Given scaling limitations of emulation, including low speeds and restriction to emulation on macOS, we only experiment with iOS built-in apps and with the Captioner + Mixtral evaluator.We first sample 737 trajectories from CogAgent, conditioned on the 80 training tasks. We use our evaluator to provide per-step evaluations to these trajectories, then apply filtered BC for fine-tuning using this data. Unlike standard fine-tuning, this method filters out data points with rewards below a specified threshold. We set this threshold at ≥ p; i.e., we retain only state-action pairs that positively influence the success of a trajectory (Section 3). Additionally, we assess CogAgent's unmodified performance on iOS and explore a self-training approach by finetuning without data filtering as baselines for comparison. We randomly sample 20 successful and 30 erroneous evaluations for each evaluation model in WebArena and manually annotate the sources of failure. 12 We categorize errors into three primary types, providing percentage estimates rounded to the nearest 5%.1. Critical information lost from captions in the modular approach (10%); errors in screenshot understanding for the end-to-end GPT-4V approach (5%).2. Errors in the reasoning process, observed in 50% of cases for GPT-4V/GPT-4-based methods and 70% for Mixtral-Captioner.3. Ambiguities in task specification and success criteria, observed in 30% of cases for GPT-4V/GPT-4-based methods and 10% for Mixtral-Captioner.We note that in our error categorization, a model must overcome errors in preceding categories to be assessed under the subsequent one. Consequently, Mixtral-Captioner's lower rate of Type 3 errors is mostly attributed to its higher frequency of Type 1 and 2 errors.Additionally, we find the model provides the correct final evaluation, but incorrect reasoning, in about 10% of correct evaluations. 
conclusion: 
 In this study, we design automatic methods to both evaluate and refine the performance of digital agents. We first describe a model that provides either trajectory-level or per-step evaluation of agent's performance. Subsequently, we propose two approaches to implement the model: an end-to-end approach using a pre-trained vision-language model, and a modular caption-then-reason approach using a VLM and a pre-trained language model together. These methods offer trade-offs between performance, cost, and modularity.Using WebArena and Android-in-the-Wild as testbeds, we first validate the effectiveness of these evaluators against oracle evaluation metrics, and highlight their advantage over standard reference-based metrics on AitW. We then show how the evaluators can be used to refine existing agents through both inference-time guidance and filtered BC. When integrated as the reward function in Reflexion, a method for inference-time refinement, our evaluators enhance the best-performing agent's success rate by up to 29%. Additionally, it boosts the performance of a strong device control policy in a domain transfer task by 75% via filtered behavior cloning, all without any extra supervision. Our findings show the potential of model-based automated evaluators for both evaluating and improving digital agents, which is especially critical in developing real-world agents where ground truth evaluation functions or human supervision are not always available. 
 
 Appendix: While our research demonstrates the potential of model-based evaluators in evaluating and improving digital agents, we also identify several areas for future exploration. First, current evaluators are still far from perfect, and any enhancement in their performance, e..g, from better representations of the action space or stronger base models, will likely directly translate to improved outcomes. Second, in this work, we focused on Reflexion and filtered behavior cloning. Future works can explore scaling up the experiments and developing better training and inference-time algorithms that are robust and efficient under noisy supervision. Finally, in this work we only make use of the evaluator's binary or ternary judgment, and discard the language-based explanation it generates. Future work can explore how to leverage this information, for example, to further enhance policies through language supervision or to provide scalable oversight of agent behavior.Most currently available digital agents are research artifacts. As the performance of these agents improve and they are increasingly deployed in the real world, they may pose security risks to their users. For example, a web agent with unconstrained access to a browser might be able to gain access a user's passwords, financial information, or social media messages. Better understanding the potential failure modes of these models in real-world use cases is critical to ensuring their safe deployment. We view our work as a first step in this direction: by developing domain-general evaluators, we hope to facilitate better understanding of models (and their risks) outside of simulated environments like WebArena. At the same time, human evaluation and oversight of these future systems will also be important for mitigating potential harms; although our work in this paper focuses on autonomous evaluation, we hope it will supplement, rather than supplant, human efforts.In this section, we provide details about our experiments. Please refer to our code at https://github.com/Berkeley-NLP/Agent-Eval-Refine for the official reference.We use gpt-4-1106-vision-preview through the OpenAI API and feed the image without resizing in "high-resolution". We use a temperature of 0 and keep other parameters at default. The prompt templates for each environment are provided in Figures 12 and 13 .Collecting screenshots As described in Table 3 , we constructed our dataset primarily through random subsampling from source datasets. However, for the iOS domain, due to limited online resources, we manually capture 50 extra screenshots in-house.We represent actions as strings, e.g., Type ''Hello'' . This method leads to information loss when processing actions like clicks for pixel-based policies, as the coordinates [x, y] become meaningless when the image is represented by its textual description. We leave the task of more adequately transforming pixel-localized actions into textual forms for future work.Collecting screenshot descriptions After obtaining the screenshots, we query GPT-4V (specifically, gpt-4-1106-vision-preview through the API) to get dense caption demonstrations. We manually fix or filter out ones with apparent errors. We use a temperature of 0 and keep the other parameters at their default settings. The prompt template is provided in Figure 15 .Finetuning Qwen-VL captioner The prompt template to query the finetuned Qwen-VLchat captioner is provided in Figure 16 . We fine-tuned the model over 3 epochs with a batch size of 72 and adamw optimizer (Kingma & Ba, 2014), employing a cosine scheduler for learning rate adjustments starting from 1e-5, a weight decay of 0.1, and a warmup ratio of 0.01. As shown in the prompt template, during both finetuning and at inference time, we provide text recognition results from the EasyOCR engine as an additional input to the model to reduce hallucination. We provide randomly-sampled model output examples in Figure 14 .Quering the reasoner After obtaining the descriptions of the screenshots and actions, we query the LM, either Mixtral-8x7B-Instruct-v0.1 or gpt-4-turbo-preview. We use a temperature of 0 and keep the other parameters at their default settings. We provide the prompts to query the trajectory-level evaluator on Web and Android, and step-wise evaluator on iOS in Figures 17, 18 , and 19 respectively.We directly use the GPT4-0613 + CoT -v2 trajectories released by WebArena for evaluation. Confusion matrices for our evaluators' predictions compared to the oracle evaluator are shown in Figure 4 (left).Emulator We use Android Studio's built-in emulator to simulate a Pixel 4 with API version 33, and we develop a Python API for agent execution based on the appium package. We opt not to use AndroidEnv for Android emulation as it lacks support for typing actions.The 120 evaluation tasks are evenly and randomly sampled from the General, WebShopping, and GoogleApps subsets of the Android-in-the-Wild test set (40 each) as shown in Listing 1. Note that we have excluded the Install and Single subsets. The install tasks require credit card information and are not safe to evaluate, while single-step tasks fall outside our focus on trajectory-level tasks.Evaluation We use temperature of 0, for all policies during evaluation. The confusion matrices comparing our evaluators with human judgments are presented in Figure 4 (right).Reflexion We implement the Reflexion agent following the original paper (Shinn et al., 2023) . The algorithm involves three key components: an Actor, an Evaluator, and a Self-Reflection module. The Actor generates thoughts and actions in text based on the state observations it receives, where the actions are parsed into executable commands to transform the environment. The Evaluator assesses the quality of the outcome produced by the Actor.It computes a reward score based on how well the generated trajectory align with the goal.If the evaluator assesses the task to be failed, the Self-Reflection model will be evoked to generate verbal reflections, which is stored in the actor's memory, facilitating improved decision-making in subsequent trials.We use the DOM tree representation from the WebArena simulator as the environment observation. The LLM we use for Actor and the Self-Reflection is the GPT-4-preview-1106 model, and the prompts for these are shown in Listing 3 and 4 respectively. For the evaluator, we experiment with all the three variants proposed as well as the oracle evaluator from WebArena which is used for performance evaluation. Note that we use the webpage snapshot images instead of the DOM tree as the input to our evaluator.Emulator We use XCode's built-in emulator to emulate an iPhone 13 device running iOS 16 and develop a Python API for the agent based on the facebook/idb package. We align its action space with that of the Android-in-the-Wild schema. Notably, since swiping up on the homescreen in Android means opening the AppDrawer, which can roughly be translated to swiping left or right on the homescreen, we bridge this domain gap by translating swiping up to swiping left or right 50% of the time during data collection and to the right 100% of the time during evaluation.Tasks As shown in Listing 2, we design 132 task instructions that covers typical iOS device control tasks on Apple's official apps, where 80 are used for data collection and training the agent, and 52 are hold out for testing. While we aimed to minimize the distribution difference between our tasks and that from the AitW dataset, the inherent differences between the platforms necessitated the inclusion of iOS-specific instructions, such as "Disable Siri's access to Photos in Settings."We finetune the CogAgent model with its official code and apply LoRA (Hu et al., 2022) , setting the parameters as follows: lora rank to 50, employing a cosine learning rate schedule, with a warmup proportion of 0.2, a learning rate (lr) of 0.00001, a batch size of 4 and train for 3000 steps. We use a releatively high temparature of 1.5 and topk=100 during data collection to improve diversity and greedy decoding (temperature=0) for all policies during evaluation.We reuse the Android emulator as described in Appendix A.4. and sample the training and evaluation tasks from AitW General split's corresponding part. We use AutoUI-base as the policy. We set the temperate for 1 during sampling and uses a learning rate of 3e -3 and a batch size of 128 for the training.WebScreenshot (Dwyer, 2020) 128 Web Mind2Web (Deng et al., 2023) 429 Web AitW (train set) (Rawles et al., 2023) 596 Android GPT-4V in Wonderland (Yan et Thoughts: The user's intent was to star the top one most starred repo in GitLab, but the bot attempted to star a specific repository that was already starred. There is no indication that the repository in question is the top one most starred, and the bot did not perform any action to ensure it was interacting with the correct repository based on the user's intent. Therefore, the task is not completed successfully. Post "lgtm" for the merge request related to semantic HTML post in the a11yproject project.Thoughts: The image shows that the comment "lgtm" has been posted twice by the user "Byte Blaze," which indicates that the bot successfully posted the comment as intended. However, the bot stopped early due to the same typing action being repeated three times, which is likely a safeguard against spamming. Since the user's intent was to post "lgtm" and it is visible in the merge request comments, the task is successful despite the early stop. The image shows the commits page of the PyTorch-GAN project on GitLab, but it does not provide a summary or a list that ranks contributors by the number of commits. The bot stopped with a "Cannot determine" response, which indicates it could not find the specific information requested by the user. The bot should have navigated to the 'Contributors' section to find the information about who has made the most contributions. Therefore, the task is not completed successfully. Thoughts: The action history indicates that the agent performed a search for directions from "5600 Fifth Avenue, Pittsburgh, PA" to a Walmart in Pittsburgh. However, the final state of the webpage shows that the estimated travel time by car is actually "1 hour 43 minutes", which contradicts the bot's response to the user that stated a 16-minute travel time. This discrepancy suggests that the bot's response is incorrect, and the task of verifying if the Walmart can be reached within one hour by car has failed. The user's intent is to know the total cost of their latest cancelled order. From the action history, we see that the bot successfully navigated to "My Account" and "My Orders" pages. The final state of the webpage shows a list of orders with details. The latest cancelled order has a price of $231.54, which matches the bot's response. Therefore, it seems that the bot has successfully completed the task.Figure 9 : Mixtral failed to identify that the price is not from a cancelled order from a cluttered webpage. The error is corrected if GPT4/GPT4V is used.type [ChatGPT] at [Search GitLab] click [Convex Eggtart / Chatgpt] click [Fork] click [Fork project] stopResponse: Forking ChatGPT is in progress.Thoughts: The action history indicates that the agent has performed the necessary steps to initiate the forking of the ChatGPT repository on GitLab. The final state of the webpage shows that the agent is on a "Fork project" form, which is consistent with the forking process. The bot's response to the user confirms that the forking process has been initiated. There is no indication of an error or incomplete action, so we can assume the task was successful unless there is evidence to the contrary. Thoughts: The bot's response only includes a single order total, but the user's intent was to know the total amount spent on food-related shopping for the entire month of March 2023. The image shows multiple orders in March, and the bot should have summed up all completed and not canceled orders to provide an accurate total. Therefore, the task has failed.Figure 11 : Though our approach successfully identifies that the task has failed, the reasoning is incorrect. The error is caused by not filtering the orders for food-related only, instead of not summing up the total price.You are an expert in evaluating the performance of a web navigation agent. The agent is designed to help a human user navigate a website to complete a task. Given the user's intent, the agent's action history, the final state of the webpage, and the agent's response to the user, your goal is to decide whether the agent's execution is successful or not. There are three types of tasks: 1. Information seeking: The user wants to obtain certain information from the webpage, such as the information of a product, reviews, map info, comparison of map routes, etc. The bot's response must contain the information the user wants, or explicitly state that the information is not available. Otherwise, e.g. the bot encounters an exception and respond with the error content, the task is considered a failure. Besides, be careful about the sufficiency of the agent's actions. For example, when asked to list the top-searched items in a shop, the agent should order the items by the number of searches, and then return the top items. If the ordering action is missing, the task is likely to fail. 2. Site navigation: The user wants to navigate to a specific page. Carefully examine the bot's action history and the final state of the webpage to determine whether the bot successfully completes the task. No need to consider the bot's response. You are an expert in evaluating the performance of an android navigation agent. The agent is designed to help a human user navigate the device to complete a task. Given the user's intent, and the final state of the screen, your goal is to decide whether the agent has successfully completed the task or not. *IMPORTANT* Format your response into two lines as shown below: Thoughts: <your thoughts and reasoning process>" Status: "success" or "failure" You are an expert in evaluating the performance of a web navigation agent. The agent is designed to help a human user navigate a website to complete a task. Given the user's intent, the agent's action history, the final state of the webpage, and the agent's response to the user, your goal is to decide whether the agent's execution is successful or not.There are three types of tasks:1. Information seeking: The user wants to obtain certain information from the webpage, such as the information of a reviews, map info, comparison of map routes, etc. The bot's response must contain the information the user wants, or explicitly state that the information is not available. Otherwise, e.g. the bot encounters an exception and respond with the error content, the task is considered a failure. Besides, be careful about the sufficiency of the agent's actions. For example, when asked to list the top-searched items in a shop, the agent should order the items by the number of searches, and then return the top items. If the ordering action is missing, the task is likely to fail. 2. Site navigation: The user wants to navigate to a specific page. Carefully examine the bot's action history and the final state of the webpage to determine whether the bot successfully completes the task. No need to consider the bot's response. STATUS: FAILED REFLECTIONS FROM PREVIOUS ATTEMPTS: none""", "I find the review from Rachel, which is the answer to the objective. I issue the stop action with the → answer. However, the task is not completed successfully. This might be because I missed other reviews → that also mention about good fingerprint resistant. I think I should read all the reviews next time." ), ], "template": """OBJECTIVE: {objective} OBSERVATION AND ACTION HISTORY: {trajectory} STATUS: {status} REFLECTIONS FROM PREVIOUS ATTEMPTS: {memory}""", "meta_data": { "observation": "accessibility_tree", "action_type": "id_accessibility_tree", "keywords": ["objective", "trajectory", "status", "memory"], "prompt_constructor": "ReflectionGenerationPromptConstructor", "answer_phrase": "", "action_splitter": "```" }, }Prompt templates and additional details are provided in Appendix A.1.Data collection process, hyper-parameters, and output examples are detailed in Appendix A.2.Table 3 in Appendix A.2 contains details of data sources and sizes.https://github.com/JaidedAI/EasyOCRWe subsample from the original test set of 1.4k tasks to facilitate acquiring human judgments of trajectories. See Appendix A.4 for details on a list of evaluated tasks and details on task sampling.Details on our emulators are available in Appendices A.4 and A.6.The human demonstrations use the original AitW emulator, which was not released by the authors; thus, these results are not directly comparable with the automated policies, which use the emulator we implement. However, the focus of our experiments is not to directly compare policies, but to compare evaluators across a variety of policies, tasks, and domains.Figure 4 in Appendix A.3 includes the confusion matrices of these predictions.Experimental setup details for AitW are provided in Appendix A.4.Action matching scores are averaged across the subsets of AitW we sample from, as reported in ? and Zhang & Zhang (2023).Reflexion prompts are detailed in Appendix A.5.Refer to Figures 5 through 11 in the Appendix for visual representations of these evaluations. 


Decision: This paper shows that domain-general LLMs are sufficiently good reward models to be able to improve web agents. It has certain weaknesses. It is primarily an empirical paper, and the range and scale of the empirical evaluations seems limited compared to other domains (in part because of the difficulty of scaling evaluations in this domain). And the methods studied are similar to or the same as existing methods. Nevertheless, this is a good study and a well-written paper in an area of intense current interest. It makes real progress in showing that domain-general evaluators (LLMs or a good VLM, really only GPT-4V currently) can be quite effective for general tasks in the web agent domain, for either inference-time guidance or behavior cloning, obviating the need for hand-designed evaluation functions. The transfer learning extension to iOS, while smaller in scale than would be ideal, nevertheless seems a very useful contribution to have.

Reviews: review: paper_summary: paper_summary: This paper empirically studies model's automatic evaluation of trajectories on digital agents, such as agents operating on the web and Android. The authors focus on two methods to construct the evaluator, one is an end-to-end approach that directly uses a pretrained VLM, another is a modular approach that first transcribes the screenshot to captions with a VLM, and then feeds all text to a LM. Both methods demonstrate high accuracies against oracle evaluator and human judge.  The authors then utilize such automatic evaluators to improve the policy agent, through reflexion and filtered behavioral cloning respectively. In both settings, the evaluation is able to improve the tasks.
strength_weakness: strength_weakness: As an empirical paper, I expect the experiments to be conducted in a more comprehensive manner with more details, and in general the empirical results are relatively sparse.  Some examples:   
(1) while the authors talk about both trajectory-level and step-level evaluators, step-level evaluation is only applied for the iOS agent refinement, and there is no study how accurate such step-level evaluation is. From Table 2, I guess step-level evaluation may not be very accurate given small improvement over self-training;   
(2) end-to-end methods only evaluate GPT-4V, while I would like to see the performance of more open VLMs;   
(3) In Table 2, filtered BC only finishes 3 more tasks than self-training in a test of 52 tasks. I am not sure how significant this result is.
questions: questions: 1. The high accuracy in Table 1 is nice, but I am feeling this is because the special properties of the two environments – for example, the final screenshot is typically enough to classify whether it satisfies the goal. I wonder whether the authors think such high accuracies are general, and whether there are environments where the final state is difficult to classify, I would like to see more discussions on this aspect as potential limitations of the paper.  
2. Webarena by design is a text-only task originally, I see the authors are using GPT4-V on that, are the authors experimenting with VisualWebArena or am i missing sth?

```
After rebuttal, I bump up my score to 6 given the clarifications and added results to increase the significance of the previous results.
```
paper_summary: paper_summary: This paper designs two lines of automatic evaluation for digital agents: (1) Using VLMs to caption the trajectory and then prompting the LLM to reason if the trajectory is correct. (2) Using an end-to-end VLM to evaluate the trajectory (GPT-4V in the paper). These methods show a high correlation with ground truth evaluation metrics, and the paper demonstrates that the agents can be improved by using reflective prompting or filtered BC.
strength_weakness: strength_weakness: - Using LLMs to evaluate the trajectory of agent policies and improving the agent seems common in existing literature. For example, [1] also uses LLM evaluators to conduct filtered BC, and [2] uses reflection.
- The evaluation of filtered BC in the iOS domain is not very persuasive because Table 2 only shows the comparison of the number of successful tasks for different methods. It is unclear if solving 14 out of 52 tasks represents a real improvement or just randomness.

[1] Du et al., Vision-Language Models as Success Detectors.

[2] Wang et al., JARVIS-1: Open-world Multi-task Agents with Memory-Augmented Multimodal Language Models.
questions: questions: I think the methods in this paper are promising for digital agent development. But the weaknesses on originality and evaluation may limit the contribution of this paper. I would like to raise my assessment for the paper in the followup phase if the authors could well address the issues and make a rounded discussion.
paper_summary: paper_summary: This paper tackles an important problem of agentic research: how to automate the evaluation of domain-general agents? Current agent benchmarks rely heavily on hard-coded rules for evaluation, which is expensive to construct and hard to scale. This paper proposes a multimodal approach to automate this process. They first train a VLM-based Captioner to generate a textual description of the agent's observation (i.e., the screenshot). Then the LLM-based Reasoner will take this description along with the actions, and the original user
instruction to judge whether the execution is successful or not. 

Given such an automatic evaluator, they further show that it can be used to improve the performance of existing agents by providing inference-time guidance for self-refinement.
strength_weakness: strength_weakness: - The key to Captioner's success is the collection of 1,272 manually curated caption annotations, whose collection is slow and expensive. Is there any way to automate this process such that it can be scaled up?
- The proposed approach, despite being effective on WebArena and AitW, might not work well in domains where screenshots cannot capture the change of state. For instance, if an agent executes "mv file_a foler_b" in the terminal. The action will be executed completely in the backend and nothing will be shown unless both folders are explicitly open and displayed on the screen.
questions: questions: What would you perceive as the biggest challenge in automating agent evaluation?

Comments: comment: Thank you for the clarifications! The added results on AitW is nice and significant, I hereby increase my score.
Dear Reviewer zFha,

Thanks again for your helpful feedback! We will definitely incorporate all the discussions and new results into the paper, and improve the organization of the experiments.
The authors,

Thank you for your responses. My concerns are addressed and I would like to raise the score. It would be better if the discussion and the new results for the rebuttal phase can be included in the revised paper, and the empirical experiments can be more organized. 

Reviewer zFha