Title: DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: LLM/VLM Agent

Device Control

GUI Navigation

Web Agent

Reinforcement Learning
Abstract: Pre-trained vision language models (VLMs), though powerful, typically lack training on decision-centric data, rendering them sub-optimal for decision-making tasks such as in-the-wild device control through Graphical User Interfaces (GUIs) when used off-the-shelf. While training with static demonstrations has shown some promise, we show that such methods fall short when controlling real GUIs due to their failure to deal with real world stochasticity and dynamism not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline and offline-to-online RL. We first build a scalable and parallelizable Android learning environment equipped with a VLM-based general-purpose evaluator and then identify the key design choices for simple and effective RL in this domain. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our 1.5B VLM trained with RL achieves a 49.5\% absolute improvement -- from 17.7 to 67.2\% success rate -- over supervised fine-tuning with static human demonstration data. It is worth noting that such improvement is achieved without any additional supervision or demonstration data. These results significantly surpass not only the prior best agents, including AppAgent with GPT-4V (8.3\% success rate) and the 17B CogAgent trained with AitW data (14.4\%), but also our implementation of prior best autonomous RL approach based on filtered behavior cloning (57.8\%), thereby establishing a new state-of-the-art for digital agents for in-the-wild device control.

DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

Abstract

Training corpuses for vision language models (VLMs) typically lack sufficient amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal for decision-making tasks such as in-the-wild device control through graphical user interfaces (GUIs). While training with static demonstrations has shown some promise, we show that such methods fall short for controlling real GUIs due to their failure to deal with real world stochasticity and non-stationarity not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline RL to initialize the model, followed by offline-to-online RL. To do this, we build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator and develop a simple yet effective RL approach for learning in this domain. Our approach runs advantage-weighted RL with advantage estimators enhanced to account for stochasticity along with an automatic curriculum for deriving maximal learning signal. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement - from 17.7 to 67.2% success rate - over supervised fine-tuning with static human demonstration data. These results significantly surpass not only the prior best agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent trained with AitW data (38.5%), but also the prior best autonomous RL approach based on filtered behavior cloning (57.8%), thereby establishing a new state-of-the-art for digital agents for in-the-wild device control.

1 Introduction

Advances in vision-language models (VLMs), especially in regards to their remarkable commonsense, reasoning, and generalization abilities imply that realizing a fully autonomous digital AI assistant, that can simplify human life by automating day-to-day activities on computer devices via natural language interfaces, is no longer a distant aspiration [16, 45, 56]. An effective device-control AI assistant should be able to complete tasks in-the-wild through Graphical User Interfaces (GUIs) on digital devices: make travel plans; experiment with presentation designs; and operate a mobile device autonomously, all while running amidst stochasticity and distractors on the device, the Internet, and the tools it interacts with. However, enhanced reasoning or common-sense abilities do not directly transfer to intelligent assistant behavior: ultimately we want AI assistants to accomplish

tasks, exhibit rational behavior, and recover from their mistakes as opposed to simply producing a plausible completion to a given observation based on the data seen during pre-training. This implies that a mechanism to channel abilities from pre-training into a deployable AI 'agent' is lacking.

Even the strongest proprietary VLMs, such as GPT-4V [24] and Gemini 1.5 Pro [7] 2 , still struggle to produce the right actions when completing tasks on devices. While general-purpose vision-language abilities help these models still make meaningful abstract deductions about novel scenes when deployed, these deductions do not transfer to accurate reasoning for control [47, 45, 55, 44]. As a result, most prior work for building device agents construct complex wrappers around proprietary VLMs by combining them with prompting, search, or tool use [47, 44, 52, 51, 45]. While building prompting or retrieval wrappers to improve decision-making performance of existing VLMs enhances their performance in the short run, without updating the weights, the effectiveness of the resulting agent is inherently limited by the capabilities of the base model [49, 3]. For example, we found that off-the-shelf VLMs make reasoning failures that derail the agent (e.g., Figure 2 and Figure 17), as direct consequences of inability of the base model to reason with low-level device-control actions. A different solution is to fine-tune the model on demonstrations via imitation learning. However, the dynamic nature of the web and device means that models trained to mimic actions in stale data can result in sub-optimalilty as the eco-system changes [26]. Agents trained in this way struggle to recover from the agents' own mistakes [8, 12].

If we can instead build an interactive approach to train a VLM to directly adapt and learn from its own experience on the device and the Internet, that can be used to build a robust and reliable device-control agent, without needing wrappers on top of proprietary models. However, this learning-based approach must satisfy some desiderata. First, it must make use of online interaction data since static demonstration data would not be representative of the task when the model is deployed: for instance, even in the setting of web navigation alone, dynamic nature of in-the-wild websites means that the agent will frequently encounter website versions that differ significantly from the sce-

narios seen during training and will need to behave reliably despite changes in visual appearance and distractions. Second, learning on-the-fly means the approach must learn from multi-turn interaction data from the model itself, a large of chunk of which would consist of failures. Proper mechanisms must be designed to automatically pick out the correct actions while filtering the wrong ones.

To this end, our main contribution is a novel autonomous RL approach, DigiRL (i.e., RL for Digital Agents), for training device control agents, as shown in Figure 1. The resulting agent attains state-of-the-art performance on a number of Android device-control tasks. To train this agent, our approach operates in two phases: an initial offline RL phase to initialize the agent using existing data, followed by an offline-to-online RL phase, that further fine-tunes the model obtained from offline RL on online rollout data. Online RL training requires access to an environment that the agent can interact with and obtain reliable reward signals, all in a reasonable amount of wall-clock time. To do so, we build a scalable and parallelizable Android learning environment equipped with a robust VLM-based general-purpose evaluator [26] (average error rate 2.8% against human judgement) that supports running up to 64 real Android emulators at the same time to make online RL real-time. Then, to effectively learn autonomously, we develop an online RL approach that retains the simplicity of supervised learning, but incorporates several key deep RL insights to enable fast fine-tuning. Concretely, our approach is a variant of advantage-weighted regression (AWR) [28], equipped with: (i) an automatic curriculum that uses an instruction-level value function to order tasks so as to extract

DigiRL

AutoUI

GPT-4V

M

Click

Skipped...

Press Back

Click

Type “logit...

Got

Wrong

stuck

page

Ceoge

211

1

Scroll Up

Press Home

Click

Type “2 bedro...

Press Enter

nt

Got.

Got

stuck

stuck.

maximal learning signal, which is inspired by prioritized replay methods [11, 32, 23], and (ii) another step-level value function trained via effective cross-entropy loss [17, 5] to extract low-variance and less-biased learning signal amidst stochasticity and diverse tasks. This RL approach allows us to fine-tune VLMs on their own experience.

We evaluate our agent trained with DigiRL in carrying out diverse instructions from Android in the Wild dataset [31] on real Android device emulators and find that our agent can achieve a 28.7% improvement over the existing state-of-the-art agents (from 38.5% to 67.2% success rate) 18B CogAgent [9], and over 9% improvement over the prior best autonomous learning approach based on Filtered Behavior Cloning [18, 26]. The performance of our agent also significantly surpasses wrappers on top of state-of-the-art proprietary VLMs such as GPT-4V [24] and Gemini 1.5 Pro [7] (17.7% success rate), despite using a significantly smaller model (with 1.3B parameters). To our knowledge, this is the first work to successfully build an autonomous offline-to-online RL approach to enable state-of-the-art performance on device-control problems.

2 Related Work

Multi-modal digital agents. In contrast to language-only agents that largely interact with both text or code inputs and outputs [33, 49, 3, 30, 46, 20, 13], training multi-modal agents capable of controlling devices presents different challenges: first, device control is done directly at the pixellevel and in a coordinate-based action space, instead of natural language [31, 44] that LLM is most familiar with, and second, the ecosystem of a device and the Internet tends to be quite stochastic and unpredictable, which is absent with high-level planning in language only. To handle these challenges, prior work largely builds on strong proprietary VLMs [24, 7], and designs complex rule-based wrappers [47, 51, 45, 52] to enhance the visual grounding capabilities of VLMs in GUI interfaces and convert text output into pixel interactions. However, without any form of fine-tuning, this limits the room for possible performance improvement [44, 47, 49, 3, 50], especially when pre-training corpora only present limited action-labeled data. A separate line of work fine-tunes VLMs with demonstration data [19, 15, 9, 53] via imitation learning, but maximizing single-step accuracy from stale demonstrations without accounting for consequences of these actions in subsequent steps may lead to poor solutions amidst stochasticity [26], as agents trained in such ways will struggle to recover from out-of-distribution states not included in the demonstration data [8, 12]. The third category, and perhaps the closest to us, are works that run filtered imitation learning on autonomously-collected data to directly maximize the episode success rate [26, 18]. In contrast, ours is the first work to scale autonomous, offline-to-online RL for device control, producing an agent that outperforms prior agents built via imitation. Even when compared to prior work running on-policy RL in simplified web navigation settings (MiniWob++ [37, 10]), our approach is 1000x more sample efficient (around 1e3 trajectories compared to around 1e6 trajectories), and operates in real-world GUI navigation tasks.

Environments for device control agents. Recent works have introduced simulated environments for building device control agents [48, 56, 16, 54, 4, 44]. However, these environments are primarily designed for evaluation, and present only a limited range of tasks within fully deterministic and

actionspace

dynamics

T

click

slide type home back

enter

real-world

agent

open-ended

environment

model

evaluator

COSTCO

51,499.99 A/ter 5300 GFF

HIN

点

P

COSTCO

King Matfess

INDEENENT

ESEQUBD

SOMOS

&

ebay

三

Human?

Are you a human?

We apotegiorfor the canfl

bxut we-cant suitr tellit yos/y

persan er asoript.

Rease dontsae ts peros

Bots-ane soriues san lt'very

Guyana condemns Venezuela

reerendumageewing

inneation cf

ted region

HM99

1850.00

fyex/re intereted in accmsing

admiLotausi

We wouldlive tsheer your

7180

non-stationarywebsite

pop-up

ads

unpredictableorder

load

identity

→

百

onen-en

stationary settings, infeasible for acquiring a diverse repertoire of skills needed for device control. Alternatively, other works use environments with a greater diversity of tasks [48, 37], but these environments often oversimplify the task complexity, thus failing to transfer to in-the-wild settings. Coversely, our training environment utilizes autonomous evaluation [26] with Gemini 1.5 Pro [7] to support diverse, open-ended tasks on parallel actual Android devices, at full scale unlike prior environments. This also contrasts other prior works that use single-threaded Android emulators [26, 39, 19] and thus inefficient for support online RL at scale.

Reinforcement learning for LLM/VLMs. The majority of prior research employing RL for foundation models concentrates on tasks that must be solved in a single turn, such as preference optimization [25, 58, 2] or reasoning [27]. However, optimizing for single-turn interaction from expert demonstrations may result in sub-optimal strategies for multi-step problems [57, 38, 42], especially amidst a high degree of stochasticity or non-stationarity. Therefore, we focus on building multi-turn RL algorithms that can learn from sub-optimal, online interaction data in this work. While prior works have developed value-based RL algorithms for LLMs [42, 38, 1, 57, 50], they typically require maintaining multiple models such as Q-networks, value-networks, and policy networks, along with their delayed target counterparts, and can be subjective to slow convergence and sensitivity to choices of hyper-parameters. In contrast, we focus on identifying the key design choices for instantiating a simple yet effective RL algorithm for practitioners to incorporate to substantially improve full-scale Android device control. Our approach can serve as a base model for future research.

3 Problem Setup and Preliminaries

Problem formulation. We are interested in pixel-based interaction with virtual devices. We scope our study in the control of Android devices: this is already significantly more challenging and more general than previous learning-based environments that focus solely on web navigation [16, 56, 4], where the web browser itself is merely one application within our broader environment, and link-based device controls [47, 51] are inadequate for tasks like games that do not support link inputs.

Each episode begins with the emulator initialized to the home screen. Subsequently, a task is selected from a predefined set of language instructions, some examples of which are shown in Appendix A.1. An agent is then tasked with manipulating the emulator to fulfill this instruction. At each time step, the agent receives a screenshot of the current screen as the observation. Following the action space in prior literature [31], the available actions include tapping and sliding based on normalized ( x, y ) coordinates (ranging from 0 to 1 relative to the screen dimensions), typing text strings of variable length, and pressing special buttons such as HOME, BACK, and ENTER, as illustrated in Figure 3. Our train and test instructions comes from General and Web Shopping subsets in AitW [31]. These tasks consist of information-gathering tasks like 'What's on the menu of In-n-Out?', and shopping tasks on the web like 'Go to newegg.com, search for razer kraken, and select the first entry'.

Challenges of stochasticity. Real-world device contrl presents unique challenges of stochasticity absent in simulated environments [56, 37] such as: (1) the non-stationarity of websites and applications, which undergo frequent updates, causing the online observations to be different from stale offline data, (2) various unpredictable distractors such as pop-up advertisements, login requests, and the stochastic order of search results. (3) technical challenges and glitches such as incomplete webpage loading or temporary access restrictions to certain sites. Examples of scenarios with such stochasticity from

our experiments are shown in Figure 3. We observe that these stochastic elements pose significant challenges for pre-trained VLMs, including even those fine-tuned on device control data. As a concrete example, Figure 4 shows an experiment result that illustrates the necessity of continuously adapting the models to the non-stationarity of websites and applications. After obtaining a good checkpoint using our approach (DigiRL), that we will introduce in the next section, with autonomous data from June.1 to June.3, we compare the performance of a frozen policy and a continuously updating policy using fresh autonomous data from June.7 to June.11. We find that indeed the the performance of the frozen policy gradually degrades over time due to the changes on websites and applications, while continuous online updates plays a key role in preventing this degradation.

Setup for reliable and scalable online RL. As autonomous RL interleaves data collection and training, to maximize learning amidst stochasticity, it is crucial to have a real-time data collection pipeline to collect enough experience for gradient updates. While this is not possible in single-thread Android emulator environments [26, 39] due to latency, we parallelize our Android emulator using appropriate error handling as discussed in Appendix A.1. In addition, the environment must provide a reward signal by judging whether the current observation indicates the agent has successfully completed the task. To generalize our evaluator to support a wide range of tasks, we extend Pan et al. [26]'s end-to-end autonomous evaluator that does not require accessing the internal states of the emulator or human-written rules for each task. This contrasts previous works that manually write execution functions to verify the functional com-

June 1

June 3

June 7

June 11

Walltime

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

0.75

Success Rate

Learning (Online)

Frozen (Online)

Learning (Online)

pleteness of each task [16, 48, 37, 44]. We adopt Gemini 1.5 Pro [6, 7] as the backbone of the autonomous evaluator. We seed this model with few-shot rollouts and the associated human-labeled success indicators to guide evaluation of novel queries. This pipeline enables a single evaluator that can evaluate all AiTW tasks. The evaluator is highly aligned with human annotations (average error rate 2.8%), validated in Figure 8.

4 DigiRL: Autonomous RL for Building a Strong Device-Control Agent

We now present our autonomous RL framework for training device agents. We pose the device control problem as a Markov decision process (MDP) and develop RL methods for this MDP. The core of our approach is based on a simple and scalable off-policy RL method, advantage-weighted regression (AWR) [29], but we make crucial modifications to handle stochasticity and highly-variable task difficulty, through the use of value functions trained with appropriate losses, and an automatic curriculum, induced by an instruction-level value function to maximize learning.

Device control and GUI navigation as a MDP. We conceptualize device control guided by natural language instructions as a finite horizon Markov Decision Process (MDP) represented by M = {S , A , T , µ 0 , R , H } and run policy gradient to solve this MDP. At the beginning, an initial state s 0 and a natural language instruction c are sampled from the initial state distribution µ 0 . A reward of 1 is given at the end if the agent successfully fulfills the task per the evaluator, otherwise a reward of 0 is given. The trajectory terminates either when the agent accomplishes the task or when the maximum allowed number of interactions H is exceeded. States are represented using the last two screenshots. To explain our approach in detail, we also include several standard definitions used in reinforcement learning (RL). The Q function for a policy π represents the expected longterm return from taking a specific action at the current step and then following policy π thereafter: Q π ( s h , a h , c ) = E π [ ∑ H t = h r ( s t , a t , c ) ] . The value function V π ( s h , c ) is calculated by averaging the Q-value, Q π ( s h , a h , c ) , over actions a h drawn from the policy π . The advantage A π ( s h , a h , c ) for a state-action pair is computed by subtracting the state's value under the policy from its Q-value: A π ( s h , a h , c ) = Q π ( s h , a h , c ) -V π ( s h , c ) .

4.1 Backbone of Our Approach: Off-Policy RL via Advantage-Weighted Regression

The starting point we choose to build our approach on is the advantage-weighted regression (AWR) algorithm [29], which says that we can improve the policy reliably by regressing the policy towards exponentiated advantages induced by the reward function, as a proxy for optimizing the policy gradient while staying close to the previous policy [14, 35, 34]:

for some positive parameter β and the distribution of past experience ν , and A ( s, a, c ) denotes the advantage of a state-action pair ( s, a ) given a context c . To avoid tuning the hyperparameter β , we consider an alternative that does 'hard filtering' on the advantages instead of computing exp( A ) , similar to prior works [22, 43]. This leads to the following loss function for fine-tuning the model:

Typically, these advantages are computed by running Monte-Carlo (MC) rollouts in the environment to estimate the value of a given state-action pair, and subtracting from it an estimate of the value of the state given by a learned value estimator alone. However, this approach is likely to produce high-variance advantages given the stochasticity of the device eco-system that affects MC rollouts.

4.2 Obtaining Reliable Advantage Estimates from Doubly-Robust Estimators

To reliably identify advantageous actions given significant environment stochasticity, we construct a per-step advantage estimator, inspired by doubly-robust estimators [40, 36]:

where λ is a weighting hyper-parameter. This construction of the advantage estimator is a simplified version of Generalized Advantage Estimation (GAE) [36] using only the next-step advantage estimator and final-step advantage estimator as there are no intermediate rewards in our problem. This construction balances an advantage estimator with higher variance Monte-Carlo estimates λ H -h r ( s H , a H , c ) (due to stochasticity) and an estimator with higher bias V step ( s h +1 , c ) + r ( s h , a h , c ) -V step ( s h , c ) (due to imperfect fitting of the value function). We observed that combining both high-variance and high-bias estimators gave us a sweet-spot in terms of performance. To implement the step-level hard filtering, we simply threshold this doubly robust estimator as A step ( s h , a h , c ) > 1 /H to decide which actions progress towards the goal.

4.3 Automatic Curriculum using an Instruction-Level Value Function

While the AWR update (Equation 4.1) coupled with a robust advantage estimator (Equation 4.3) is likely sufficient on standard RL tasks, we did not find it to be effective enough for device control in preliminary experiments. Often this was the case because the task set presents tasks with highlyvariable difficulties that collecting more data on tasks that the agent was already proficient at affected sample efficieny negatively. In contrast, maximal learning signal can be derived by experiencing the most informative tasks for the agent during training. To this end, we design an instruction-level value function V instruct ( c ) to evaluate if a given rollout can provide an effective learning signal:

where ∑ H t = h r ( s t , a t , c ) is a Monte-Carlo estimator of Q ( s h , a h , c ) . The equality holds because the MDP formulation only provides rewards at the end of a rollout. Intuitively, if a rollout attains a high value of A instruct ( s h , a h , c ) , it means the value function V instruct is small. Therefore, this rollout represents a valuable experience of the agent accomplishing a difficult task, and thus should be prioritized, akin to ideas pertaining to prioritized experience [32] or level replay [11]. When training the actor with a buffer of historical off-policy data, we first perform a filtering step to identify the topp datapoints with highest A instruct ( s h , a h , c ) . Then, we use it for AWR (Equation 4.1) with the doubly-robust advantage estimator (Equation 4.3).

Implementation details. Inspired by the findings in some recent works [5, 17] that modern deep learning architectures like transformers [41] are better trained with cross-entropy losses instead of mean-squared losses, we utilize a cross-entropy objective based on the Monte-Carlo estimate of the trajectory reward for training both of our value functions:

Final algorithm. The final practical algorithm is shown in Figure 5. The instruction-level value function estimates the values of the trajectories, which is trained with loss shown in Equation (4.5). The step-level value function estimates the values of states, which is trained with loss shown in Equation (4.6). When training the actor, we first filter out trajectories and states using the value functions as shown in Equation (4.4) and Equation (4.3), then train the actor with the MLE loss shown in Equation (4.2) on the filtered data.

5 Experimental Evaluation

The goal of our experiments is to evaluate the performance of DigiRL on challenging Android device control problems. Specifically, we are interested in understanding if DigiRL can produce agents that can effectively learn from autonomous interaction, while still being able to utilize offline data for learning. To this end, we perform a comparative analysis of DigiRL against several prior approaches, including state-of-the-art agents in Section 5.1. We also perform several ablation experiments to understand the necessity and sufficiency of various components of our approach in Section 5.2.

Baselines and comparisons. We compare DigiRL with: (a) state-of-the-art agents built around proprietary VLMs, with the use of several prompting and retrieval-style techniques; (b) running imitation learning on static human demonstrations with the same instruction distribution, and (c) a filtered BC approach [26]. For proprietary VLMs, we evaluate GPT-4V [24] and Gemini 1.5 Pro [7] both zero-shot and when augmented with carefully-designed prompts. For the zero-shot setting, we use the prompt from Yang et al. [47] and augment the observation with Set-of-Marks [55]. Set-ofMarks overlays a number for each interactable element over the screenshot, so that a VLM can directly output the number of the element to interact with in plain text instead of attempting to calculate pixel coordinates, which is typically significantly harder. We also compare with AppAgent [47], which first prompts the VLM to explore the environment, and appends the experience collected to the test-time prompt. We also compare with two state-of-the-art fine-tuning methods for Android device control: AutoUI (specifically AutoUI-Base [53]) and CogAgent [9]. AutoUI-Base uses an LM with 200M parameters, and a a vision encoder with 1.1B parameters. CogAgent has 11B parameters for its vision encoder and 7B for its LM. The supervised training corpus for both AutoUI-Base and CogAgent contains AitW, including the instruction set and the emulator configuration we use.

Base VLM and offline dataset. Both Filtered BC and DigiRL use trained AutoUI-Base checkpoints with the image encoder frozen. The instruction and step-level value functions for DigiRL employ this same frozen image encoder. The visual features output from the encoder are concatenated with instruction features derived from RoBERTa [21]. A two-layer MLP is then used to predict the value function. In the offline phase, the offline dataset is collected by rolling out the initial AutoUI-Base supervised trained checkpoint as policy. For fair comparisons, we keep the number of offline data collected in the pure offline training roughly the same as the total number of data collected in the offline-to-online training. Due to the dynamic nature of the Internet-device eco-system, our offline data was stale by the time we were able to run our offline-to-online experiments, and this presented additional challenge in offline-to-online learning. In both General and Web Shopping subsets, offline experiments make use of around 1500 trajectories while offline-to-online experiments start with

0

320

640

960

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Success Rate

0

320

640

960

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Filtered BC-1

Filtered BC-2

DigiRL-1

DigiRL-2

GPT-4V

around 500 offline trajectories and update with another 1000 online trajectories. In the offline phase, DigiRL skips instruction-level filtering and instead trains the actor with all successful trajectories to make full use of the offline data. See a detailed breakdown of our dataset in Appendix A.1.

5.1 Main Results

Our main results are summarized in Table 1 and Figure 6. We find that on both AitW General and AitW Web Shopping subsets, the agent trained via DigiRL significantly outperforms prior state-of-the-art methods based on prompting and retrieval (AppAgent + GPT-4V/Gemini 1.5 Pro) or training on static demonstrations (CogAgent and AutoUI), by a large margin with more than 49.5% absolute improvement (from 17.7% to 71.9% on the General subset and from 17.7% to 67.2% on the Web Shopping subset). Notably, this improvement from DigiRL is realized fully autonomously without making use of human supervision (e.g. manually labeled rollouts or hand-written verifiers).

Are inference-time prompting and retrieval techniques or supervised training enough for device control? Delving into Table 1, we observe that off-the-shelf proprietary VLMs, even when supplemented with the set-of-marks mechanism, do not attain satisfactory performance: both GPT-4V and Gemini 1.5 Pro achieve success rates under 20%. One possible cause could be the underrepresentation of Android device data in the pre-training data. Moreover, inference-time adaptation strategies such as AppAgent [47] show minimal improvement, with gains not exceeding 5% for either model. All this evidence suggests a limited scope for improvement without fine-tuning of some sort. As illustrated in Figure 7, the primary failures of these VLMs stem from hallucinatory reasoning that lead the VLMs to land on a relevant but wrong page. This suggests that while state-of-the-art VLMs excel at reasoning problems in code and math, their reliability in less-familiar domains, such

Fail to recover from mistakes

Get stuck midway

Arrive at wrong goal

Failure Mode

0.0

0.2

0.4

% in All Trajectories

General

Fail to recover from mistakes

Get stuck midway

Arrive at wrong goal

Failure Mode

0.0

0.2

0.4

% in All Trajectories

Web Shopping

Set-Of-Marks

GPT4V

Set-Of-Marks

Gemini-1.5-Pro

AppAgent

GPT4V

AppAgent

Gemini-1.5-Pro

AutoUI

CogAgent

Filtered BC

Offline

DigiRL

Offline

Filtered BC

Online

DigiRL

Online

as device control, remains inadequate. For example, for the instruction 'Go to newegg.com, search for alienware area 51, and select the first entry', a GPT-4V based agent erroneously searched 'alien area 51 ebay' in Google.com and decided that it had made progress towards the task (Figure 17).

Training on domain-specific human demonstrations, however, does boost performance, allowing the smaller, specialized VLM, AutoUI with 1.5 billion parameters, to match or surpass the larger, generalist VLMs like GPT-4V and Gemini 1.5 Pro. Nonetheless, this supervised imitation learning approach still fall short, with success rates on both subsets remaining below 20%. This shortcoming is not fundamentally addressed via enhancements in model scale or architecture, as evidenced by CogAgent [9], with 18 billion parameters still achieving performances below 40% success rate. As depicted in Figure 7, a predominant failure mode for these agents is an inability to rectify their own errors. An example trajectory that we observed is that for the instruction 'what's on the menu of In-n-Out', the agent accidentally activated the voice input button, and failed to quit that page until the step limit. In contrast, DigiRL is able to recover from the errors more efficiently( Appendix C.2).

Comparison of different RL approaches. In Table 1 and Figure 6, we present a comparative analysis of various autonomous approaches. Notably, both offline and offline-to-online configurations demonstrate that our RL approach, when augmented with a continuous stream of autonomous interaction data and reward feedback, substantially improves performance. This improvement is evident from an increase in the success rate from under 20% to over 40%, as the agent learns to adapt to stochastic and non-stationary device interfaces. Moreover, although the total sample sizes for offline and offline-to-online settings are equivalent, the top-performing offline-to-online algorithm markedly surpasses its offline counterpart (75% versus 62.8% on the General subset). This highlights the efficacy of autonomous environment interaction, and establishes the efficacy of DigiRL in learning from such uncurated, sub-optimal data. Lastly, DigiRL consistently outperforms the state-of-the-art alternative, Filtered BC, across both the General and Web Shopping subsets, improving from 61.5% to 71.9% and 57.8% to 61.4%, respectively, highlighting DigiRL's performance and efficiency.

5.2 Analysis and Ablations

Failure modes analysis. We conduct an additional user study to annotate the failure modes for each agent as shown in Figure 7, and a more fine-grained breakdown can be found in Appendix D. At a high level, we classify the major failure modes of all agents into the following three categories: (1) Failure to recover from mistakes refers to the scenario where the agent made a mistake that led it to states from which it failed to quickly recover and resume the task, such as a wrong search page. (2) Getting stuck midway refers to the failure mode where the agent gets distracted on the right track to completing the instruction and as a result fails to accomplish the task. For example, failing to click on the right link or failing to search after typing the key words. (3) Arriving at wrong goal refers to the failure mode where the agent arrives at a wrong page and mistakenly thinks that it had completed the task. For e.g, the agent finds a macbook on costco.com instead of finding a macbook on ebay.com.

While all the types of failure modes benefit from offline and offline-to-online RL training as shown in Figure 7, the most consistent and significant reduction is probably for the failure mode of failing to recover from mistakes. This is because while pre-trained models, generating plausible future tokens, can get distracted by the dynamic nature of the environment and, as a result, encounter at never-before-seen states. With no clue of how to escape such states, these methods are unable to recover and fail to solve the task. In contrast, by training on autonomously-collected rollouts, our agent DigiRL is able to learn from its own mistakes and reduces failures to recover over training.

General

Policy Model

Set-Of-Marks

GPT4V

Set-of-Marks

Gemini-1.5-Pro

AppAgent

GPT4V

AppAgent

Gemini-1.5-Pro

AutoUI

CogAgent

Filtered BC

Offline

DigiRL

Offline

Filtered BC

Online

DigiRL

Online

Policy Model

0

50

% Success Rate

17.7

13.5

16.7

16.7

15.6

17.7

18.8

16.7

12.5

14.6

25.0

25.0

55.2

53.1

56.3

63.5

59.4

61.5

70.0

72.9

Human

Gemini-1.5-Pro Evaluator

Set-Of-Marks

GPT4V

Set-Of-Marks

Gemini-1.5-Pro

AppAgent

GPT4V

AppAgent

Gemini-1.5-Pro

AutoUI

CogAgent

Filtered BC

Offline

DigiRL

Offline

Filtered BC

Online

DigiRL

Online

0

50

% Success Rate

11.4

8.3

15.6

11.5

13.5

8.3

13.5

5.2

18.8

17.7

42.6

38.5

45.8

46.7

57.3

55.2

61.5

60.4

68.8

71.9

Web Shopping

Human

Gemini-1.5-Pro Evaluator

Ablation study of each component in DigiRL. Weconduct an ablation study on different components of DigiRL in Figure 9. We find that all the components used by our approach are necessary: (1) using cross-entropy for training the value functions boosts performance by around 12% (compare Ours and Ours w/ Regression); (2) using step-level advantages improves efficiency by 12% (comparing Ours and Ours w/o step-level advantage); (3) the use of automatic curriculum improves the speed of learning by around 25% (comparing Ours w/o step-level advantage and Filtered BC); (4) Ours outperforms vanilla AWR that does not employ a doubly-robust advantage estimator or curriculum.

Additionally, we also observe no degradation in performance as a result of 'hard-filtering', as show by nearly comparable performance of our approach and the best run of exponential filtering obtained via an extensive tuning of the temperature hyperparameter τ in naïve AWR (comparing Ours and Ours w/ vanilla AWR reweighting), despite simplicity of implementation in the hard filtering approach. Putting together, these choices result in a new state-of-the-art RL approach for device control.

Evaluation of our autonomous evaluator. In Figure 8, we present the findings from a user study aimed at assessing the accuracy of our autonomous evaluator. Our results indicate that

0

100

200

300

400

500

600

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

Success Rate

Ours

Ours w/ regression

Ours w/o step-level advantage

Vanilla AWR

Ours w/ AWR reweighting

Filtered BC

the success rates reported by our automatic evaluator are remarkably consistent with those assessed by human evaluators across almost all models, with differences less than 3%. Furthermore, we observed that evaluations on the Web Shopping subset are more precise compared to those on the General subset. This increased accuracy likely stems from the fact that tasks in the General subset are formulated in free-form language, which can introduce ambiguity, whereas the Web Shopping subset features a narrower range of language expressions, reducing potential variability.

6 Discussion and Limitations

In this paper, we propose a novel autonomous RL approach, DigiRL, for training in-the-wild, multimodal, device-control agents that establish a new state-of-the-art performance on a number of Android control tasks from Android-in-the-Wild dataset [31]. To achieve this, we first build a scalable and parallelizable Android environment with a robust VLM-based general-purpose evaluator that supports fast online data collection. We then develop a system for offline RL pre-training, followed by autonomous RL fine-tuning to learn via interaction, admist the stochasticity of the real-world Internet and device eco-system. Our agent achieves a 280% improvement over the previous state-of-the-art agents (from 17.7% to 68.2% in terms of task success rate), including AppAgent based on GPT-4V and Gemini 1.5 Pro, and supervised trained models such as AutoUI and CogAgent.

Due to computational limitations, and despite the fact that the parallel emulator and autonomous evaluator can be easily extended to complicated tasks, our agent is trained only with tasks from AitW instead of a all possible tasks on the device. Our design of the DigiRL algorithm aims for maximal implementation simplicity, so we hope that our approach to serve as a base algorithm for future research to build on, including algorithmic research as well as expanding the space of tasks.

References

Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023.

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback, 2023.

Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. ArXiv , abs/2310.05915, 2023. URL https: //api.semanticscholar.org/CorpusID:263829338 .

Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024.

Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep rl, 2024.

2023 Gemini Team. Gemini: A family of highly capable multimodal models, 2024.

2024 Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.

Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability. NeurIPS , 2021.

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023.

Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers, 2022.

Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. CoRR , abs/2010.03934, 2020. URL https://arxiv.org/abs/2010.03934 .

Yiding Jiang, J Zico Kolter, and Roberta Raileanu. On the importance of exploration for generalization in reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024.

Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning , 2002. URL https://api. semanticscholar.org/CorpusID:31442909 .

Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web, 2024.

Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649 , 2024.

Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes, 2023.

Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: Bootstrap and reinforce a large language model-based web navigating agent, 2024.

Juyong Lee, Taywon Min, Minyong An, Changyeon Kim, and Kimin Lee. Benchmarking mobile device control agents across diverse configurations, 2024.

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents, 2023.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR , abs/1907.11692, 2019. URL http://arxiv.org/abs/1907. 11692 .

Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR , abs/2006.09359, 2020. URL https: //arxiv.org/abs/2006.09359 .

OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand, 2019.

2023 OpenAI Team. Gpt-4 technical report, 2023.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv , abs/2203.02155, 2022. URL https://api.semanticscholar.org/ CorpusID:246426909 .

Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.06474 , 2024.

Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. CoRR , abs/1910.00177, 2019. URL http://arxiv.org/abs/1910.00177 .

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.

Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. arXiv preprint arXiv:2307.10088 , 2023.

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, 2016.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023.

John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR , abs/1502.05477, 2015. URL http://arxiv.org/abs/ 1502.05477 .

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017. URL http://arxiv.org/abs/ 1707.06347 .

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation, 2018.

Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 3135-3144. PMLR, 06-11 Aug 2017. URL https://proceedings.mlr.press/v70/shi17a.html .

Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning, 2023.

Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv preprint arXiv:2105.13231 , 2021.

Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. CoRR , abs/1509.06461, 2015. URL http://arxiv.org/abs/1509.06461 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.

Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning, 2022.

Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression, 2021.

Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972 , 2024.

An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation, 2023.

John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback, 2023.

Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771 , 2023.

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023.

Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for llms, 2023.

Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. arXiv preprint arXiv:2405.10292 , 2024.

Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction. arXiv preprint arXiv:2402.07939 , 2024.

Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for gui agents, 2024.

Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents, 2023.

Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. Mmina: Benchmarking multihop multimodal internet agents. arXiv preprint arXiv:2404.09992 , 2024.

Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded, 2024.

Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. ArXiv , abs/2307.13854, 2023. URL https: //api.semanticscholar.org/CorpusID:260164780 .

Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446 , 2024.

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR , abs/1909.08593, 2019. URL http://arxiv.org/abs/1909.08593 .

Appendices

A Environment details

A.1 Post-processing of AitW

The Android in the Wild (AiTW) task set is a large-scale dataset for android device control, containing five subsets: GoogleApps, Install, Web Shopping, General, and Single, where we select the General and Web Shopping subsets. Single subset is not considered here because all tasks in Single can be completed within one step and thus this subset fails to examine the multi-step challenges that we are interested in this paper. Install and GoogleApps are not considered due to security reasons as those tasks require an active Google account and parallel emulations can flag security concerns.

General. The General set focuses on searching for information and basic application usage. For example, it contains searching for latest news in Chile, search for flights from NYC to Sydney, opening Gmail, etc. We use all 545 tasks in the training set for training and the first 96 tasks in the test set for testing due to computational and budget constraints. The maximum allowed number of steps for this subset is 10. Offline data is collected by rolling our the initial AutoUI policy with tasks from the training set. The offline data used for the offline-to-online setting contains 608 trajectories while the offline data used for the offline setting contains 1552 trajectories. Some task examples are shown in Table 3.

Web Shopping. The Web Shopping subset comprises search instructions on various shopping websites, like searching for razer blader on ebay. As some websites (e.g. Amazon) and operations (e.g. adding items to cart) frequently require captcha verifications, we post-process the Web Shopping subset to exclude such operations and websites and also make the task easy to evaluate for our autonomous evaluator. The resulting task set involves navigating through five websites (costco.com, bestbuy.com, target.com, walmart.com, newegg.com) and three basic operations (go to website, search in the website, and select items from the searched results). Our post-processed training set contains 438 tasks and our testing set contains 96 tasks. Example tasks after post-processing can be found in Table 3. The maximum allowed number of steps for this subset is 20. Offline data is collected by rolling our the initial AutoUI policy with tasks from the training set. The offline data used for the offline-to-online setting contains 528 trajectories while the offline data used for the offline setting contains 1296 trajectories.

B Other Quantitative Experiments

B.1 Curriculum Learning

When running experiments on the AitW Web Shopping subset, we find solving easier tasks helps improve solving harder tasks, where the difficulty is identified in Table 3. By specifying the difficulty DigiRL-Run1 in Figure 6, we empirically show the success rates of each difficulty across the online learning process in Figure 12, we observe that a significant increase of success rate of tasks of difficulty 1 leads to increasing success rate of difficulty 2, and the same pattern for difficulty 2 and 3, demonstrating effective curriculum learning.

0

200

400

600

800

0.0

0.2

0.4

0.6

0.8

1.0

Difficulty 1

Difficulty 2

Difficulty 3

0

200

400

600

800

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Success Rate

Filtered BC-20

Filtered BC-10

DigiRL-20

DigiRL-10

GPT-4V

B.2 Learning Method

We ablate on the learning method, i.e. online learning or offline-to-online learning. We find that offline-to-online learning converges faster than online learning, and is not necessarily worse than online learning in terms of final performance, as shown in Figure 11.

B.3 Horizon Limit

We investigate the horizon limit of filtered BC and DigiRL on the AitW General subset. As most tasks can be effectively solved within 10 steps, we specify two horizon limits: a sufficient horizont H = 10 , and a redundant horizon H = 20 . Results in Figure 12 show that a redundant horizon introduces significantly faster learning speed for both filtered BC and DigiRL, presumbaly because longer horizon means more opportunity to try in a single trajectory. In both horizon settings, we observe the DigiRL offers a significant speedup of around 100 trajectories over Filtered BC.

B.4 Trajectory Length

We investigate the rollout length of DigiRL compared to filtered BC. Results in Table 4 demonstrate that DigiRL consistently achieves shorter average rollout lengths compared to filtered BC across both

0

200

400

600

800

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

0.65

0.70

Success Rate

Online Run1

Online Run2

Off-On Run1

Off-On Run1

0

200

400

600

800

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Success Rate

Filtered BC-20

Filtered BC-10

DigiRL-20

DigiRL-10

GPT-4V

subsets. This observation holds true whether considering all rollouts for computing this correlation or only investigating this correlation on rollouts that eventually succeed. This indicates the capability of DigiRL to solve tasks in a more efficient and directed manner. Qualitative examples can be found in Figure 16.

C Qualitative Examples

C.1 Random sample of trajectories for different agents

In Figures 13 and 14, we provide trajectories of DigiRL, AutoUI, and GPT-4V randomly sampled from our test set to offer a qualitative understanding of the agents' performance. As shown in these examples, DigiRLcan efficiently carry out in-the-wild device control tasks and less likely to get stuck or get to a wrong page compared to AutoUI and GPT-4V.

C.2 Error Recovery

We observe that DigiRL is able to recover from its own mistakes. As shown in Figure 15, we find that DigiRL explores ways to get back to the original screen in order to perform a search. As a comparison, AutoUI fails to reset to the original screen and gets stuck at the diverged screen. Under the hood, we find DigiRL trying to maximize the state value, which usually induces it to reset to the original screen (that has a large value to success).

DigiRL:

Google

AutoUI:

Got

stuck

GPT-4V

Google

Google

DigiRL:

Google

AutoUI:

Skipped

GPT-4V

Stops

Early.

DigiRL:

COSTCO

CoSTCO

1

CoSTCO

COSTCO

COSTCO

COSTCO

Top20

Top20

=2-

Find a Warehouse

Pdand set goah

ore hours and upc

400515

2-Day Delivery

2461

Got

2.9F

122.99

My Warehouse

529

'stuck

My Ware

4

AutoUI:

Early

.stop

GPT-4V

leonh-wtw.Lep lekgou.ksde

AirPods Pro (2nd Gen) with AppleCare*

COSTCO

X

mecome le Cestoe Wheies

$189.99 Free Shipping

Included

X

AirPods Pro (2nd Gen) with AppleCare+

COSTCO

Welcome te CostioeT

S0s #0 666818

Included

1

COSTCO

AirPods Pro (Znd Gen) with AppleCane+

Sudes e 666815

Included

品

1

american idel fnele

dutinportalny

rbis daftas mevericks

plter cseseo fias irjury updline

1 gift canf belanoe

三

HOTBUYS

Dtazy larntst

09

p

三op

HC

Macllook Pro laplop

Pro.chie.12-cere.

Aco+oriet

DW iddy -(puHL)

mxcteekgs]

a96 6id uot

X

HCGot

stuck

(16-nct) - Apple M3

nkSok 9

X

odop

DigiRL:

Skipped

633

Wrong

AutoUI:

Page

DURACELL

AAA

Skipped

GPT-4V

FUTURE

Skipped

BESTGSS

ou pnon.

search

24000

Best luy I Offcial Orlne Sho.

2:460 0

<

BackvardBoosted."

人

BEST

×

Best Buy Summervile in Su

<

desk.Airght

Best Buy: Page Not Found

<

2:50 0 0

X

Best Buy: Page Not Found

wwu.bevtiby.com

<

三

Summerille

Oyen* #10 a

Summervile. 5C 29483

Summerille

Svmmevle, 9C 29483

our Store:ummervil

BestBuy Summerville

STORE DETAILS

IN THIS STORE

Q

BEST

piidayDealt

三

Qift ideas

lop Deals

Q

三

Holiday Dels

Gn Canls

masbees

Cint ldeat

Find anether stere

Find another slore

321 Azalea Square Blvd

wwwbestby.cen wants to vse your

三

Q

三

Summerville, SC 29483

X0010

ALLOW

feg ep zo peg

NL tueg.ing m9g )

Get Dinections

Store Hours

10:00 AM 1o 8:00 PM

=> Curbside Pickup Hours

Thanks

We

macbook |

10:00 AM to 8:00 PM

M

R

W

Tueger Grils-

ROWOOO XL

盈

Oosi-Koda 1-Gas

NIU-602

FoodsbleL

Kick 9

Frarper Crils:

Back

Oeni-ikoda 16 Cas

Pizzas Oven -Illack

+ (2)

NU-KO

Foidsble

On This Page

shop

A

S

Z

D

X

F

C

G

V

H

N

M

K

S

2

d

X

9

Y

h

n

m

口

Most-vlewed items.

Most-vieweditems

Specialty Shops & Slervices

123

?123

4

4

42600

0061

XBest By Ofcial Online Sto.

人

Best Buy I Official Online Sto

Thesmartesttechfor

The smartesttechfor

The smartesttechfor

Oruha (al

The smartesttechfor

三

三

lop Deals

Omaha East

15:N 76h 5t

ufsmnvefo

Omaha East

115 N7h

K93N T

Open until 9 pm

Find another store

s0g820( s,aosAap

wants tio use yost

三

leerch

S三

BL0CK

ALL0W

Fop-Dea/s

lag ga o (mg

Top Deals

DeaiuttheDay

Got

1Ng 0*g Ang 2kg ,

Stuck

Samweg-Griry

Nnje-Miege

LtchenSyute

(次82

Wde L

Xoteo -flunturg

WM

Oe.I

Nnja - tMega

g-p

(C824)

72

HP-L/V

WattiMi Aluminum

Smurteutch.

Samsung-Gelery

Or. lender - llack.

★*** (100

HP-M

Most-viewed items

Most-viewed nem

口

C.3 Trajectory Length

Qualitative example on the number of steps in trajectories of DigiRL and filtered BC are shown in Figure 16. We find consistent cases where DigiRL has shorter trajectory length than filtere BC.

C.4 Reasoning failure of GPT-4V

The performance of GPT-4V failed on AiTW tasks predominantly due to not being able to carry out control actions as it plans on a high level, and then not being able to recover from these mistakes. Moreover, one of the main reasons why it is not able to recover from a mistake is that it might hallucinate and make itself believe that it is a wrong app or website. Indeed, GPT-4V constructs a plan of further actions when provided a task from either Web Shopping or General dataset of AiTW. Then, when it makes a misclick and fails to successfully proceed in an intermediate step, it might think that it actually solved that intermediate step and is in the correct app or website to execute further actions, causing the overall trajectory to fail. An example of this is provided in Figure 17. Here, we ask the model to search for an item in a webshopping website, in particular in 'newegg.com'. However, the model fails to proceed to that website due to not being able to precisely locating the search button. Then, instead of trying to go to that website again, the model thinks it is already in that webshopping website, and mistakes the search bar of Google with the search bar of 'newegg.com'. Hence, the rest of the trajectory also fails. Another slightly different phenomenon is illustrated in Figure 18. Here, the model is able to proceed to the correct website and search for an item, but this time it fails to tap on the search button on the website and clicks to an advertisement instead. Consequently, the model fools itself to think it successfully searched the item, and scrolls the page hoping to find that item, but it cannot do so because in reality it views the results of the advertisement. The primary reason of these failures is the challenge of grounding the control actions in GUI interfaces to realize the intermediary goals laid out by GPT-4V model's thoughts. As an example, we provide an illustration of trying to set up an alarm task in Figure 19. Here, in the last frame, it fails to execute the precise movements in the necessary amount of rounds to correctly set up the alarm to the desired time, and in the last frame we see that the action taken does not align with the thought process of the model.

Gotoebay.com,searchfor"lenovothinkpad"

Google

15flaig

L

)[ s*s

C

Coogle

M

L

Cooge

M

C

M

?

D Fine-grained failure modes

In Figure 20, we present a more fine-grained breakdown for all six failure modes provided in the user study. Those failure modes include:

Failure to recover from mistakes refers to the scenario where the agent made a mistake that led it to states from which it failed to quickly recover and resume the task, such as a wrong google search page.

Failure to click on the right link or failure to click refers to the failure mode where the agent either fails to locate the element that it tries to click on and keeps clicking on the nearby region, or fails to start typing in the string when it is supposed to do so.

Failure to take reasonable attempts at all refers to the failure mode where there is no clear reason that the agent fails to complete the task and does not seem to be on the right track throughout the trajectory.

Quit or press HOME early refers to the failure mode where the agent decided to finish the task or press HOME to start over before the task is actually finished.

Stops at wrong but relevant page refers to the failure mode where the agent arrives at a wrong page and mistakenly thinks that it had completed the task. For example, the agent finds a macbook on costco.com while the instruction asked it to find a macbook on ebay.com.

Technical issues refer to the failure mode that either the task is impossible (e.g. the tasks asks to open Amazon app but this app is not installed) or the agent is temporarily blocked from a certain website due to frequent visits.

The translation between fine-grained failure modes and coarse-grained failure modes is presented in Table 5.

10:12

10:12

TE B

10:14

UE 0

10:14

UE

10:15

/3

10:18

电

newegg.coml

四

earch or type web adds

alenware area 51 ebayg

WednesdagMay 15

Electronics Store: Tech, PC Parts, AI PC & More I Ne

newegg.com

日

( aauj Lodspunos asoq uaco-5ibamau

a

X aauj Lodspunos esoq woo-Sbewau

aliemware area 51 ebg

newegg.com

日

aliemware area 51

Search or type web adgiress

卧

Shop for newegg.com bose soundspo.

rodspunos esoq waoo-56emau ao, dous

alienware are 51m g

Best Buy1.

Belmart I $.

GLOTRENDS Store - Newegg.com

newegg.com bose soundsport free y - Google

$88.00 used

$88.00 used

四

neweg9.c0.

四

m9, - KW9H

com

$47.50 used

Your sugte arices ,期

That's all for now

hen

thanks

we

thanks

we

MO

W

b

M

1

W

P

q

W

a

S

d

e

S

d

a

d

e

S

P

h

k

口

To get personalized copg

gent suggested by

?123

Z

?

X

b

n

w

Z

?

X

C

b

n

m

214

Z

/

?

X

C

b

m

?123

/

Z

?

X

n

m

230

Trending Searches

mexico earthquakgtoday

cassio dias irjurygpdate

stranger things segpon 5 release date

nyc dublin portal @ towers

Kylian Mbappe

alabama lsu basepgl

口

四

国

8:230

a

ogdey woooosoo

costco.com comaers

costco.com tave

costco.com t

costco.com

四

AirPods Pro (2nd Geh) with AppleCare+

RicmCA

978

My tarhouse

COSTCO

三pp

HOT BUYS

202818208装88320814192522

Thanks

$189.99 Free Shipping

g.n

Search

Delvery Locstion

Starts Today!

94

Inclugled

四

5

自

AirPods Pro (2nd Geh) with AppleCare+

8:26

My Warehoss

COSTCO

Richm

sop

HOT BUYS

2220203189838540883421924

predator

hp/w.gpo.com

$189.99 Free Shipping

acer laptgom

acer aspl

Aoer Chrommpoks

Acer Monitors

acer

Incluled

predators

四

predatory

5

AirPods Pro (2nd Geh) with AppleCare+

8:27

四

MayOffers

Sort by:

My'tarhoute

Showing 1-24 of 268

COSTCO

三p

20

Best Match

p./w.ggpo.om

$189.99 Free Shipping

MayOffers

9ogloe1.Lar(p0

Inclurled

四

四

Fl

0

28

Showing 1-24 of 268

Sort by:

Delvegn

Best Match

012 00 9 8

****

$1,599.99

Tember Croga

After $200 0FF

Set

四

24

F

M

Q'WE'RT'YU'IOP

M

?123

a

S

Z

?

X

d

1

m

X

?123

Z

X

m

?123

A

Z

S

?

X

D

C

B

N

M

7123

a

Z

?

C

V

6

b

W

66665'15

After $200 0FE

Timber Crogig Swing

Set

国

$2,799.99

Lifetime Double Slide

① oeivery

Yourself

★★★★业

30

√

口

Thought:To complete

Thought:To complete

Thought:To complete

Thought:Tocomplete

Thought:To complete the

Thought:Tocompletethe

3:480

3:48

3:48

3:49

3:50

3:50

WednesdgMay 15

G

AUA

cuo2x

is

ALARM

世

C0010

TIMER

ALAM

10010

TIMER

24

3:48 Am

8:30

AM

8:30 Mm

8:30 Mm

Fhato

Mopa

All apps

Gmrl

Cock

WED, MAY 15

Mon, Tue, Wed, Thu, Fri

Mor

3:49 胃

Mon

3:49

盟

D

9:00 AM

Aapium S,

Calculater

Calendar

Camera

Sun, Sat

11

1

10

10

日

Clock

Centacts

Drive

Files

G

Coogle

Maps

国

Messagis

Phione

Phete

CANGELO%

CANCL8

a

Piay Movl,

Play Music

Settings

WeoVoee

国

16

Thought: To set an

Thought: To set an

Thought: To set an

Thought:To set a

Thought: To

Thought:To

General

Set-Of-Marks

GPT4V

Set-Of-Marks

Gemini-1.5-Pro

AppAgent

GPT4V

AppAgent

Gemini-1.5-Pro

AutoUI

CogAgent

Filtered BC

Offline

DigiRL

Offline

Filtered BC

Online

DigiRL

Online

Set-Of-Marks

GPT4V

Set-Of-Marks

Gemini-1.5-Pro

AppAgent

GPT4V

AppAgent

Gemini-1.5-Pro

AutoUI

CogAgent

Filtered BC

Offline

DigiRL

Offline

Filtered BC

Online

DigiRL

Online

Web Shopping

Fail to recover from mistakes

Fail to click on the right link or fail to type

Fail to take reasonable attempts at all

Quit or press HOME early

Stops at wrong but relevant page

Technical issues

Task success

E Experiment machines

Our main experiments are conducted on VM instances from Google Cloud Platform. Each VM instance comes with 1x Tesla T4 GPU and 16x Intel(R) Xeon(R) CPU.

F Setup for parallel environment

Running multiple emulators in parallel can be challenging due to the inefficiency in thread synchronization and frequent fault propagation when one emulator runs into an unknown error. To address this challenge, we set up a server-client system where all emulator processes are running in independent server processes. Each emulator process communicates with the main training process through different UIAutomotor servers. The main training process sends high-level instructions to UIAutomotor servers (such as reset and step), while UIAutomotor servers parse high-level instructions into low-level UI commands (such as typing a character and tapping at a coordinate) and such UI commands are executed by the emulator processes. When an exception is thrown in the emulator, the UIAutomotor examines if it is recoverable (e.g. an UI command takes too long to execute in the emulator) and reset the emulator process if it is not. When an exception is thrown in the UIAutomotor server, the main training process stops and resets the UIAutomotor server to ensure data correctness.

This design can easily be scaled up to a multi-machine setting. As illustrated in Figure 21, one host machine equipped with GPU accelerator has a local copy of the current policy π t , and distributes the policy to all worker machines equipped with only one GPU and multiple CPUs. Each worker machine will then collect trajectories of different tasks using π t . After all collection processes are synchronized, the host machine gathers all the trajectories together to update the policy to π t +1 . This process keeps iterating until the policy converges.

Speedup of emulation parallel. The performance boost with respect to the number of worker machines is nearly linear, as demonstrated in Figure 22 (right), where we conduct experiments that examine the scaling performance of our parallel emulator. Our distributed emulator that runs emulations across multiple servers can reliably collect data with up to 64 parallel emulators on 128 CPUs with near-linear speedup. In contrast, a naive baseline that runs all parallel emulations on the same server achieves much inferior performance (0.74 compared to 1.74 trajs/min using 64 CPUs).

G Autonomous evaluator details

Our autonomous evaluator gives a reward to each observation we get. The observation is composed of the current screenshot of device and the task. The evaluator gives a reward of 1 if the screenshot shows a completion of the task, and will terminate the POMDP as a result result.

The optimized prompt is shown in Figure 23 and Figure 24 for General and Web Shopping subsets respectively.

8 16

32

64

128

#CPUs

0

1

2

3

4

5

Emulation Speed (traj/min)

0.36

0.53

0.68

0.74

0.49

0.99

1.74

3.55

Vanilla Emulator

Distributed Emulator

H Zero-shot Baseline Details

Figure 25 shows the prompt that we used for testing the Set-of-Marks performance for GPT-4V and Gemini 1.5 Pro. This prompt is directly taken from Yang et al. [47].

I Hyperparameters

Hyperparameters for both Filtered BC and DigiRL are carefully tuned through binary search on the training set of General and Web Shopping subsets. The final choice of hyperparameters for both methods can be found in Table 6. As shown in the table, the only hyperparameters introduced by DigiRL are supervised training hyperparameters for the value function and instruction value function (including number of iterations and learning rate) and GAE λ .

11:21

1:26

1:33

G

G

hotels in washinton dc

三

Goog

Q

5-star hotels in washington dc

a

good restaurant in portland

Photos

Maps

M

Clock

Settings

a

hotels in washington d.c. downtown

ALL

MAPS

IMAGES

VIDEOS

NEWS

B0:

All aps

Q

cheap hotels in washington dc

Places

near Portland, OR

best hotels in washington dc

Appium S.

Calculater

显

Calendar

2

Camera

Chrome

Q

hotels in washington dc near

museums

K

日

M

Le Pigeon

4.6★★★★1 (919) ·SSSS

Clock

Cortacts

Drive

Fles

Gmall

French - 738 E Burnside St

Portland City Grill

G

4.5★★★★ (6,918) - SSS

Seafood - 111 SW 5th Ave 30th Floor

Google

Maps

Phone

Photos

W

4.5 ★★★★ (1,124) - SSS

Pacific Northwestem - 1239 SW Broadway

S

Sophisticated, locally focused NW bistro

Play Movi. Play Music

Settings

WebView

YouTube

Z

X

C

n

m

→More places

7123

?

pdx.eater.com > maps > 38-best-portland.

2:03

→ 0

Google

What is the price of a 12 ladd ×

ALL

SHOPPING

IMAGES

BO0KS

MAPS

Shop What is the price of a 12 la sponsoret ①

CURBSIDE

CURBSIDE

CURBSIDE

Pick up tomomow

Pick up tomorow

Pick up tomc

12 ft.

Wemer 12 ft. H

Wemer 12 ft.

$309.00

$319.99

$439.99

Home Depot

Spensored - www.homedepot.com/

Home Depot Ladders For Sale - Shop

Now - Home Depot@ Ladders

DIY Projects & Ideas ~ Free Shipping On Select Ladders

Purchased Online. Shop Home Depot..

Uikiah - North Orchard Avenue

Google uses your location to give you

×

local content. You can change this in

Settings.

toct

3:32

11:15 ○

7:27

0

G

bestbuy.com|

×

Clothing Shoes &Accessori.

/wwt.ebay.com

www.ebay.com

?

htps./ww.bestbuy.com/

×

ebay

8

三

bestbuy.com/creditcard

K

corsair k70

Clear

lenovo thinkpad x1 carbon

a

Q

bestbuy.com account

corsair k70 rgb mk.2

K

WS

Sort

Filter

a

bestbuy.com login

K

corsair k70 rgb pre

K

Auction

BuylItNow

corsair k70 pro mini

bestbuy.com gift card balance

K

corsair k70 rgb td

K

Condition

Processor

corsair k70 pro mini wireless

K

Lenovo ThinkPad T460s 14° Laptop

corsair k70

K

PC Intel Core i7 8GB RAM 128GB

SSD Windows 10

Very Good - Refurbished Lenovo

corsair k70 tkd

corsair k70 rgb tkl champion serles

k70

$

K

Free 2-4 day shipping

$219.99

BuylIt Now

M

e

Lenovo ThinkPad X1 Carbon 7th Gen

B

S

d

9

h

k

a

S

d

6

h

26# yono Lum

I4° FHD i7-8565U 8GB 256G8 SSD

Pre-Owned

Z

X

C

A

b

n

m

Z

X

C

A

b

n

m

$349.99

?123

?

7123

?

1,700+ resuts

9:00

woo-f6amaN1 sdx llap

www.newegg.com

EEE

DEALS

Q, dellxps

Hello Select address

"DELL XPS"

942 Results Ω

COMPARE (0)I

C

88

Featured Items Price   Newegg+Sold by Filter (0)

Dell XPS 15 9000 9530 15.6*

x 1200 - Intel Core i7 13th Ge..

Notebook - Full HD Plus - 1920

$1,499.99

FREE SHIPPING

@Sponsored

32081T88SD

Dell XPS 15 9520 15.6* FHD+

$1,849.99

FREE SHIPPING

Dell XPS 8950 Desktop

Computer - 12th Gen Intel Core

4000.00

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction explicitly state the contributions of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

Table 7: Hyperparameters for DigiRL and Filtered BC on both General and Web Shopping subset of AitW..

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in the last section of the paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be

used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not provide theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All loss functions and implementation details are provided in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We are still actively cleaning the code and make the environment more accessible to a broader audience. Once we are done with that, we will open-source the code along with the release of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Dataset details are provided in Appendix A.1 and hyperparameters are provided in Appendix I.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Repeated experiments are carried out with their means and standard deviations reported in Table 1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This information is provided in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeuIPS code of Etics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The positive societal impacts are discussed in the Introduction while the negative societal impacts are discussed in Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The capability of the model that we will be releasing is limited to simple tasks in Android in the Wild dataset, and therefore does not have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly cited the assets that we are using.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This submission does not include new assets. New assets including opensourced code, model checkpoints, and model trajectories will be released with documentation when we release the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This research does not involve crowdsourcing or human subjects. Annotations of trajectories in Figure 7 and Figure 8 are carried out by authors alone.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This research does not involve crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows the potential of Reinforcement Learning (RL) for designing an effective digital agent for in-the-wild control through Graphical User Interfaces (GUIs). The proposed approach relies on the advantage of the pre-trained visual language models (VLMs) while tackling real-world stochasticity by training an RL agent that interacts with an environment instead of relying on static demonstrations. Accordingly, this work proposes a novel autonomous RL approach, namely DigiRL, for training device control agents, which consists of two stages: an offline RL phase where the agent is trained on static demonstrations, followed by an in-the-wild, offline-to-online RL stage for training the agent through interacting with an environment. Consequently, this work also introduces a scalable and parallelizable Android learning environment with a reward model (evaluator) based on a robust VLM-based model. To show the effectiveness of the proposed method, an evaluation of different tasks given diverse instructions is carried out from the Android in the Wild dataset on real Android device emulators. The results show a significant improvement of DigiRL compared to the existing state-of-the-art agents, including wrapped proprietary VLMs such as GPT-4V and Gemini 1.5 pro. The paper claims to be the first to succeed in developing an autonomous offline-to-online RL approach to enable state-of-the-art performance on device control problems.\n\nSTRENGTHS:\n- The usage of RL in designing a successful digital agent for a device control task is fascinating. \n- I appreciate implementing such a scalable Android learning environment, and I hope the authors will open-source everything so that other researchers can reuse it.\n- The clarity of the paper is worth mentioning. \n- The experimental results section is rich, especially the ablation studies.\n\nWEAKNESSES:\n- Although the POMDP definition sounds correct when defining the problem, a contextual POMDP is even more appropriate for such a problem [1].\n- A pseudo-code or an illustrative diagram should be added to facilitate understanding the method.\n- It is not clear how the policy and the value network are conditioned given the context $c$. (implementation-wise)\n\n[1] Hallak, Assaf, Dotan Di Castro, and Shie Mannor. \"Contextual markov decision processes.\" arXiv preprint arXiv:1502.02259 (2015).\n\nQUESTIONS:\n- How are the policy and the value network conditioned given the context $c$?\n- Is the RL agent trained in a multi-task fashion (which I believe is true)? I mean the agent is trained with more than one task concurrently accessing fully or partially the same models.\n- If yes, do you think classical Multi-task learning or Multi-task reinforcement learning approaches would enhance the performance even more?\n\nLIMITATIONS:\nI believe the authors discussed the method's limitations in the final section. I agree with the authors regarding the impact of such application on the economy, society, and privacy, and that needs careful review in the future to limit any harm.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel autonomous reinforcement learning (RL) approach, DigiRL, for training in-the-wild device control agents. DigiRL first employs offline RL to fine-tune a pre-trained vision-language model (VLM as the agent) using stale task-specific data, and then further refines the agent through online RL by continuously interacting with parallelized emulators. DigiRL achieves a 49.5% absolute improvement in task success rate over existing state-of-the-art agents, establishing a new benchmark for digital agents in device control.\n\nSTRENGTHS:\n1. The paper is well-written and motivated, many important technical/implementation details are covered.\n\n2. The paper considers a challenging problem, autonomous device control, where existing LLM-based methods struggle to achieve acceptable success rate. The proposed method leverages VLM and RL techniques and significantly improves compared to these baselines.\n\n3. The experiments are comprehensive and informative, covering LLM/RL agents, prompting and learning paradigms, offline and off-to-on RL, as well as failure modes analysis.\n\n4. The authors implement a multi-machine emulator system to support parallel and real-time training of online RL.\n\nWEAKNESSES:\nMajor Points:\n\n1. From a ML methodological point of view, the novelty/contribution of the paper is limited. To perform offline and off-to-on RL, the paper adopts a number of existing techniques such as AWR, doubly-robust estimators with little customization (e.g.,  hard filtering on the advantages instead of computing $\\exp(A)$, which is mainly indended for easier implementation), all well-known to the community. The only thing seems \"new\" is training value functions with cross-entropy losses, also directly taken from [1], and the equations in line 250-251 seems to be qustionable (see my comments in the Questions section). Moreover, no theoretical insight is provided to elucidate why these specific designs are chosen.\n\n[1] Stop regressing: Training value functions via classification for scalable deep rl, 2024.\n\n2. Limited Scope. The entire paper focuses on a very specific domain (autonomous device control). The scope of the proposed method might be too narrow to be of general interest to the ML/RL community.\n\nMinor Points:\n\n- In section 4.2, how to properly balance the two estimators, one with higher variance and one with higher bias to achieve the optimal result? What's the hyperparameter profile of the combined estimator? Have you tried any alternative designs and can you give theoretical insight to justify this specific design choice?\n\n- Regarding the offline and off-to-on RL setting: to my knowledge, the main point of offline RL is to leverage a large body of stale data to safely and efficiently pretrain a RL agent. Therefore, for off-to-on RL, where online RL operates as the fine-tuning stage, one should use data far less than the offline pretrained dataset to ensure the setting is meaningful. The fact that authors intentionally use the same amount of data for both offline and online stage, which assumes access to a large amount of online data might make the offline pretraining unecessary. To see this, I recommend the authors to directly perform online RL on the combined dataset and it's highly possible that such \"purely online\" agent outperforms its off-to-on counterpart. \n\n-  The authors spend quite a few words discussing the challenges of stochasticity and device control as a POMDP. However, I do not see any specific design or techincal contribution targeting such problems.\n\nQUESTIONS:\n1. In line 250-251, the CE loss pairs r with log V and (1-r) with log (1-V). Intuitively, this means one would like to make the distribution of r and V as close as possible (when r-> 1, V -> 1 and vice versa). This seems to contradict the claim in line 240-241 that \"Intuitively, if a rollout attains a high value of A(sh, ah, c), it means the value function V is small\".\n\n2. How do you perform the train/test task split, if not random split? It is odd to see in Table 1 that almost all testing performance clearly surpass the training performance (normally should be the opposite), which suggest that the testing tasks are in general easier than the training tasks and not i.i.d?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an autonomous RL approach, RL for digital agent (DigiRL), to finetune a pretrained VLM as an in-the-wild device control agent through GUI. The authors build a parallelizable Android learning environment with VLM-based evaluator to identify the key design choices for RL. The training include two stages an offline RL phase on existing data, then followed by an online RL phase by interacting with real-world graphical user interfaces using the Android learning environment. The proposed method with only 1.5B model size outperforms other state-of-the-art models such as GPT4-V or 17B CogAgent in the Android-in-the-Wild (AitW) dataset.\n\nSTRENGTHS:\n- The paper is well-structured and easy to follow.\n- Many design choices are well motivated.\n- The experiments are nice and well support the claims.\n\nWEAKNESSES:\n- Overall there is no major weakness. There are only several questions and potentially interesting empirical studies to look at. Check more details in the question sections.\n\nQUESTIONS:\n- Could the authors compare more advanced LLM reasoning or planning algorithms like Chain of Thoughts (CoT), Tree of Thoughts (ToT), Reasoning as Planning (RaP), etc.?\n- Following the previous question, is it possible to compare with those planning/search-based methods with the autonomous evaluator or the trained value model of DigiRL as value functions?\n- What is the training time and compute requirement for online training?\n- In Figure 7, what does the AWR reweighting mean? Is it simply AWR?\n- With the auto-curriculum setup, it may be interesting to look at what types of data/replay are critical throughout the online learning process; with simple categorization like failure mode in Figure 5 or whatever characterization can be interesting.\nAs detailed in the \"Challenges of stochasticity\" paragraph in section 3, could the authors provide some studies on unpredictable distractor and technical glitches?\n\nLIMITATIONS:\nLimitations are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles AI agent training for controlling digital devices (e.g., web navigation). The proposed framework, named DigiRL, is a 3-stage training process consisting of model pre-training, offline fine-tuning (offline RL), and online fine-tuning (online RL). To achieve this goal, the authors first build a parallelizable Android learning environment that enables fast online interactions for policy learning; they then adopt a VLM-based evaluator to provide reward signals for the agents; finally, they perform ablation studies to examine several key design choices in typical policy-based RL methods for the third stage. Compared to larger models trained without this stage, the proposed approach enjoys significant performance enhancement due to the online fine-tuning stage.\n\nSTRENGTHS:\n+ The authors did a good job introducing the background, the problem setup, the baselines, and the details of their proposed method.\n+ Fine-tuning large VLMs in an online fashion can be challenging; the performance improvement obtained by the proposed method, which is relatively simple, is substantial and the overall approach looks promising.\n\nWEAKNESSES:\nThe main issue is the limited comparison with online RL methods for fine-tuning LLM-based agents. The only RL method compared in the experiments is Filtered BC (besides vanilla AWR, which the proposed method is based on). Filtered BC is strictly speaking not an online RL method. Admittedly, AI agent training for device control is a relatively under-explored new area, and the authors claim that theirs is the first successful offline-to-online RL approach for device control AI agents, I believe more experiments with other online RL baselines not originally designed for device control is required to justify DigiRL's advantages. For example, the classic on-policy methods such as REINFORCE and PPO, or the more recent ones that are more sample-efficient, such as [1, 2] (which might be considered more-or-less recurrent work, though). Further comparisons also help to provide more insight into the unique challenges of the device control problem for digital agents.\n\n[1] Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents\n[2] REBEL: Reinforcement Learning via Regressing Relative Rewards\n\nQUESTIONS:\n- Will the proposed method further scale well with more online interactions?\n- While multi-turn interactions are a challenge of the device control problem, is there any component in the proposed framework that specifically helps to tackle it?\n\nI am willing to adjust my ratings after seeing the authors' responses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}