DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Authors: Hao Bai
Yifei Zhou
Jiayi Pan
Mert Cemri
Alane Suhr
Sergey Levine
Aviral Kumar
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM/VLM Agent
Device Control
GUI Navigation
Web Agent
Reinforcement Learning
Primary area: natural_language_processing

Abstract: Pre-trained vision language models (VLMs), though powerful, typically lack training on decision-centric data, rendering them sub-optimal for decision-making tasks such as in-the-wild device control through Graphical User Interfaces (GUIs) when used off-the-shelf. While training with static demonstrations has shown some promise, we show that such methods fall short when controlling real GUIs due to their failure to deal with real world stochasticity and dynamism not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline and offline-to-online RL. We first build a scalable and parallelizable Android learning environment equipped with a VLM-based general-purpose evaluator and then identify the key design choices for simple and effective RL in this domain. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our 1.5B VLM trained with RL achieves a 49.5\% absolute improvement -- from 17.7 to 67.2\% success rate -- over supervised fine-tuning with static human demonstration data. It is worth noting that such improvement is achieved without any additional supervision or demonstration data. These results significantly surpass not only the prior best agents, including AppAgent with GPT-4V (8.3\% success rate) and the 17B CogAgent trained with AitW data (14.4\%), but also our implementation of prior best autonomous RL approach based on filtered behavior cloning (57.8\%), thereby establishing a new state-of-the-art for digital agents for in-the-wild device control.

O DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
Hao Bai
Yifei Zhou!*
Mert Cemri!
Jiayi Pan
Alane Suhr!
Sergey Levine
Aviral Kumar
UC Berkeley
PUICC $CMU
4 *Google DeepMind
Abstract
Training corpuses for vision language models (VLMs) typically lack sufficient amounts of decision- centrrc data. This renders off-the-shelf VLMs sub-optimal for rocision-making tasks such as in-the-wild device control through graphical user interfaces (GUIs). While training with static demonstrations has shown some promise, we show that such methods fall short for controlling real GUTs due to their failure to deal with real world stochasticity and non-stationarity not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM two stages: offline RL to initialize the model, followed by offline-to to-eee-tto--neeeee RL. To do this, we build scalable and parallelizable Android learning environment equipped with a VLM-based evaluator and develop simple yet effective RL approach for learning in this domain. Our approach runs wrraanage-weihheee RL with advantage estimators enhanced to account for stochasticity along with automatic curriculum for deriving maximal learning signal. We demonstrate the effectiveness of DigiRL. using the Android-in-the Wild (AitW) dataset, where 1.3B VLM trained with RL achieves 49.5% absolute improvement- - from 17.7 to 67.2% success rate over supervised fine-tuning with static human demonstration data. These results significantly surpass only the prior best agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent trained with AitW data (38.5%), but also the prior best autonomous RL approach based on filtered behavior cloning (5788)), thereby establishing a new Mtate-of-the-ut for digital agents for in-the-wild device control.
Introduction
Advances in vision-language models (VLMs), especially in regards their remarkable common- sense, reasoning, and generalization abilities imply that realizing a fully autonomous digital AI assistant, that can simplify human life by automating day-to-day activities computer devices via natural language interfaces, is no longer a distant aspiration [16, 45, 56). An effective device-control AI assistant should be able to complete tasks in-the-wild through Graphical User Interfaces (GUIs) on digital devices: make travel plans; experiment with presentation designs; and operate a mobile device autonomously, all while running amidst stochasticity and distractors on the device, the Internet, and the tools it interacts with. However, enhanced reasoning or common-sense abilities do not directly transfer intelligent assistant behavior: ultimately we want AI assistants accomplish
Equal contribution, listed in alphabetical order; work done at UC Berkeley. E-mails: haob2 illinois.edd edu, yifei zhou @ berkeley.edu, aviralku andrew.cmu..dd Project page: https://digirl-agent github.io Code available https://github com/DigiRL MeaiigiiL-ggga con/DigiRL-ageen/ddigr...
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

tasks, exhibit rational behavior, and recover from their mistakes as opposed to simply producing plausible completion to a given observation based on the data seen during pre-training. This implies that a mechanism to channel abilities from pre-training into deployable Al "agent" is lacking.
Even the strongest proprietary VLMMs, such as GPT-VV [24] and Gemini 1.5 Pro [7] 2, still struggle to produce the right actions when completing tasks on devices. While general-purpose vision-language abilities help these models still make meaningful abstract deductions about novel scenes when deployed, these deductions do not transfer to accurate reasoning for control [47, 45, 55, 44]. As a result, most prior work for building device agents construct complex wrappers around proprietary VLMs by combining them with prompting, search, or tool use [47, 44, 52, 51, 45]. While building prompting or retrieval wrappers to improve decision-maie making performance of existing VLMs enhances their performance in the short run, without updating the weights, the effectiveness of the resulting agent is inherently limited by the capabilities of the base model [49, 3]. For example, we found that off-the-shelf VLMSs make reasoning failures that derail the agent (e.g., Figure 2 and Figure 17), as direct consequences of inability of the base model to reason with low-level device-control actions. A different solution is to fine-tune the model on demonstrations via imitation learning. However, the dynamic nature of the web and device means that models trained to mimic actions in stale data can result in sub-optimalilty as the eco-system changes [26]. Agents trained in this way struggle to recover from the agents' own mistakes [8, 12].
If we can instead build an interac- tive approach to train a VLM to di- rectly adapt and learn from its own experience on the device and the In- ternet, that can be used to build ro-
Pretraining
Step II: Online
VLM is generolly y pre trafnedd on internet scale vaion--nd---n caan-and--nnuuugeee coto
Tases are campled from task detasee
@
chow me cat video
Model sowcrries tnnic poporllicl produce trojenories
Model
bust and reliable device-control agent, without needing wrappers on top of proprietary models. However, this learning-based approach must satisfy some desiderata. First, it must make use of online interaction data since static demonstration data would not be representative of the task when the model is deployed: for instance, even in the setting of web navigation alone, dynamic nature of in-the-wild web- sites means that the agent will fre- quently encounter website versions that differ significantly from the sce- narios seen during training and will need to behave reliably despite changes in visual appearance and distractions. Second, learning on- tt-hhe-fly means the approach must learn from multi-turn interaction data from the model itself, a large of chunk of which would consist of failures. Proper mechanisms must be designed to automatically pick out the correct actions while filtering the wrong ones.
Step Offline RL
AutoExal onnoturces rewnrd for with Irgectlory
offine
Autolbval
the Offine Modul
Annotated trgjestarics to through RL
DigiRL, overview. DigiRL is built upon a VLM that has been pre-trained on extensive web data to develop fundamental skills such common knowledge, reasoning, and visual grounding. Initially, we employ offline RL fine-tune the VLM using stale task specifie data, which helps in eliciting goal- coal-oriented behaviors. Subsequently, our agent engages with real- world graphical user interfaces, continuously enhancing its performance through online RI. and autonomous performance evaluations.
Figure 1:
To this end, our main contribution is a novel autonomous RL approach, DigiRL (i.e., RL for Digital Agents), for training device control agents, as shown in Figure 1. The resulting agent attains state-of-the-art performance on a number of Android device-control tasks. To train this agent, our approach operates in two phases: an initial offline RL phase to initialize the agent using existing data, followed by an offline-to-online RL. phase, that further fine-tunes the model obtained from offline RL on online rollout data. Online RL training requires access an environment that the agent can interact with and obtain reliable reward signals, all in a reasonable amount of wall-clock time. To do so, we build a scalable and parallelizable Android learning environment equipped with a robust VLM-based general-purpan purpose evaluator [26] (average error rate 2.8% against human judgement) that supports running up to 64 real Android emulators at the same time to make online RL real-time. Then, to effectively learn autonomously, we develop an online RL approach that retains the simplicity of supervised learning, but incorporates several key deep RL insights to enable fast fine-tuning. Concretely, our approach is variant of advantage-weighter regression (AWR) [28], equipped with: (i) an automatic curriculum that uses an instruction-level value function to order tasks so as to extract
We use external versions of these models as of June 11, 2024. Experiments with GPT and Gemini models were performed entirely by Hao Bai, Yifei Zhou, Mert Cemri, and Jiayi Pan.
2

---

DigiRL
AutoUI
GPT-4V
WebShop
② Go to bestbuy.com, search for "logitech g933"
X
Got
Wrong page
stuck
General
? How much does & bedroom
Buatiyy
spartment rent for in Denver?
Got stück
stuck
Figure 2: Qualitative comparison between DigiRL, and other approaches. AutoUl trained from static human demonstrations can easily get stuck in dutto--distribution states while GPT-4V often get a wrong goal (searched "logitech g933bestbuy.com logitech g933" in Google instcad of bestbuy.com). In contrast, DigiRL can recover from such states and complete complex instruction as requested.
maximal learning signal, which is inspired by prioritized replay methods [11, 32, 23], and (ii) another step-level value function trained via effective cross-entrops loss [17, 5] to extract low-variance and less-biased learning signal amidst stochasticity and diverse tasks. This RL approach allows us to fine-tune VLMs their own experience.
We evaluate our agent trained with DigiRL in carrying out diverse instructions from Android in the Wild dataset [31] on real Android device emulators and find that our agent can achieve a 28.7 % improvement over the existing state-of-the-e
agents (from 38.5% to 67.2% success rate) 18B CogAgent [9], and over 9% improvement over the prior best autonomous learning approach based on Filtered Behavior Cloning [18, 26]. The performance of our agent also significantly surpasses wrappers top of state-of-the-eee art proprietary VLMs such as GPT-4V [24] and Gemini 1.5 Pro [7] (17.7% success rate), despite using significantly smaller model (with 1.3B parameters). To our knowledge, this is the first work to succesffully build an autonomous offfine-to- to-online RL approach to enable state-of-the the- performance on device-contee problems.
2 Related Work
Multi-modal digital agents. In contrast to language-only agents that largely interact with both
text or code inputs and outputs [33, 49, 3, 30, 46, 20, 13], training multi-modal agents capable of controlling devices presents different challenges: first, device control is done directly the pixel- level and in a coordinate-base, action space, instead of natural language [31, 44] that LLM is most familiar with, and second, the ecosystem of device and the Internet tends be quite stochastic and unpredictable, which is absent with high-level planning in language only. To handle these challenges, prior work largely builds on strong proprictary VLMs [24, 71, and designs complex rule-based wrappers [47, 51, 45, 52] to enhance the visual grounding capabilities of VLMs in GUI interfaces and convert text output into pixel interactions. However, without any form of tuninnn this limits the room for possible performance improvement [44, 47, 49, 3, 50], especially when pre-training corpora only present limited action-labeled data. A separate line of work fine-tunes VLMs with demonstration data [19, 15, 9, 53] via imitation learning, but maximizing single-step accuracy from stale demonstrations without accounting for consequences of these actions subsequent steps may lead to poor solutions amidst stochasticity [26], as agents trained in such ways will struggle to recover from out-of-distribution states not included in the demonstration data [8, 12]. The third category, and perhaps the closest to us, are works that run filtered imitation learning data to directly maximize the episode success rate [26, 18J. In contrast, ours is the first work to scale autonomous, offline-to offline-to-online RL for device control, producing an agent that outperforms prior agents built via imitation. Even when compared to prior work running on on-policy RL in simplified web navigation settings (MiniWob++ [37, 10]), our approach is 1000x more sample efficient (around le3 trajectories compared to around lеб trajectories), and operates in real-world GUI navigation tasks.
nutonomously-collee. collected
Environments for device control agents. Recent works have introduced simulated environments for building device control agents [48, 56, 16, 54, 4, 44]. However, these environments are primarily designed for evaluation, and present only limited range of tasks within fully deterministic and
3

---

action space
dynamics
click
slide
type
home
back
enter
real -wold environment
agent model
open-ended
evaluator
non statioonryy website
pop-up
ads
unpredictable order
load
identity
Figure 3: Environment details. Top: actions space and dynamics of the environment. Bottom: examples of the word-world non-stationarity and dynamism of environment.
stationary settings, infeasible for acquiring a diverse repertoire of skills needed for device control. Alternatively, other works use environments with a greater diversity of tasks [48, 37J, but these environments often oversimplify the task complexity, thus failing to transfer to in-the-wild settings. Coversely, our training en environment utilizes autonomous evaluation [26] with Gemini 1.5 Pro [7] to support diverse, open-ended tasks on parallel actual Android devices, at full scale unlike prior environments. This also contrasts other prior works that use single-thre threaded Android emulators [26, 39, 19] and thus inefficient for support online RL at scale.
Reinforcement learning for LLM/VLMs. The majority of prior research employing RI. for foundation models concentrates on tasks that must be solved in single turn, such preference optimization [25, 58, 2] or reasoning [27]. However, optimizing for single-turn interaction from expert demonstrations may result in sub-optimal strategies for multi-step problems [57, 38, 42], especially amidst high degree of stochasticity or non-stationarity. Therefore, we focus on building multi-turn RL algorithms that can learn from sub-optimal, online interaction data in this work. While prior works have developed value-based RL algorithms for LLMs [42, 38, 1, 57, 50], they typically require maintaining multiple models such as Q-networks, nelue-networks, and policy networks, along with their delayed target counterparts, and can be subjective to slow convergence and sensitivity to choices of saprepparameters In contrast, we focus on identifying the key design choices for instantiating a simple yet effective RL algorithm for practitioners to incorporate to substantially improve full-scale Android device control. Our approach can serve as base model for future research.
3 Problem Setup and Preliminaries
Problem formulation. We are interested in pixel-based interaction with virtual devices. We scope our study the control of Android devices: this is already significantly more challenging and more general than previous learning- based environments that focus solely web navigation [16, 56, 4], where the web browser itself is merely one application within our broader environment, and link- based device controls [47, 51] are inadequate for tasks like games that do not support link inputs.
Each episode begins with the emulator initialized the home screen. Subsequently, a task selected from a predefined set of language instructions, some examples of which are shown in Appendix 1. An agent is then tasked with manipulating the emulator to fulfill this instruction. At each time step. the agent receives a screenshot of the current screen as the observation. Following the action space in prior literature [31], the available actions include tapping and sliding based on normalized (v. v) coordinates (ranging from 0 to relative to the screen dimensions). typing text strings of variable length, and pressing special buttons such as HOME, BACK, and ENTER, as illustrated in Figure 3. Our train and test instructions comes from General and Web Shopping subsets in AitW [31]. These tasks consist of information seriuttionngtthiinng tasks like "What's on the menu of In-n-On In-n-Out?". and shopping tasks on the web like "Go to newegg.com com, search for razer kraken, and select the first entry".
Challenges of stochasticity. Real-world device contrl presents unique challenges of stochasticity ab- sent in simulated environments [56, 37] such as: (1) the non-stationarity of websites and applications, which undergo frequent updates, causing the online observations to be different from stale offline data, (2) various unpredictable distractors such pop-up advertisements, login requests, and the stochastic order of search results. (3) technical challenges and glitches such as incomplete webpage loading or temporary access restrictions to certain sites. Examples of scenarios with such stochasticity from

---

our experiments are shown in Figure 3. We observe that these stochastic elements pose significant challenges for pre-trained VLMs, including even those fine-tuned on device control data. As a concrete example, Figure 4 shows an experiment result that illustrates the necessity of continuously adapting the models to the non-stationarity of websites and applications. After obtaining a good checkpoint using our approach (DigiRL), that we will introduce in the next section, with autonomous data from June. I to June.3, we compare the performance of a frozen policy and a continuously updating policy using fresh autonomous data from June.7 to June. 11. We find that indeed the the performance of the frozen policy gradually degrades over time due to the changes on websites and applications, while continuous online updates plays key role in preventing this degradation.
Setup for reliable and scalable online RL. As autonomous RL interleaves data collection and
Learning 10-line))
Frezen (Oriins)
3.3
2.70
training, to maximize learning amidst stochas- ticity, it is crucial to have a real-time data col- lection pipeline to collect enough experience for gradient updates. While this is not possi- ble in single-thread Android emulator environ- ments [26, 39] due to latency, we parallelize our Android emulator using appropriate error han-
0.55
0.55
0.50
0.55
0.30
0.20
5.15
dling as discussed in Appendix 1. In addition, the environment must provide a reward signal by judging whether the current observation in- dicates the agent has successfully completed the task. To generalize our evaluator to support wide range of tasks, we extend Pan et al. [26]'s end-to-end autonomous evaluator that does not require accessing the internal states of the emu- lator human-written rules for each task. This contrasts previous works that manually write execution functions to verify the functional com-
Jone
kove
Walttime
Figure 4: Performance of our approach (DigiRL) in different training modes on the Wehshop subset. When utilizing a stale checkpoint, i.c., "frozen" (black+blue curve) performance generally begins degrade time evolves, whereas autonomous online training (black+red curve) via DigiRL allows us to retain performance de- spite non-stationarity and stochasticity.
pleteness of each task [16, 48, 37, 44]. We adopt Gemini 1.5 Pro [6, 7] as the backbone of the autonomous evaluator. We seed this model with few-shot rollouts and the associated human-labelee success indicators to guide evaluation of novel queries. This pipeline enables a single evaluator that can evaluate all AITW tasks. The evaluator is highly aligned with human annotations (average error rate 2.8%), ), validated in Figure 8.
4 DigiRL: Autonomous RL for Building a Strong Device-Control Agent
We now present our autonomous RL framework for training device agents. We pose the device control problem Markov decision process (MDP) and develop RL methods for this MDP. The core of our approach is based on a simple and scalable off-policy RL method, advintage-weighted regression (AWR) [29], but we make crucial modifications to handle stochasticity and highly-variable task difficulty, through the use of value functions trained with appropriate losses, and an automatic curriculum, induced by an instruction-leve value function to maximize learning.
Device control and GUI navigation as a MDP. We conceptualize device control guided by nat- ural language instructions as finite horizon Markov Decision Process (MDP) represented by M = (S, A. T, po. R, H} and run policy gradient to solve this MDP. At the beginning, an initial state so and a natural language instruction c are sampled from the initial state distribution μoo A reward of is given at the end if the agent successfully fulfills the task per the evaluator, otherwise a reward of is given. The trajectory terminates either when the agent accomplishes the task or when the maximum allowed number of interactions H is exceeded. States are represented using the last two screenshots. To explain our approach in detail, we also include several standard definitions used in reinforcement learning (RL). The Q function for policy π represents the expected long- term return from taking a specific action at the current step and then following policy и thereafter: (Sh,, Ch. c) E,
- r(st, 44 })). The value function V (sh: c) is calculated by averaging the Q-vale,, (=(, ah, c), over actions an drawn from the policy π. The advantage ( an c) for a state-action pair is computed by subtracting the state's 's value under the policy from its Q-value: AT (Sh, an, c) = Q (sn, Un, c) V (81,
5

---

4.1 Backbone of Our Approach: Off-Policy RL via Hhantage-WWighteer Regression
The starting point we choose to build our approach on is the advantage-weighted regression (AWR) algorithm [29], which says that we can improve the policy reliably by regressing the policy towards exponentiated advantages induced by the reward function, as a proxy for optimizing the policy gradient while staying close the previous policy [14, 35, 34]:
arg maxx E, [log (aass c) exp (A(s, a, c)))))
(4.1)
for some positive parameter / and the distribution of past experience v, and A(s, a, e) denotes the advantage of state-action pair (s, a) given context To avoid tuning the hyperparameter 9, we consider an alternative that does "hard filtering" on the advantages instead of computing exp(A), (A), similar prior works [22. 43]. This leads the following loss function for fine-tuning the model:
(xx) -Bmer())) [lg (a]s
(4.2)
Typically, these advantages are computed by running Monte-Carlo (MC) rollouts in the environment to estimate the value of a given state-action pair, and subtracting from it an estimate of the value of the state given by a learned value estimator alone. However, this approach is likely to produce high-variance advantages given the stochasticity of the device eco-system that affects MC rollouts.
4.2 Obtaining Reliable Advantage Estimates from Doubly- Robust Estimators
To reliably identify advantageous actions given significant environment stochasticity, we construct per-step advantage estimator, inspired by doubly-robust estimators [40, 36]:
(sn, an. c) == X" ("'""rsHH (* cc.c) + (1 - -2=-"--aa > (rsss,asss.a........... (x, (V*P() (s)+1.)) + r(*,, an, c) - (sn, c)),
(4.3)
where A is a weighting syper-parameter. This construction of the advantage estimator a simplified version of Generalized Advantage Estimation (GAE) [36] using only the next-step advantage estimator and final-step advantage estimator as there are no intermediate rewards our problem. This construc- tion balances an advantage estimator with higher variance Monte-Carke estimates ===Rr($11 am, c) (due to stochasticity) and an estimator with higher bias / e) + r(8,, an: e) - 1/stp c) (due to imperfect fitting of the value function). We observed that combining both high-variance and high-bias estimators gave us a sweet-spot in terms of performance. To implement the step-level hard filtering. we simply threshold this doubly robust estimator as A ( ahe c) > 1/H H to decide which actions progress towards the goal.
4.3 Automatic Curriculum using an Instruction-LLevel Value Function
While the AWR update (Equation 4.1) coupled with robust advantage estimator (Equation 4.3) is likely sufficient on standard RL tasks, we did not find it to be effective enough for device control in preliminary experiments. Often this was the case because the task set presents tasks with highly- variable difficulties that collecting more data on tasks that the agent was already proficient at affected sample efficieny negatively. In contrast, maximal learning signal can be derived by experiencing the most informative tasks for the agent during training. To this end, we design an instruction-level value function / Vinstruct
(c) evaluate if given rollout can provide an effective learning signal:
A Ansttuut (sh: c) == -Et ac, Viinsmetcc = r( cc Vrinsmmttc)) (c), (4.4)
where Σ r(st, az, c) is a Monte-Carlo estimator of Q(sh, an, c). The equality holds because the MDP formulation only provides rewards at the end of a rollout. Intuitively. if rollout attains high value of A Ainstruct (sh. ax, e). it means the value function Vinstruct is small. Therefore, this rollout represents valuable experience of the agent accomplishing a difficult task, and thus should be prioritized, akin to ideas pertaining to prioritized experience [32] or level replay I]. When training the actor with a buffer of historical off-policy data, we first perform a filtering step to identify the top-p datapoints with highest A Ainstmact (s), an c). Then, we use it for AWR (Equation 4.1) with the doubly-robust advantage estimator (Equation 4.3).
Implementation details. Inspired by the findings in some recent works [5, 17] that modern deep learning architectures like transformers [41] are better trained with cross-entropy losses instead of mean-squared losses, we utilize a cross-entropy objective based on the Monte-Carlo estimate of the trajectory reward for training both of our value functions:
L(V"))- -By(r(sH an, c) log (C) + (1 rrss, an, c)) (Vr()))))
(4.5)
<(V"*)) -E, (r(ss, az c) log (st, an, c) + (1 -1-10 r(sH, an, c)) log(1 - V an, an, c]]]. (4.6)

---

Instruction-levll value Function
Step-level Value L'nction
Task
Task
Task
Oo searth Taras oebbcck
1itfloulyy saro
0.8
0.2
dicarted
Uo embhook"
0.20
8.00
Hilffealy mediam)
"hees sundopant and odter tint ecory totticuay hand)
8.01
-0.0
dicaried
(4 - po
(o,))) <.
Train MLE Equation (1.2)
instruction-leww level
step-level
value function
value function
actor
Figure 5: Algorithm visualization. The two value function first trained with original distribution of collected trajectories according to Equation (4.5) and Equation (4.6), then used to filter the trajectories for training the actor. We use the MLE loss (Maximum Likelihood Estimation loss) to train the actor.
Final algorithm. The final practical algorithm shown in Figure 5. The instruction-level value function estimates the values of the trajectories, which is trained with loss shown in Equation (4.5). The step-level value function estimates the values of states, which trained with loss shown in Equa- tion (4.6). When training the actor, we first filter out trajectories and states using the value functions as shown in Equation (4.4) and Equation (4.3), then train the actor with the MLE loss shown in Equation (4.2) on the filtered data.
5 Experimental Evaluation
The goal of our experiments is to evaluate the performance of DigiRL on challenging Android device control problems. Specifically, we are interested in understanding if DigiRL can produce agents that can effectively learn from autonomous interaction, while still being able utilize offiine data for learming. To this end, we perform comparative analysis of DigiRL against several prior approaches, including ootte--f-thh-arr agents in Section 5. We also perform several ablation experiments to understand the necessity and sufficiency of various components of our approach in Section 5.2.
Baselines and comparisons. We compare DigiRL with: (a) state-of-the-art agents built around
proprietary VLMs, with the use of several prompting and retrieval-style techniques; (b) running imitation learning static human demonstrations with the same instruction distribution, and (c)a filtered BC approach [26]. For proprietary VLMs, we evaluate GPT-4V [24] and Gemini 1.5 Pro [7] both zero-shot and when augmented with carefully-designed prompts. For the zero-shot setting, we use the prompt from Yang et al. [47] and augment the observation with Set-of-Marks [55]. Set-of- Marks overlays a number for each interactable element over the screenshot, so that a VLM can directly output the number of the element to interact with in plain text instead of attempting to calculate pixel coordinates. which is typically significantly harder. We also compare with AppAgent [47], which first prompts the VLM to explore the environment, and appends the experience collected to the test-time prompt. We also compare with two state-of offthhe-rt fine- the-ttunng methods for Android device control: AutoUI (specifically AutoUI-Base [53]) and CogAgent [9]. Auto Base uses an LM with 200M parameters, and a a vision encoder with 1.IB parameters. CogAgent has 11B parameters for its vision encoder and 7B for its LM. The supervised training corpus for both AutoUl- and CogAgent contains AitW, including the instruction set and the emulator configuration we use.
Base VLM and offline dataset. Both Filtered BC and DigiRL, use trained AutoUI-Base checkpoints with the image encoder frozen. The instruction and step-level value functions for DigiRL employ this same frozen image encoder. The visual features output from the encoder are concatenated with instruction features derived from RoBERTa [21]. two-layer MLP is then used to predict the value function. In the offline phase, the offline dataset is collected by rolling out the initial AutoUl- supervised trained checkpoint as policy. For fair comparisons, we keep the number of offline data collected in the pure offline training roughly the same as the total number of data collected in the offline-to-online training. Due to the dynamic nature of the Internet-device eco-system, our offline data was stale by the time we were able to run our offline-to-online experiments, and this presented additional challenge in offline-to-online learning. In both General and Web Shopping subsets, offline experiments make use of around 1500 trajectories while offline-to-online experiments start with
7

---

AitW General
AitW Web Shopping
Train
Test
Train
Test
GPT-4V
5.2
13.5
3.1
8.3
Gemini 1.5
32.3
16.7
6.3
11.5
Prompting
GPT-4V
13.5
17.7
12.5
8.3
APPAGENT
Gemini 1.5 Pro
14.6
16.7
5.2
8.3
SUPERVISED
CogAgent
25.0
25.0
31.3
38.5
TRAINING
AutoUI
12.5
14.6
14.6
17.7
Filtered BC
51.7 5.4
50.7 1.8
44.7 1.6
45.8 0.9
Learning
OFFLINE
Ours
16.9 5.66
62.8 1 1.0
39.3 6.0
45.8 6.6
Filtered BC
53.5 0.8
61.5 1.1
53.6 = 4.7
57.8 2.6
OFF-TO-ON
Ours
63.5 = 0.0
71.9
68.2 6.8
67.2 #7.2:11
Table 1: Main comparisons of different agents across various settings. Each offline experiment is repeated
three times and the mean and standard deviation are reported. Each online experiment is repeated two times. Results are evaluated with autonomous evaluator with the first 96 instructions in the train and test set. Correlation our correlation and human judgements can be found Figure 8.
Filtered BC-1
Filtered BC-2
DigiRL-1
DigiRL-2
GPT-4V
0.8
0.7
0.7
Success Rate
0.6
0.6
0.4
0.4
0.3
0.2
0.2
0.1
0.1
320
640
960
0
320
640
960
# Trajectories
# #rajeectories
Figure 6: Offline-to-online training curves Filtered BC and DigiRL. Curves are smoothed with expo- nential weighting over the x-axis. Left: AitW General. Right: AitW Web Shopping. Two runs for each model are started on two different dates with at least two days apart. Observe that DigiRL is able to improve faster with fewer number of samples. Since the data collection frequency the bottleneck, these performance trends directly reflect performance trends against wall-clock time as well.
around 500 offline trajectories and update with another 1000 online trajectories. In the offline phase, DigiRL skips instruction-level filtering and instead trains the actor with all successful trajectories to make full of the offline data. See detailed breakdown of our dataset in Appendix A.
5.1 Main Results
Our main results are summarized in Table 1 and Figure 6. We find that on both AitW General and AitW Web Shopping subsets, the agent trained via DigiRL, significantly outperforms prior state-of-the-at methods based on prompting and retrieval (AppAgent + GPT-4V/Gemini 1.5 Pro) or training on static demonstrations (CogAgent and Auto by a large margin with more than 49.5% absolute improvement (from 17.7% to 71.9% on the General subset and from 17.7% to 67.2% on the Web Shopping subset). Notably, this improvement from DigiRL realized fully autonomously without making use of human supervision (e.g. manually labeled rollouts or hand-written verifiers).
Are inference-time prompting and retrieval techniques supervised training enough for device control? Delving into Table 1, we observe that off-the-shelf proprietary VLMs, even when supplemented with the set-of-marks mechanism, do not attain satisfactory performance: both GPT-4V and Gemini 1.5 Pro achieve success rates under 20%. One possible cause could be the under- representation of Android device data in the pre-training data. Moreover, inference-time adaptation strategies such as AppAgent [47] show minimal improvement, with gains not exceeding 5% for either model. All this evidence suggests a limited scope for improvement without fine-tuning of some sort. As illustrated in Figure 7, the primary failures of these VLMSs stem from hallucinatory reasoning that lead the VLMs to land on relevant but wrong page. This suggests that while ofate-of-the-ar VLMs excel at reasoning problems in code and math, their reliability in less- familiar domains, such

---

Set. or Marka
General
Grrey
Gemini S-Pro
Appdgent
HiLmb
Fall kes
sussstckmieee Fallure Mode
Arrive at goal
Astoull
Gemin-1 5-Pro
Cogaggont
in Tralectories
Web Shapping
R BC
Offine
BC
Onäin
Fall recover from mistakes
Get stuck midhosy F Fallure Mode
Arrive at wrong gowl
Ondha
Figure 7: Failure modes for each approach on both the AiTW General and Web Shopping subsets. We found that the failure mode RL training is most effective at reducing compared to model supervised trained on human data is "Fail to recover from mistakes". A more fine-grained decomposition can be found in Appendix D.
as device control, remains inadequate. For example. for the instruction "Go to newegg. com, search for alienware area 51, and select the first entry", a GPT-4V based agent erroneously searched "alien area 51 ebay" in Google.com and decided that it had made progress towards the task (Figure 17).
Training on domain speciffc human demonstrations, however, does boost performance, allowing the smaller, specialized VLM, AutoUI with 1.5 billion parameters, to match or surpass the larger, generalist VLMs like GPT-4V and Gemini 1.5 Pro. Nonetheless, this supervised imitation learning approach still fall short, with success rates on both subsets remaining below 20%. This shortcoming not fundamentally addressed via enhancements in model scale or architecture, as evidenced by CogAgent [9], with 18 billion parameters still achieving performances below 40% success rate. As depicted in Figure 7, predominant failure mode for these agents an inability to rectify their own errors. An example trajectory that we observed is that for the instruction "what's on the menu of In-n-Out", the agent accidentally activated the voice input button, and failed to quit that page until the step limit. In contrast, DigiRL is able to recover from the errors more efficiently( Appendix C.2).
Comparison of different RL approaches. In Table I and Figure 6, we present comparative analysis of various autonomous approaches. Notably, both offline and offline-to-online configurations demonstrate that our RL approach, when augmented with a continuous stream of autonomous interaction data and reward feedback, substantially improves performance. This improvement is evident from an increase in the success rate from under 20% over 40%, as the agent learns to adapt to stochastic and non-stationary device interfaces. Moreover, although the total sample sizes for offline and offline-to-onling settings are equivalent, the top-performing difline-to-oning algorithm markedly surpasses its offline counterpart (75% versus 62.8% on the General subset). This highlights the efficacy of autonomous environment interaction, and establishes the efficacy of DigiRL in learning from such uncurated, sub-optimal data. Lastly, DigiRL consistently outperforms the state-of-the--ee ahe alternative, Filtered BC, across both the General and Web Shopping subsets, improving from 61.5% 5% to 71.9% and 57.8% to 61. 61.4% respectively, highlighting DigiRL's performance and efficiency.
5.2 Analysis and Ablations
Failure modes analysis. We conduct an additional user study to annotate the failure modes for each agent as shown in Figure 7, and more fine-grained breakdown can be found in Appendix At a
high level, we classify the major failure modes of all agents into the following three categories: (1) Failure to recover from mistakes refers to the scenario where the agent made a mistake that led it to states from which it failed to quickly recover and resume the task, such wrong search page. (2) Getting stuck midway refers to the failure mode where the agent gets distracted on the right track to completing the instruction and as result fails to accomplish the task. For example, failing to click on the right link or failing to search after typing the key words. (3) Arriving at wrong goal refers to the failure mode where the agent arrives at wrong page and mistakenly thinks that it had completed the task. For c.g, the agent finds a macbook on costco.com instead of finding a macbook on ebay.com.
While all the types of failure modes benefit from offline and cifline-to-onlingg RL training as shown in Figure 7, the most consistent and significant reduction is probably for the failure mode of failing to recover from mistakes. This is because while pre-trained models, generating plausible future tokens, can get distracted by the dynamic nature of the environment and, as a result, encounter at never-before-seet states. With no clue of how to escape such states, these methods are unable to recover and fail to solve the task. In contrast, by training on autonomously-collette rollouts, our agent DigiRL is able to learn from its own mistakes and reduces failures recover over training.

---

General
53.1
594
443
72.8
125
25.
250
AppAgeett
AppAggent
Autoli
EogAgent
Fitered PC Offinnc
Dani Offine
Fillared IC: Online
Piali Onlino
OPTRY Gamni 1.5
Gemini
Policy Model
Web Shopping
614 124
11.
225
12
Set OF WoN.eStt Marks AspAgent Aggiggnt
Autoli
CogAgent
FiterdBBCC Driine
DigiiL Offine
Filtorel BC Online
DigRL Online
GPTAV Gemmii Serro Grady
Gemino Sahro
Policy Model
Figure 8: Correlation between autonomous evaluator and human judgements for all policy models on General and Web Shopping subsets. For repeated offline and online runs, we report the correlation results for the run with the highest autonomous evaluation success rate.
Ablation study of each component in DigiRL.. We conduct an ablation study on different components of DigiRL in Figure 9. We find that all the components used by our approach are necessary: (1) using cross-entropy for training the value functions boosts performance by around 12% (compare Ours and Ours w/ Regression); (2) using step-level advantages improves efficiency by 12% (comparing Ours and Ours w/o step-level advantage); (3) the use of automatic curriculum improves the speed of learning by around 25% (comparing Ours w/o step-level advantage and Filtered BC); (4) Ours outperforms vanilla AWR that does not employ a doubly-robust ad advantage estimator or curriculum.
Additionally, we also observe degradation in performance as a result of "hard-filtering." as show by nearly comparable performance of our approach and the best run of exponential filter- ing obtained via extensive tuning of the tem- perature hyperparameter 7 in naïve AWR (com- paring Ours and Ours w/ vanilla AWR reweight- ing), despite simplicity of implementation in the hard filtering approach. Putting together, these choices result in a new state-of-ther offt---rr-t RL ap- proach for device control.
meseighting
2.65
0.60
2.55
8.25
5:20
100
200
300
400
560
600
Evaluation of our autonomous evaluator. In Figure 8, present the findings from a study aimed at assessing the accuracy of our autonomous evaluator. Our results indicate that the success rates reported by our automatic evaluator are remarkably consistent with those assessed by human evaluators across almost all models, with differences less than 3%. Furthermore, we observed that evaluations on the Web Shopping subset are more precise compared to those on the General subset. This increased accuracy likely stems from the fact that tasks in the General subset are formulated in free-form language, which can introduce ambiguity, whereas the Web Shopping subset features a narrower range of language expressions, reducing potential variability.
#Tr #Trajectories
Figure 9: Ablation study results on the AitW Web Shopping subset.
6 Discussion and Limitations
In this paper, we propose a novel autonomous RL approach, DigiRL, for training in tn-the-wild, multi- modal, device-control agents that establish new state-of-the-an performance on number of Android control tasks from Android-in-the Wild dataset [31]. To achieve this, we first build a scalable and parallelizable Android environment with a robust VLM-based general-purye enteral-purpose evaluator that supports fast online data collection. We then develop a system for offline RL pre-training, followed by autonomous RL fine-tuning learn via interaction, admist the stochasticity of the real-world Internet and device eco-system. Our agent achieves a 280% improvement over the previous state-oftthe-an agents (from 17.7% to 68.2% in terms of task success rate), including AppAgent based GPT-4V and Gemini 1.5 Pro, and supervised trained models such as AutoUI and CogAgent.
Due to computational limitations, and despite the fact that the parallel emulator and autonomous evaluator can be easily extended to complicated tasks, our agent is trained only with tasks from AitW instead of all possible tasks the device. Our design of the DigiRL algorithm aims for maximal implementation simplicity, so we hope that our approach to serve as a base algorithm for future research to build on, including algorithmic research as well as expanding the space of tasks.
10

---

Acknowledgements
We thank Yi Su, Izzedin Gur, Xinyang Geng, and Sandra Faust for feedback on an earlier version of this paper and for informative discussions. This work is supported by NSF IIS-224681 and ONR N00014-21-1-2836 and Gemini 1.5 Pro credit donations for academic use and cloud resources from Google Cloud.
References
[1] Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023.
[2] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback, 2023.
[3] Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. ArXiv, abs/2310.05915, 2023. URL https:
/ seantticaant org/ httg///rruSSD--- 263829338..
[4] Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024.
[5] Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taīga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep rl, 2024.
[6] 2023 Gemini Team. Gemini: A family of highly capable multimodal models, 2024.
[7] 2024 Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan Adams, and Sergey Levine. Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability. NeurlPS, 2021.
[9] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: visual language model for gui agents, 2023.
[10] Peter C Humphreys, David Raposo, Toby Pohlen, Gregory Thortoon, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy Lillicrap. data-driven approach for learning control computers, 2022.
[11] Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. CoRR, abs/2010.03934, 2020. URL. htttp:://aaiiii htttp::://arii..og//b////////// 03934.
[12] Yiding Jiang, J Zico Kolter, and Roberta Raileanu. On the importance of exploration for generalization in reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[13] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024.
[14] Sham M. Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning, 2002. URI. https://api. semanticscholar.cm 3at.aaarrrrrrroorus1111334111

---

[15] Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web, 2024.
[16] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents realistic visual web tasks. arXiv preprint arXiv:2401. 2024.
[17] Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q-learning on diverse multi-task data both scales and generalizes, 2023.
[18) Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Autowebglm: Bootstrap and reinforce large language model-based web navigating agent, 2024.
[19] Juyong Lee, Taywon Min, Minyong An, Changyeon Kim, and Kimin Lec. Benchmarking mobile device control agents across diverse configurations, 2024.
[20] Xiao Liu, Hao Yu, Hanchen Zhang. Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlic Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating Ilms agents, 2023.
[21] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 11692, 2019. URL 11692.
[22] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online re- inforcement learning with offline datasets. CoRR, abs/2006.09359, 2020. URL https: 09359.
/ //arxiv.org//bs////
[23] OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand, 2019.
[24] 2023 OpenAl Team. Gpt-4 technical report, 2023.
[25] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, 01l/2203.021555 2022. URL https:// //api. cemanticscholar.om org/ CorpusID: 246426909.
[26] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Au- tonomous evaluation and refinement of digital agents. arXiv preprint arXiv:2404.06474, 2024.
[27] Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. 2024.
[28] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage regression: Simple and scalable off off-policy reinforcement learning. CoRR, abs/1910.001777. 2019. URL http:// /ttt:://axxv...o orgoorgg/bbbo 00t.//990.000000
[29] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightte regression: Simple and scalable off off-po reinforcement learning, 2019.
[30] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
12

---

[31] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. arXiv preprint arXiv:2307.1008: 2023.
[32] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. 2016.
[33] Timo Schick, Jane Dwivedi-Yoe Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools, 2023.
[34] John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. CoRR, abs/1502.05477, 2015. URL http:://atiii...tt/ httt:://axxiv.org/bbs... 1502.05477.
[35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 707 707.06347, 2017. URL http:://axxi..orrr http://axiv...rr//bbba/aa// org/abs/ 1707.06344
[36] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation, 2018.
[37] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1135-3144. PMLR, 06-11 Aug 2017. URL https://proceedingg .ariiinnnsmml.........m mlr. wwwps//esvvveee https/////shhi//a html.
[38] Charlic Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning, 2023.
[39] Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv preprint wrXiv:2105.13233 2021.
[40] Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. CoRR, Mass/1509.066 2015. URL http::/arxxv.ce http://wwwiv.rraa////////// 06461.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023.
[42] Siddharth Verma, Justin Fu, Mengjiao Yang, and Sergey Levine. Chai: A chatbot ai for task-oriented dialogue with offline reinforcement learning. 2022.
[43] Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression, 2021.
[44] Tianbao Xie, Danyang Zhang. Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint XXXx:2444.00972. 2024.
[45] An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, Zicheng Liu, and Lijuan Wang. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation, 2023.
[46] John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with exccution feedback, 2023.
[47] Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.1377. 2023.
[48] Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023.
13

---

[49] Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning: Enabling generalized agent abilities for 1lms, 2023.
[50] Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine. Fine-tuning large vision-language models as decision mriisinn-mmaing agents via reinforcement learning. arXiv preprint arXiv:2405. 10292, 2024.
[51] Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, al. Ufo: A ui-focused agent for windows interaction. arXiv preprini urXiv:2402.07939, 2024.
[52]
Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang, Android in the zoo: Chain-of-action--hhuuhtt for gui agents, 2024.
[53] Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents, 2023.
[54] Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. Mmina: Benchmarking multihop multimodal internet agents. arXiv preprint arXiv:2404.09992, 2024.
[55] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is a generalist web agent, if grounded, 2024.
[56] Shuyan Zhou, Frank Xu, Hao Zhu, Xuhui Zhou, Robert Lo., Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. ArXiv, abs/2307.13854 2023. URL https:
//api. seaanticcanic htt.:aarrrrgg/oopusD:226011111 250164788.
[57] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint UrXiv:2402.19446. 2024.
[58] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1 909.08593, 2019. URL http: / /awxiv.vvvvv.////aa org hagg/abs/11 08593.
14

---

Appendices
A Environment details
A.1 Post-processing of AitW
The Android in the Wild (AiTW) task set is a large-scale dataset for android device control, containing five subsets: GoogleApps, Install, Web Shopping, General, and Single, where we select the General and Web Shopping subsets. Single subset is not considered here because all tasks in Single can be completed within one step and thus this subset fails to examine the multi-step challenges that we are interested in this paper. Install and GoogleApps are not considered due security reasons those tasks require an active Google account and parallel emulations can flag security concerns.
The General set focuses on searching for information and basic application usage. For example, contains searching for latest news in Chile, search for flights from NYC to Sydney, opening Gmail, etc. We use all 545 tasks in the training set for training and the first 96 tasks in the test set for testing due to computational and budget constraints. The maximum allowed number of steps for this subset is 10. Offline data is collected by rolling our the initial AutoUI policy with tasks from the training set. The offline data used for the offline-to-online setting contains 608 trajectories while the offline data used for the offline setting contains 1552 trajectories. Some task examples shown in Table 3.
General.
Task Example
How do get to the nearest Verizon Store?
How much does 2 bedroom apartment rent for in Denver?
Search for flights from Barcelona to Boston
What's good restaurant in New York?
What's on the menu at Burger King?
Table 2: Examples of task descriptions in the ATTW General task set.
Web Shopping. The Web Shopping subset comprises search instructions on various shopping websites, like searching for razer blader on ebay. As some websites (e.g. Amazon) and operations (e.g. adding items to cart) frequently require captcha verifications, we post-process the Web Shopping subset to exclude such operations and websites and also make the task casy to evaluate for our autonomous evaluator. The resulting task set involves navigating through five websites ccostco.com, bestbuy.com, target.com, walmart.com, search in the website, and select items from the searched results). Our post-processed training set contains 438 tasks and our testing set contains 96 tasks. Example tasks after post- processing can be found in Table 3. The maximum allowed number of steps for this subset 20. Offline data is collected by rolling our the initial AutoUl policy with tasks from the training set. The offline data used for the -ffline-t--onlie setting contains 528 trajectories while the offline data used for the offline setting contains 1296 trajectories.
newegg.com) and three basic operations (go to website,
Difficulty
Task Example
Go to costco.com
1
Go to walmart.com
Go to costco.com, search for "bose soundsport free"
2
Go walmart.cm com, search for "logitech g910"
Go to costco.com, search for "bose soundsport free" and select the first entry
3
Go to walmart.com, search for "logitech g910" and select the first entry
Table 3: Examples of task descriptions in the AITW Webshopping task set.
15

---

AitW General
AitW Web Shopping
All Trajectories
Successful Trajectories
All Trajectories
Successful Trajectories
DigiRL. Runl
6.31
4.40
11.35
7.23
DigiRI. Run2
6.64
5.04
10.86
6.55
Filtered BC Runl 8.08
6.56
12.05
6.88
Filtered BC Run2
7.36
6.13
14.72
9.62
Table 4: Average rollout length of the DigiRL, agent compared to filtered BC. Darker green means shorter rollout length. On both AitW General and AitW Web Shopping test subsets, we find that DigiRL consistently produces shorter length rollouts than filtered BC.
B
Other Quantitative Experiments
B.1
Curriculum Learning
When running experiments the AitW Web Shopping subset, we find solving easier tasks helps improve solving harder tasks, where the difficulty is identified in Table 3. By specifying the difficulty DigiRL-Runl in Figure 6, empirically show the success rates of each difficulty across the online learning process Figure 12, we observe that a significant increase of success rate tasks of difficulty leads to increasing success rate of difficulty 2, and the same pattern for difficulty 2 and demonstrating effective curriculum learning.
Difficutty 1
Difficulty 2
Difficltty 3
Filardd BC-20
#Success Rate
0.6
0.6
0.5
0.4-
0.2
0.0
0.1
Trajectories
#Trajectories
Figure 10: Left: Success rate under different difficulties for the AiTW Webshopping task set. Right: Success rate under different methods with different horizon length (H {10, 20}) the AiTW Google Search task set.
B.2
Learning Method
We ablate on the learning method, i.e. online learning or offline-to-online learning. We find that offline-to-online learning converges faster than online learning, and necessarily worse than online learning in terms of final performance, as shown in Figure 11.
B.3 Horizon Limit
We investigate the horizon limit of filtered BC and DigiRL the AitW General subset. As most tasks can be effectively solved within 10 steps, we specify two horizon limits: sufficient horizont H 10, and a redundant horizon H 20. Results in Figure 12 show that redundant horizon introduces significantly faster learning speed for both filtered BC and DigiRL, presumbaly because longer horizon means more opportunity to try single trajectory. In both horizon settings, we observe the DigiRL offers significant speedup of around 100 trajectories over Filtered BC.
B.4 Trajectory Length
We investigate the rollout length of DigiRI. compared to filtered BC. Results Table 4 demonstrate that DigiRL consistently achieves shorter average rollout lengths compared to filtered BC across both
16

---

Online
On
Online Run2
On-On Runl
0.70
0.65
0.60
Success Rate
0.55
0.45
0.40
0.35
0.25
200
400
600
800
# #Trajectories
Figure 11: Success rate with pure online learning offline-to-onlin learning w.r.t. the number of online trajectories trained on the AitW General dataset. The starting points of curves this figure look different from the main results figure because the starting points of the main results figure is smoothed at the average performance of the offline trajectories collected for the offline-to-online learning.
Filored BC-30
0pfi-20
OFT-4/
Dyffus
Success Rate
64-
200
450
TTrjeectories
Figure 12: Success rate with different horizon length (H E (10, 20) junder different methods the AiTW Google Search task set.
subsets. This observation holds true whether considering rollouts for computing this correlation or only investigating this correlation on rollouts that eventually succeed. This indicates the capability of DigiRL solve tasks in a more efficient and directed manner. Qualitative examples can be found in Figure 16.
C Qualitative Examples
C.1 Random sample of trajectories for different agents
In Figures 13 and 14, we provide trajectories of DigiRL, AutoUI, and GPT-4V randomly sampled from our test set to offer qualitative understanding of the agents' performance. As shown these examples, DigiRLcan efficiently carry out in-the-win wild device control tasks and less likely to get stuck or get to a wrong page compared to AutoUI and GPT-4V.
C.2 Error Recovery
We observe that DigiRL is able to recover from its own mistakes. As shown in Figure 15, we find that DigiRL explores ways to get back to the original screen order to perform search. As a comparison, AutoUl fails to reset to the original screen and gets stuck at the diverged screen. Under the hood, we find DigiRL. trying to maximize the state value, which usually induces to reset the original screen (that has large value success).
17

---

What the new products by Samsung?
DigiRL:
AutoUI:
Got
stuck
GPT-4V
Show me some nice wallpapers my tablet
DigiRL:
AutoUI:
GPT-4V
Stops
Early
Figure 13: Agents' trajectory on two randomly sampled tasks on the General split of AitW.
18

---

Go costco.com, search for 'macbook pro,, and select the first entry
DigiRL:
Got stuck
AutoUI:
Early
stop
GPT-4V
HOT BUYS
GOL.
stuck
Go to now, search for duracell triple a'
DigiRL:
Supped
AutoUT:
Wrong Page
GPT-4V
Skpped
Could no
search
Figure 14: Agents' trajectory on two randomly sampled tasks on the WebShop split of AitW.
19

---

g Go to bestbuy.com, search for 'macbook'
DigiRL:
RACTVAEE RECESSAL
W
Bay Surmendi
Skipped
AutoUl:
Skipped
Got
Stuck
Figure 15:
Error recovery cases. In bestbuy. comm we systematically find DigiRL able to recover from its own mistakes, while AutoUI fails to do
C.3 Trajectory Length
Qualitative example on the number of steps in trajectories of DigiRL. and filtered BC are shown in Figure 16. We find consistent cases where DigiRL. has shorter trajectory length than filtere BC.
C.4 Reasoning failure of GPT44V
The performance of GPT-4V failed on AiTW tasks predominantly due to not being able to carry out high level, and then not being able to recover from these mistakes.
control actions as it plans Moreover, one of the main reasons why it is not able to recover from mistake is that it might hallucinate and make itself believe that it is wrong app website. Indeed, GPT-4V constructs a plan of further actions when provided task from either Web Shopping or General dataset of AITW. Then, when it makes misclick and fails to successfully proceed in intermediate step, it might think that it actually solved that intermediate step and is in the correct app or website to execute further actions, causing the overall trajectory to fail. An example of this is provided in Figure 17. Here, we ask the model to search for an item in a webshopping website, in particular in "newegg.com com". However, the model fails to proceed that website due to not being able to precisely locating the search button. Then, instead of trying to go to that website again, the model thinks is already in that webshopping website, and mistakes the search bar of Google with the search bar of comm". Hence, the rest of the trajectory also fails. Another slightly different phenomenon is illustrated Figure 18. Here, the model is able to proceed the correct website and search for an item, but this time it fails to tap on the search button on the website and clicks to an advertisement instead. Consequently, the model fools itself to think it successfully searched the item, and scrolls the page hoping to find that item, but it cannot do so because in reality it views the results of the advertisement. The primary reason of these failures is the challenge of grounding the control actions in GUI interfaces to realize the intermediary goals laid out by GPT-4V model's 's thoughts. As an example, we provide an illustration of trying to set up an alarm task in Figure 19. Here, in the last frame, it fails to execute the precise movements in the necessary amount of rounds to correctly set up the alarm to the desired time, and in the last frame we see that the action taken does not align with the thought process of the model.
20

---

ehay.com, thinkpad"
DigiRL
Filtered
Search for flights from Scoul to Mexico city
DigiRL
Piltered BC
Figure 16: Examples where DigiRL. has shorter trajectory length than online filtered BC.
D Fine-grained failure modes
In Figure 20, we present a more fine-grained breakdown for six failure modes provided in the user study. Those failure modes include:
Failure to recover from mistakes refers the scenario where the agent made mistake that led to states from which it failed to quickly recover and resume the task, such as a wrong google search page.
Failure to click on the right link or failure to click refers to the failure mode where the agent either fails to locate the element that it tries click on and keeps clicking on the nearby region, or fails to start typing in the string when it is supposed to do so.
Failure to take reasonable attempts at all refers to the failure mode where there is no clear reason that the agent fails to complete the task and does not seem to be the right track throughout the trajectory.
Quit or press HOME early refers to the failure mode where the agent decided finish the task or press HOME to start over before the task is actually finished.
Stops at wrong but relevant page refers to the failure mode where the agent arrives at a wrong page and mistakenly thinks that it had completed the task. For example, the agent finds machook on costco.com while the instruction asked to find a macbook on ehay.com.
Technical issues refer to the failure mode that either the task is impossible (e.g. the tasks asks to open Amazon app but this app is not installed) or the agent is temporarily blocked from a certain website due to frequent visits.
The translation between fine-grained failure modes and coarse grained failure modes is presented in Table 5.
21

---

Go to newegg.com, search for "alienware area 51" and select the first entry.
Thought: To
Thought To
Thought
Thought: To complese givers
Thoughe To
Chought
complete the glven took, ahhual leunch the Googll Chrone brommer navigane newegg com, The alement lobeled with number is docamentod
completee the given taok, the neec stop should du in untte the
takk the giren should clear the current bett the edätres bur and enter the search term bitsmoree anea Ei'
complete the
task, the should navigate to the Newegg website by tapping the seanch gn button aesociated with the untured USL.
the should is
URE fueweag cout" lnto the address bar.
submii the search query ullenweze area 5t on the Newegg welsite. Since the soorch wern aloudy exsered, seed to sup the soorch or go button initiate eearch.
'alieroazee search that sprcifis groduct newegg. com.
Action:
beatt cast"aewwggg.c. soun))
since the browser has already
app lauselie
Google Chrome.
nevigated to the Newwgg welliite.
Action: txp(111)
tap(5)
Action: tigi2)
Astions
tost alienwas ance
Actions tag(s1))
Figure 17: Failure of GPT-4V, with its thoughts and link-based actions given. A typical cause of failure that it cannot tap on the correct "search" button after entering a query and mistakenly tapped onto the "x" symbol in the search bar the "scarch" button. Here the goal is: Go to newegg.com com, search for "alienware area 51" and select the first entry. As seen in red emboldened actions, it fails to press search button and deletes the query instead. Also, seen in red highlighted parts in thoughts, it thinks it is in "newegg.com website even though it is not.
costco.com, search "acer predator", and select the first entry.
wenget
wiight
HOT BUYS
HOT BUYS
TUWN
Thought compiete
Thought: complltte
Thougl: To complec the gheen tacs, the neat stap abould term gewdatur' bus using inpul fanction.
Thought complete the give taok, next stop ahould de is search togping beyhoard.
Thought: complets tas
Thogght To complete the given tass, should continus swige up prodiet listings find prodater
given
pren
"oostro lnto
ishoul
should is Castos vebhatu. Since
Hilings fadd the peoduror ttem. Since first what need to select, I should svipe up on the stement labeled '2a' view more producte..
search since Laave
already inillated the process masigatt
adrees "coettocomn" is
aloady untared in the address bus seed to initaae the web search or pags tapping the
coston coui by lapping on the Goople rearch bar.
Action: awriget21, "ep".
Aucion: centiacce Tacer predator?)
Aotion: sap(24)
"meediun"")
keyhoard.
Action: swipet26, Tep?
Action: sxp44
Figure 18: Failure of GPT-4V with its thoughts and link-based actions given. This time the reason for failure is misclick on the wrong button. The task is "Go costco.com. search for "acer predator", and select the first entry". Notice that up until the fourth frame in this Figure, the trajectory goes correct. But then it clicks on the generic advertisements on the Costco. com website, and it cannot recover back. It continues to scroll the page and takes wrong actions thereafter.
22

---

Set alarm for 4pm.
3:48 AM
8:30
8:30
8:30
3:49
3:49
9:00
Thoughtc To set un alorm Aprta I need find and open the apo. opp dionely
Thought: To unn alamn for fpm,, need open the agp. next stop Clock app
Thought: akemm fyms, J ssedd nasigite the alarm owetion of the clock app not altundy there, and Cless use the button add a new alarm.
Thought io alarm for PMd, shorld tup the buroon to adl new alarm.
thought: To comulete the given tosk, I Leed aberm for 4s38 PMt The hour chow to the desinel limu, so should flsst switcl from using
thought: to comslete the gloun tasly Labould set minetee so foa to maku the abomm for 4.00 PMC.
the
Action tap(l44)
tapi2e)
Action: tap(()
tap(l)
the drower
tap(()
(if prewent to view
more and loasee the alarm clock
Action: wwipeD, Tup.. "meeduus??
Figure 19: Failure of GPT-4V, with an example task on the AiTW general test set. The task is "Set an alarm for 4pm". Here, GPT-4V is able to successfully navigate to the clock app, and the alarm settings of that app. However, it cannot take the correct precise actions to set the alarm quickly enough, and fails due to maximum rounds reached. In the last round, notice that the action of tap( contradict with its own thought process of setting minutes to "00"
General
GPERY
Gemine-..5PP Pm
GITEY
Gemini-1 5-mm
Aueol
Cogigent
Omt ne
Oftine
Online
Onäna
raszonale attompts
Qui press HONE earry
Web Shopping
Technical losues
Stops wrong but eeveennt page
Aspégent
AppAgant
Dytti
DIAhL
Task succoss
CPT4Y
Gamines,2
AMtUU
Cogagent
Off
Online
Figure 20: Failure modes decomposition for each policy model for both General and Web Shopping subsets.
Fine- Grainer Failure
Searse-Grained Failure
Fail to recover from mistakes
Fail to recover from mistakes
Fail to click on the right link or fail to type
Get stuck midway
Fail to take reasonable attempts all
Get stuck midway
Quit Press HOME early
Arrive at wrong goal
Stops at wrong but relevant page
Arrive at wrong goal
Technical Issues
None
Table 5: Examples of task descriptions in the AiTW Webshopping task set.
23

---

distribute updated policy Rt
π
worker machines
aggregate trajectories
host machine
+
1-0-9
0-C-9
T
T T++1
emulators
Figure 21: Multi-machine parallel emulator execution. The host machine is equipped with GPU accelerators and the worker machines are equipped only with CPUs. The policy update executed on the worker machine and the trajectory collections are executed distributedly the worker machines and aggregated by the host machine.
E
Experiment machines
Our main experiments are conducted on VM instances from Google Cloud Platform. Each VM instance comes with 1x Tesla T4 GPU and 16x Intel(R) Xeon(R) CPU.
F
Setup for parallel environment
Running multiple emulators in parallel can be challenging due to the inefficiency in thread syn- chronization and frequent fault propagation when one emulator runs into unknown error. To address this challenge, we set up a server-client system where all emulator processes are running in independent server processes. Each emulator process communicates with the main training process through different UlAutomotor servers. The main training process sends high-level instructions to UlAutomotor servers (such as reset and step), while UlAutomotor servers parse high-level instruc- tions into low-level UI commands (such as typing a character and tapping at a coordinate) and such UI commands are executed by the emulator processes. When exception is thrown in the emulator, the UlAutomotor examines it is recoverable (e.g. an UI command takes too long to execute in the emulator) and reset the emulator process if it is not. When exception is thrown in the UlAutomotor server, the main training process stops and resets the UlAutomotor server to ensure data correctness.
This design can easily be scaled up to multi-machine setting, As illustrated in Figure 21, one host machine equipped with GPU accelerator has a local copy of the current policy Te. and distributes the policy to all worker machines equipped with only one GPU and multiple CPUs. Each worker machine will then collect trajectories of different tasks using T. After all collection processes are synchronized, the host machine gathers all the trajectories together to update the policy to πT+1- This process keeps iterating until the policy converges.
Speedup of emulation parallel. The performance boost with respect to the number of worker machines is nearly linear, as demonstrated in Figure 22 (right), where we conduct experiments that examine the scaling performance of our parallel emulator. Our distributed emulator that runs emulations across multiple servers can reliably collect data with up to 64 parallel emulators on 128 CPUs with near-linear speedup. In contrast, naive baseline that runs all parallel emulations on the same server achieves much inferior performance (0.74 compared to 1.74 trajs/min using 64 CPUs).
G Autonomous evaluator details
Our autonomous evaluator gives reward to each observation get. The observation is composed of the current screenshot of device and the task. The evaluator gives a reward of if the screenshot shows a completion of the task, and will terminate the POMDP as result result.
The optimized prompt is shown Figure 23 and Figure 24 for General and Web Shopping subsets respectively.
24

---

Boanz
Emmilaiinn Spoed (traj/min)
3.5
0.38
0.53
0.74
di CPUs
Figure 22: Emulation speed w.r.f number of CPUs used. The upper bound can only achieved when there is no communication and error handling cost. Our design of distributed emulator can significantly improve the efficiency of emulation compaared to the vanilla method of running all emulations over the same instance.
H Zero-shot Baseline Details
Figure 25 shows the prompt that we used for testing the Set-of-Marks performance for GPT-4V and Gemini 1.5 Pro. This prompt is directly taken from Yang et al. [47].
I Hyperparameters
Hyperparameters for both Filtered BC and DigiRL are carefully tuned through binary search on the training set of General and Web Shopping subsets. The final choice of hyperparameters for both methods can be found in Table 6. As shown in the table, the only hyperparameters introduced by DigiRL are supervised training hyperparameters for the value function and instruction value function (including number of iterations and learning rate) and GAE A.
25

---

Prompt
You're an expert in evaluating whether the Screenshot successfully completes the Task,
Examples=====
Screenshot: (train_ 1.pgg)
Task: Open the settings.
Q: What should expect to the screenshol if I've opened the settings?
A: should expect to see I'm in the settings app. The screenshot shows the home screen of mobile device, with various app icons displayed, including the settings app icon, but the settings is not opened.
Status: failure
Screenshot: 2rain_2.pnn)t
Task: Find hotels in washington
on screenshot if I've searched for hotels Washington, I'm search results page for hotels Washington, DC. The screenshot shows Google search page with the search field populated with the query "hotels in washington de" and a list of suggested searches related to hotels in Washington, DC, but it does not show search results for hotels in Washington, DC.
Q: What should expect
A: should expect
Status: failure
Screenshot: (train_3.pmg 3.pmg)
Task: What's a good restaurant in Portland?
Q: What should I expect to see on screenshot I've searched good restaurant Portland? should expect
I'm search results page a good restaurant in Portland. screenshot shows Google search page with search input field for "good restaurant in portland" and map results preview showing business locations Portland, like "Li Pigeon', "Portland City Grill", and "Higgins",
Status: success
(more cases)
Your Turn
Screenshot: (tost.pog)
Task: (aask_hiss_traaj
Respond this format:
Q: What should expect to the screenshot if I've crepeal the task
A: should expect to see <first expectation, then what's in the given screenshot,>
Status: success or failure (don't return anything else) Start with "Q=".
Response
Q: What should I expect to see on screenshot searched price should expect to the price of 12' ladder Home Depot; the screenshot shows search result page the p rice of 12' ladder, with product advertisements showing prices from Home Depot Status: success
Home Depot?
Image Sources
Google
Houea dera or Landery
TEGDTA
train_L.png
train_2.png
train_3.png
test.png
Figure 23; Prompt for our autonomous evaluator for tasks in AitW General subset.
26

---

Prompt
You're an expert in evaluating whether the Screenshot successfully completes the Task.
Examples==s=
Screenshot: (train_1.png
Task: Go to bestbuy.com
Q: What should I expest to see on the screenshot I've gone to bestbuy.com?
should expect to see I'm in the Best Buy website, which usually shows the best buy logo with some featured prod and categories. The screenshot shows I'm searching for "bestbuy.com" the Google search (with some search uggestions) instead of being in the Best Buy website. Status: failure
Screenshot: (train_2-png)
Task: ebay.com, search 'corsair
What should I expect to see the screenshot gone ebay.com. AND searched for 'corsair k70"? A: should expect to see I'm in the eBay website and search results for "corsair k70". The screenshot shows I'm in the eBay ebay website with some search suggestions for "corsair k70", but it does not show search results of the product, which usually includes price and the product details.
Status: failure
Screenshot: (train_3.png)
Task: Go ebay.com, search for "lenovo thinkpad"
Q: What should I expect to see on the screenshot I've gone to ehay.com AND searched for "lenovo thinkpad"? A: should expect to see I'm in the eBay website and search results for "lenovo thinkpad'. The screenshot shows I'm eBay website and have several search results "lenovo thinkpad".
Status: success
(more cases)
Your Turn
Screenshot: (test.png)
Task: (task, this, traj)
Respond in this format:
Q: What should I expect to see the screenshot if I've crepeat the task??
A: should expect to see <first expectation, then what's in the given screenshot.> Status: success or failure (don't return anything else) with "Q:".
Response
Q: What should I expect to see on the screenshot I've searched for the price of 12' ladder at Home Depot? should expect to see the price of a 12' ladder at Home Depot; the screenshot shows a search result page for the p rice of a ladder, with some product advertisements showing prices from Home Depot Statust
Image Sources
PAB
WAE
TELL1
BRAWARRI KETAY ALHONAA
2004
train train.1.ppr 1.png
train train.2.png 2.png
train. 3,png
test.png
Figure 24: Prompt for our autonomous evaluator for tasks in AitW Web Shopping subset.
27

---

Prompt
"You are an agent that is trained to perform some basic tasks on smartphone. You will be given a Insmartphone screenshot. The interactive UI elements on the screenshot are labeled with numeric tags starting from 1. The Innumeric tag of each interactive element is located in the center of the elementipianou can call the following functions to control the smartphone/nint.. tap(element: int)s function used tap an element shown on the smartphone screen. felemmet" is a numeric tag assigned to an UI element shown on the smartphone screen. InA simple use case can be tap(5), which taps the UI element labeled with the number 5.m))22 text(text input: str) function used to insert text input an input field/box. text, input is the string you want to insert and must (nbe wrapped with double quotation marks, A simple use case can be text("Hello, world!"", which inserts the string Hnl'Hlllo, world!?' into the input area on the smartphone screen. This function is usually callable when you see keyboard (nshowing in lower half of screen.nnj33 long- long-press(elemee pressseeeeeeent int)/nThis function used to long press an UI element shown on the smartphone sereen.|mm"elee ellmmeemett"" is numeric tag assigned to an element shown on the smartphone screen.mA simple use case can be long-press(5), which long presses the element labeled with the number 51m)n4, swipe(element: int, direction: str, dist: stri\nThis function used swipe an element shown on the smartphone screen, usually a seroll view or slide barl)//elemment is a numeric tag assigned to an element shown on the smartphone screen. ('drrcttoon" a string that |nrepresents directions: up, down, left, right. \'directionl" must wrapped with double quotation |nmarks. \'disti" determines the distance of the swipe and be one of the three options: short, medium, long. You should (nchoose the appropriate distance option according to your need,nA simple use can be swipe l'up",, ('medium"", which swipes up the element labeled with the number for 'mmedium distance)mins, gridO should call this function when you find the element you want interact with not labeled with numeric tag and inother elements with numeric tags cannot help with the task. The function will bring up a grid overlay divide the Insmartphone screen into small areas and this will give you freedom choose any part of the long inpress, swipe.
of the four
tap,
The task you need to complete is to How much does 2 bedroom apartment rent for in Denver?.
Your past actions to proceed with this task are summarized as follows: None
Now, given the documentation and the following labeled screenshot, you need to think and call the function needed to proceed with the task. Your output should include three parts in the given format:
Observation: <Describe what you observe in the image>
Thought: <To complete the given task, what the next step I should do>
Action: <The function call with the correct parameters to proceed with the task. When you are certain that the task is successfully done and the goal is reached as of the current observation, you should output FINISH. You cannot output anything else except function call or FINISH Inin this field.>
Summary: <Summarize your past actions along with your latest action in one or two sentences. Do not include the numeric (ntag in your summaryzinYou can only take one action at time, so please directly call the function."
Figure 25: Set-of-Marks prompting- The boldened inputs can be changed according to our goal. The task changes for every different task. The past actions change as we take actions (it is None now since this is the prompt for the first round).
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction explicitly state the contribu- tions of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
28

---

Table Hyperparameters for All Experiments
Method
Hyperparameter
Offline
Offline-to-Online
actor Ir batch size rollout trajectories replay buller size rollout temperature maximum gradient norm actor updates per iteration
3e-3
3e-3
128
128
16
Filtered BC
5000
1.0
0.01
0.01
number of iterations for offline actor updates actor Ir value function Ir
10
3e-3
3e-3
3e-3
3e-3
instruction value function Ir instruction value function Ir batch size rollout trajectories replay buffer size rollout temperature maximum gradient norm
3e-3
3e-3
3e-3
3e-3
128
128
16
5000
DigiRL
1.0
0.01
0.01
GAEA À
0.5
0.5
actor updates per iteration value function updales per iteration instruction value function updates per iteration
20
number of iterations for offiine actor updates number of iterations for offline value function updates
10
20
20
number of iterations for offline instruction value function updates
20
Table 7: Hyperparameters for DigiRL and Filtered BC on both General and Web Shopping subset of AitW..
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: Limitations are discussed in the last section of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be
29

---

used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not provide theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
All loss functions and implementation details are provided in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
30

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are still actively cleaning the code and make the environment more accessible to broader audience. Once we done with that, we will open-source the code along with the release of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
public/gu Soaidee/CooeSSumissiiiic
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /niiss cttppcppppcccccccccccccccccccccc httgg/udds//odeeeeee
ssionPolicy Poylyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Dataset details are provided in Appendix A. and hyperparameters are provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
31

---

7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Repeated experiments are carried out with their means and standard deviations reported in Table 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is provided in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /t/pt///es///////eecoo///ooooole http://wwrrpppecoooo/o///tctctta ines?
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeulPS code of Etics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
32

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The positive societal impacts are discussed in the Introduction while the negative societal impacts are discussed in Section 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymennt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The capability of the model that will be releasing is limited to simple tasks in Android in the Wild dataset, and therefore does not have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
33

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited the assets that are using.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com/ com/dataser datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
This submission does not include new assets. New assets including open- sourced code, model checkpoints, and model trajectories will be released with documentation when we release the paper.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: This research does not involve crowdsoureing or human subjects. Annotations of trajectories in Figure 7 and Figure 8 are carried out by authors alone.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
34

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This research does not involve crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "FPlVG0wNDL", "reviewer": "Reviewer_JuUV", "summary": "This paper tackles AI agent training for controlling digital devices (e.g., web navigation). The proposed framework, named DigiRL, is a 3-stage training process consisting of model pre-training, offline fine-tuning (offline RL), and online fine-tuning (online RL). To achieve this goal, the authors first build a parallelizable Android learning environment that enables fast online interactions for policy learning; they then adopt a VLM-based evaluator to provide reward signals for the agents; finally, they perform ablation studies to examine several key design choices in typical policy-based RL methods for the third stage. Compared to larger models trained without this stage, the proposed approach enjoys significant performance enhancement due to the online fine-tuning stage.", "questions": "- Will the proposed method further scale well with more online interactions?\n- While multi-turn interactions are a challenge of the device control problem, is there any component in the proposed framework that specifically helps to tackle it?\n\nI am willing to adjust my ratings after seeing the authors' responses.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The authors did a good job introducing the background, the problem setup, the baselines, and the details of their proposed method.\n+ Fine-tuning large VLMs in an online fashion can be challenging; the performance improvement obtained by the proposed method, which is relatively simple, is substantial and the overall approach looks promising.", "weaknesses": "The main issue is the limited comparison with online RL methods for fine-tuning LLM-based agents. The only RL method compared in the experiments is Filtered BC (besides vanilla AWR, which the proposed method is based on). Filtered BC is strictly speaking not an online RL method. Admittedly, AI agent training for device control is a relatively under-explored new area, and the authors claim that theirs is the first successful offline-to-online RL approach for device control AI agents, I believe more experiments with other online RL baselines not originally designed for device control is required to justify DigiRL's advantages. For example, the classic on-policy methods such as REINFORCE and PPO, or the more recent ones that are more sample-efficient, such as [1, 2] (which might be considered more-or-less recurrent work, though). Further comparisons also help to provide more insight into the unique challenges of the device control problem for digital agents.\n\n[1] Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents\n[2] REBEL: Reinforcement Learning via Regressing Relative Rewards", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback on our paper. To address the main concern regarding comparisons, we provide additional results and appeal to comparisons from prior work to demonstrate that DigiRL outperforms several online RL methods. These comparisons include REINFORCE, PPO, and Q-value based methods. We also provide **additional results** to show the favorable scaling of our method with more interactions. **Please let us know if your concerns are addressed and if so, we would appreciate it if you might be willing to upgrade your score. We are happy to discuss further.**  We answer your questions below:\n\n___\n\n### **[New result] Comparisons to other online RL methods**\n\nWe also provide new results for a comparison against several online RL methods.\n\n1) **REINFORCE**: We note that policy gradient via REINFORCE reduces to an online filtered BC loss when the rewards are given by +1 and 0, as a result, our filtered BC results in the paper already indicate that DigiRL outperforms REINFORCE. To see this equivalence between REINFORCE and online filtered BC, note that: \n$ L_\\text{REINFORCE} = E_{\\tau \\sim \\mathcal{D}}(\\sum_{h=1}^{H}r_h)(\\sum_{h=1}^{H}\\log \\pi(a_h|s_h))$ (surrogate for REINFORCE)\nand \n$L_\\text{Filtered-BC} =E_{\\tau \\sim \\mathcal{D}}\\mathbb{1}\\{\\sum_{h=1}^{H}r_h > \\text{t}\\}(\\sum_{h=1}^{H}\\log \\pi(a_h|s_h))$, for trajectories $\\tau$ and threshold $t$.  \n\nSince our reward takes a binary 0/1 value at the end of a rollout; both $(\\sum_{h=1}^{H}r_h)$ and $\\mathbb{1}\\{\\sum_{h=1}^{H}r_h > \\text{thresh}\\}$ will only evaluate to 1 if the trajectory is successful or be 0 otherwise, resulting in an identical multiplication factor for both REINFORCE and online filtered BC. Our results already show that DigiRL outperforms online filtered BC, which should imply that DigiRL outperforms REINFORCE as well. \n\n2) **PPO**: We have run experiments comparing DigiRL with online PPO, and we found that DigiRL is more efficient than PPO. The efficiency of DigiRL compared PPO stems from 1) DigiRL starts from an offline RL checkpoint so that it maintains an initial advantage over PPO via the use of offline data, and 2) DigiRL is able to make use of off-policy data to stabilize training while PPO always updates on the small batch of on-policy data. The inefficiency of on-policy PPO corroborates findings in recent multi-turn RL literature [1,2]. We will run this comparison fully and add this baseline in all settings.\n\n3) **Q-value based actor-critic methods.** We have also tried some other designs involving Q-function training [1, 2] for the step-level advantage estimator and found that training a Q-function in all of our experiments obtained inferior performance. Concretely, we attempted to use $Q(s, a) - V(s)$ to compute step-level advantages following Zhou et al. [2]. In this case, we were not able to get the Q-function to correctly pay attention to the action input, leading to Q and V collapsing to very similar values everywhere. We hypothesize this is because it is relatively easy to understand what elements are present in a screenshot, but learning how an action, which appears as a small change on the screen affects the screenshot is more challenging because this requires inferring relationships between precise relative locations of each element and clicking coordinates.\n\nAs shown in Table 1 of the 1-page PDF for the Web Shopping subset, the design choice of using $V(s’) + r - V(s)$ to calculate one-step advantage to instead of $Q(s,a) - V(s)$ led to better offline RL performance by 9% and reduced variance by 5.9%. Of course, this does not mean that Q-functions cannot be trained on this task, but that within the time of the rebuttal period, we found it quite hard to get reasonable Q-functions needed for value-based online RL methods.  \n\nFinally, thanks for bringing up the concurrent work that we will discuss. We are happy to explore such methods in an updated version of the paper.\n___\n\n### **[New result] Scaling with more online interactions?**\n\nIn Figure 8 of the submission, we provide a learning curve plotting performance as more online data is collected. We also include a new result Figure 1 of 1-page PDF, where we set the agent to be updated with even more online data for four days after convergence. Compared to a frozen policy, the agent trained with more online interactions data maintains a stable performance despite the changing nature of the websites and the device state, while the performance of a frozen RL policy gradually decays as time goes on. **This indicates that DigiRL utilizes online interaction effectively.**\n\n___\n\n### “While multi-turn interactions are a challenge of the device control problem, is there any component in the proposed framework that specifically helps to tackle it?”\n\nThe design of the doubly robust estimator for estimating step-level advantage that balances bias and variance is specifically useful in multi-turn settings, in stochastic environments. Such a design for variance reduction has been shown to be unnecessary in the single-turn setting [3], but our prior results on comparing a one-step advantage estimator $V(s’) + r - V(s)$ and doubly-robust advantage estimator (Eqn. 4.3 in paper) show that it is critical.\n\nLikewise, the use of a curriculum is especially important in highly multi-task environments, where training uniformly on all initial states is likely to not provide a strong signal to update the policy [4] (as shown in the comparison between “Ours w/ step-level advantage” and “Filtered BC” in Fig 7 in paper). We will clarify this.\n\n\n[1] Song, Yifan, et al. ‘Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents’. ACL 2024\n\n[2] Zhou, Yifei, et al. ‘ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL’. ICML 2024\n\n[3] Ahmadian, Arash, et al. ‘Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs’. \n\n[4] Jiang, Minqi, et al. ‘Prioritized Level Replay’.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FlRa24qJId", "reviewer": "Reviewer_XhdC", "summary": "This paper proposes an autonomous RL approach, RL for digital agent (DigiRL), to finetune a pretrained VLM as an in-the-wild device control agent through GUI. The authors build a parallelizable Android learning environment with VLM-based evaluator to identify the key design choices for RL. The training include two stages an offline RL phase on existing data, then followed by an online RL phase by interacting with real-world graphical user interfaces using the Android learning environment. The proposed method with only 1.5B model size outperforms other state-of-the-art models such as GPT4-V or 17B CogAgent in the Android-in-the-Wild (AitW) dataset.", "questions": "- Could the authors compare more advanced LLM reasoning or planning algorithms like Chain of Thoughts (CoT), Tree of Thoughts (ToT), Reasoning as Planning (RaP), etc.?\n- Following the previous question, is it possible to compare with those planning/search-based methods with the autonomous evaluator or the trained value model of DigiRL as value functions?\n- What is the training time and compute requirement for online training?\n- In Figure 7, what does the AWR reweighting mean? Is it simply AWR?\n- With the auto-curriculum setup, it may be interesting to look at what types of data/replay are critical throughout the online learning process; with simple categorization like failure mode in Figure 5 or whatever characterization can be interesting.\nAs detailed in the \"Challenges of stochasticity\" paragraph in section 3, could the authors provide some studies on unpredictable distractor and technical glitches?", "limitations": "Limitations are discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-structured and easy to follow.\n- Many design choices are well motivated.\n- The experiments are nice and well support the claims.", "weaknesses": "- Overall there is no major weakness. There are only several questions and potentially interesting empirical studies to look at. Check more details in the question sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and feedback on the paper. To address the raised questions, we add new results to include comparisons with  **CoT based planning**, and **a state-of-the-art LLM planning algorithm for device control called AppAgent [1]**. We also provide **additional results** for the auto-curriculum setup and ablation studies illustrating the challenge of stochasticity and dynamism for device control. We also provide clarifications on the rest of the questions, and will incorporate these results and clarifications in the final version of the paper.\n\n**Please let us know if your concerns are addressed and if so, we would appreciate it if you could upgrade your score. We are happy to discuss further.**  We answer your questions below:\n\n___\n\n### **[New result]** “Following the previous question, is it possible to compare with those planning/search-based methods with the autonomous evaluator or the trained value model of DigiRL as value functions?”\n\nGood question! In Table-3 of one-page PDF, we have included additional results of comparing planning/search-based methods with the autonomous evaluator. In particular, we compare with Reflexion[2] that for each task, first reflects on a trial run with the result given by the autonomous evaluator and performs a new trial with the reflection included in the prompt. We found that the use of Reflexion+autonomous evaluator can indeed enhance the performance (comparing GPT4V Reflexion+Set-Of-Marks 14.6% and GPT4V Set-Of-Marks 8.3%). However, **this approach still performs worse than our method DigiRL (14.6% compared to 67.2%)**.\n\n___\n\n### “Could the authors compare more advanced LLM reasoning or planning algorithms like Chain of Thoughts (CoT), Tree of Thoughts (ToT), Reasoning as Planning (RaP), etc.?”\n\nThanks for bringing this up. We want to clarify that several baseline results in the submission were already prompted with a chain of thought. For instance, the “Set-of-Marks” approach with GPT4V and Gemini-1.5-Pro, and CogAgent are prompted with Chain of Thought (CoT) to produce an action. With regards to a SoTA planning baseline, we remark that we also compare DigiRL to AppAgent [1] in Table 1, which is a state-of-the-art VLM Retrieval Augmented Generation (RAG) approach specifically designed for device control. In both cases, we find that **DigiRL outperforms these prior methods**, indicating the superiority of training with autonomous RL for user-level device control over planning / prompting with frozen models.\n\nThe methods discussed above constitute the state-of-the-art planning based approaches for device control. We are also happy to add any other comparisons if the reviewer has particular pointers for existing approaches that use planning / reasoning in the device-control domain. \n\n___\n\n### **[New result]** With the auto-curriculum setup, it may be interesting to look at what types of data/replay are critical throughout the online learning process; with simple categorization like failure mode in Figure 5 or whatever characterization can be interesting. \n\nWe have run additional experiments to understand what types of data are being trained upon during the course of online learning. In particular, we categorize the tasks in the Web Shopping subset into three levels of difficulties according to the criterion in Table 3 in appendix, and plot the learning curve for each difficulty level during the online learning process in Figure 5 in Rebuttal PDF. From this plot, we can see that the performance of difficulty 1 (easy) tasks improves significantly in the first 200 trajectories indicating that much difficulty 1 data is trained upon in the very beginning. In the later stage of training, the performance of difficulty 1 tasks stay around the same while the performance of difficulty 2 and difficulty 3 tasks increase a lot, indicating data from difficulty 2 and difficulty 3 tasks is replayed more at this stage. This indicates that the auto-curriculum helps us upweight the right trajectories to focus at different stages of training.\n\n___\n\n### **[New result]** As detailed in the \"Challenges of stochasticity\" paragraph in section 3, could the authors provide some studies on unpredictable distractor and technical glitches?\n\nWe have run an additional experiment where we take a trained DigiRL agent and continue to train it further with more online data for four days wall-clock time, even though the previous  training run that produced that agent had already converged. As shown in Figure 1 in the 1-page PDF, as more time passes the performance of the previously-trained DigiRL policy begins to drop. On the other hand, despite having converged in the previous training run, continuing to update the DigiRL agent with more online interactions lead to more robust and stable performance. We believe that this experiment precisely illustrates the issues with stochasticity and dynamic nature of websites: the decay in performance of the optimal policy trained previously underscores the challenge of dynamism and stochasticity of the dynamics of the device control setup, and stable performance with continued DigiRL training demonstrates the efficacy of our approach in enabling performance despite these challenges.\n\n___\n\n### In Figure 7, what does the AWR reweighting mean? Is it simply AWR?\n\nYes you are correct it is simply AWR. We call it reweighting as it uses reweights to “soft filter” advantageous actions while we use a “hard filtering” for implementation simplicity.\n\n___\n\n### What is the training time and compute requirement for online training?\n\nFor our main experiments, we are able to run 8 emulators in parallel on a 16GB T4 GPU with 32 CPUs, and finish an online training run with 1k trajectories within 3 days. Figure 16 in appendix illustrates a relative speed up obtained if we use more resources. For example, if we use 128 CPUs and 4 GPU, we can achieve a speed up of 3.55x when it is set up properly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QB3MDukB0X", "reviewer": "Reviewer_251X", "summary": "This paper introduces a novel autonomous reinforcement learning (RL) approach, DigiRL, for training in-the-wild device control agents. DigiRL first employs offline RL to fine-tune a pre-trained vision-language model (VLM as the agent) using stale task-specific data, and then further refines the agent through online RL by continuously interacting with parallelized emulators. DigiRL achieves a 49.5% absolute improvement in task success rate over existing state-of-the-art agents, establishing a new benchmark for digital agents in device control.", "questions": "1. In line 250-251, the CE loss pairs r with log V and (1-r) with log (1-V). Intuitively, this means one would like to make the distribution of r and V as close as possible (when r-> 1, V -> 1 and vice versa). This seems to contradict the claim in line 240-241 that \"Intuitively, if a rollout attains a high value of A(sh, ah, c), it means the value function V is small\".\n\n2. How do you perform the train/test task split, if not random split? It is odd to see in Table 1 that almost all testing performance clearly surpass the training performance (normally should be the opposite), which suggest that the testing tasks are in general easier than the training tasks and not i.i.d?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and motivated, many important technical/implementation details are covered.\n\n2. The paper considers a challenging problem, autonomous device control, where existing LLM-based methods struggle to achieve acceptable success rate. The proposed method leverages VLM and RL techniques and significantly improves compared to these baselines.\n\n3. The experiments are comprehensive and informative, covering LLM/RL agents, prompting and learning paradigms, offline and off-to-on RL, as well as failure modes analysis.\n\n4. The authors implement a multi-machine emulator system to support parallel and real-time training of online RL.", "weaknesses": "Major Points:\n\n1. From a ML methodological point of view, the novelty/contribution of the paper is limited. To perform offline and off-to-on RL, the paper adopts a number of existing techniques such as AWR, doubly-robust estimators with little customization (e.g.,  hard filtering on the advantages instead of computing $\\exp(A)$, which is mainly indended for easier implementation), all well-known to the community. The only thing seems \"new\" is training value functions with cross-entropy losses, also directly taken from [1], and the equations in line 250-251 seems to be qustionable (see my comments in the Questions section). Moreover, no theoretical insight is provided to elucidate why these specific designs are chosen.\n\n[1] Stop regressing: Training value functions via classification for scalable deep rl, 2024.\n\n2. Limited Scope. The entire paper focuses on a very specific domain (autonomous device control). The scope of the proposed method might be too narrow to be of general interest to the ML/RL community.\n\nMinor Points:\n\n- In section 4.2, how to properly balance the two estimators, one with higher variance and one with higher bias to achieve the optimal result? What's the hyperparameter profile of the combined estimator? Have you tried any alternative designs and can you give theoretical insight to justify this specific design choice?\n\n- Regarding the offline and off-to-on RL setting: to my knowledge, the main point of offline RL is to leverage a large body of stale data to safely and efficiently pretrain a RL agent. Therefore, for off-to-on RL, where online RL operates as the fine-tuning stage, one should use data far less than the offline pretrained dataset to ensure the setting is meaningful. The fact that authors intentionally use the same amount of data for both offline and online stage, which assumes access to a large amount of online data might make the offline pretraining unecessary. To see this, I recommend the authors to directly perform online RL on the combined dataset and it's highly possible that such \"purely online\" agent outperforms its off-to-on counterpart. \n\n-  The authors spend quite a few words discussing the challenges of stochasticity and device control as a POMDP. However, I do not see any specific design or techincal contribution targeting such problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review! At the outset, we want to clarify our scope: our goal is to show that autonomous RL can be used to build SOTA device control agents that outperform proprietary models (Gemini/GPT-4). Our methodological contribution involves identifying and designing a good RL objective to be able to do so (robust advantages + curriculum + cross-entropy loss + AWR). We believe that our scope of device control is akin to several prior ICML/ICLR/NeurIPS papers that focus on applying ML / RL techniques to one domain (e.g., RL for chip design (ICML) [6], LLMs for web navigation (ICLR)  [7], web shopping (NeurIPS) [1]), and were still judged to be valuable contributions. In fact, device control is already more general than several important agent domains (e.g., shopping, web navigation, traveling planning, etc) that have been considered individually. We therefore think that an RL approach that attains SoTA **for the first time**, in a more general setting than recent papers, should not be a ground for rejection. To address the questions, we add **new results for hyperparameter profiling, advantage design, online RL from scratch, and \nstochasticity**. *Please let us know if the concerns are addressed and if so, we would be grateful if you upgrade your score.*  \n___\n\n### “Limited Scope; From a ML methodological point of view, the novelty/contribution of the paper is limited.”\n\nAs mentioned, our work is already in a more general setting than work in foundation agents that appears in ICML / NeurIPS / ICLR [1,2,3,4]. We use two subsets of AitW that focus on different parts of device control with around 200 tasks each (web shopping, device management; see Tables 2, 3). For the RL community, we identify challenges in a real-world problem setting of device control (e.g., see Fig 1, Table 1 in the PDF), and design an RL approach that can efficiently learn in this environment by combining robust advantages, curriculum, cross-entropy losses & AWR. While each piece individually is not as novel, combining each piece into an effective system for a real-world, user-scale problem is our contribution. As a systems paper (that has been of interest in ML/RL [5,6,7]), we think we should be evaluated on the efficacy of our system rather than a novel RL algorithm. \n___\n\n### “Step level advantage estimator”\n\n**Alternate designs:** We tried using a Q-function $Q(s, a)$ for computing advantages, but this leads to significantly worse and less stable results (see Table 1 of the PDF). This is because the action coverage is not high enough to  properly attribute reward signals to the action instead of states.\n\n**The theoretical justification:** follows the analysis of GAE[8], where the one-step estimator $V^{step}(s_{h+1}) + r(s_h, a_h) - V^{step}(s_{h})$ corresponds to the high-bias estimator $\\hat{A}^{(1)}$ in Eq. 11 of the GAE paper and the MC reward estimator $\\lambda^{H-h}r(s_H, a_H, c)$ corresponds to the high-variance estimator $\\hat{A}^{\\infty}$ in Eq. 15 in GAE. While GAE takes an average of a series of k-step estimators, we choose to omit the intermediate estimators for simplicity, nonetheless we enjoy similar bias-variance trade-offs. Similar to the GAE[8], the combined estimator is $\\gamma$-just (Proposition 1[8]) when $V^{step}$ is accurate, i.e.\n$E_{s,a \\sim d^\\pi}A^{step}(s,a)\\nabla \\pi_{\\theta}(a|s) = E_{s,a \\sim d^\\pi}A^{\\pi}(s,a)\\nabla \\pi_{\\theta}(a|s)$, where $A^{\\pi}$ is the true advantage function. \n\n**Hyperparam tuning:** The only hyperparameter is $\\lambda$, which is tuned similar to the GAE $\\lambda$. We provide a result in  **table 2 of 1-page PDF** ablating $\\lambda$ from 0.0 to 0.9, and found DigiRL to not be very sensitive to it.\n___\n\n### Benefits of offline-to-online RL\n\nWe ran online RL from scratch in Figure 3 of the PDF. We see that our off-to-on agent smoothly transitions into the online phase, without any unlearning + results in a lower cumulative regret than online RL from scratch. Avoiding unlearning while benefitting from a better init is crucial in scenarios where the desideratum is to keep adapting the agent quickly upon deployment.\n___\n\n### Handling stochasticity and POMDP via DigiRL\n\nTo show the challenge of stochasticity and dynamism, we add Figure 1 in 1-page PDF comparing the performance of a stale offline DigiRL agent and an agent updated via online. Despite being trained with RL, performance of a stale agent decays as time moves on, whereas continually training with online DigiRL avoids this issue, hence DigiRL addresses performance drop amidst stochasticity. \n\nWe also clarify that utilizing a POMDP formulation is important because at some instants, the exact state of the device may simply be unknown (e.g., when a page is loading it is impossible to know what is loading without referring to the previous state). Our practical implementation handles this by using the history of the last two screenshots as the state of the RL agent (see Lines 199-200).\n___\n\n### Line 250-251, clarification about r, V and cross-entropy\n\nThis would not make r and V close because r is a function of both s & a, but V is only a function of the instruction. As long as the reward values for all (s,a) for a given instruction are not 1, then, $V$ will take a value smaller than 1. Concretely, $V$ is the average $r$ for all (s, a) pairs at the same instruction. Hence, this **does not** contradict the claim in 240-241: say, the agent has 10% of success rate for a particular instruction, then $V=0.1$. For successful rollouts, $A(s_h, a_h, c) = r(s_H, a_H, c) - V^\\text{instruct}(c)=1-0.1=0.9$. But if the agent has 30% of success for an instruction, $A(s_h, a_h, c)=0.7$ for a successful rollout, because now $V=0.3$. \n___\n\n### How do you perform the train/test task split\n\nWe use the standard train/test task split from AitW. Train and test tasks are generated following a set of same templates such as “Go to {shopping website}, and search for {item name}”, which might allow for generalization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6pVuEEhqBh", "reviewer": "Reviewer_zDzF", "summary": "This paper shows the potential of Reinforcement Learning (RL) for designing an effective digital agent for in-the-wild control through Graphical User Interfaces (GUIs). The proposed approach relies on the advantage of the pre-trained visual language models (VLMs) while tackling real-world stochasticity by training an RL agent that interacts with an environment instead of relying on static demonstrations. Accordingly, this work proposes a novel autonomous RL approach, namely DigiRL, for training device control agents, which consists of two stages: an offline RL phase where the agent is trained on static demonstrations, followed by an in-the-wild, offline-to-online RL stage for training the agent through interacting with an environment. Consequently, this work also introduces a scalable and parallelizable Android learning environment with a reward model (evaluator) based on a robust VLM-based model. To show the effectiveness of the proposed method, an evaluation of different tasks given diverse instructions is carried out from the Android in the Wild dataset on real Android device emulators. The results show a significant improvement of DigiRL compared to the existing state-of-the-art agents, including wrapped proprietary VLMs such as GPT-4V and Gemini 1.5 pro. The paper claims to be the first to succeed in developing an autonomous offline-to-online RL approach to enable state-of-the-art performance on device control problems.", "questions": "- How are the policy and the value network conditioned given the context $c$?\n- Is the RL agent trained in a multi-task fashion (which I believe is true)? I mean the agent is trained with more than one task concurrently accessing fully or partially the same models.\n- If yes, do you think classical Multi-task learning or Multi-task reinforcement learning approaches would enhance the performance even more?", "limitations": "I believe the authors discussed the method's limitations in the final section. I agree with the authors regarding the impact of such application on the economy, society, and privacy, and that needs careful review in the future to limit any harm.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The usage of RL in designing a successful digital agent for a device control task is fascinating. \n- I appreciate implementing such a scalable Android learning environment, and I hope the authors will open-source everything so that other researchers can reuse it.\n- The clarity of the paper is worth mentioning. \n- The experimental results section is rich, especially the ablation studies.", "weaknesses": "- Although the POMDP definition sounds correct when defining the problem, a contextual POMDP is even more appropriate for such a problem [1].\n- A pseudo-code or an illustrative diagram should be added to facilitate understanding the method.\n- It is not clear how the policy and the value network are conditioned given the context $c$. (implementation-wise)\n\n[1] Hallak, Assaf, Dotan Di Castro, and Shie Mannor. \"Contextual markov decision processes.\" arXiv preprint arXiv:1502.02259 (2015).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback on the paper. We provide responses to the questions raised below that we will also include in the updated version of the paper. We commit to open sourcing open-source the code, environment, checkpoint, and the data. In this rebuttal period, we provide an anonymous link to our code: (link sent via a private message to AC per rebuttal instruction), and will make this public  with the final version. \n\nThanks so much for your appreciation of our work that “the usage of RL in designing a successful digital agent for device control tasks is fascinating”! **Please let us know if your concerns are addressed and if so, we would appreciate it if you might be willing to upgrade your score.**  We answer your questions below:\n\n___\n\n### “It is not clear how the policy and the value network are conditioned given the context c (implementation-wise)”\n\nImplementation-wise, we use a 1B VLM AutoUI for the policy network so the context (in our case the task to complete) is directly included in the language input / encoder module of the VLM. For the value network, we encode the image with a CLIP encoder and the context with 110M BERT-base encoder for computational efficiency reasons. After this, an additional MLP layer is added on top of CLIP and BERT encodings concatenated with each other. We will include this clarification in the implementation detail section of a revised version of the paper.\n\n___\n\n### “A pseudo-code or an illustrative diagram should be added to facilitate understanding the method.”\n\nThanks for the suggestion. We have made an illustrative diagram to facilitate readers to understand the method in Figure 2 of Rebuttal PDF. In this diagram, we have an instruction-level value function and a step-level value function, both of which serve as “filters” to identify the “advantageous” data that the agent should try to train on with AWR. We will include this diagram in an updated version of the paper!\n\n___\n\n### “Is the RL agent trained in a multi-task fashion (which I believe is true)? I mean the agent is trained with more than one task concurrently accessing fully or partially the same models.”\n\n\nYes, the RL agent is trained in a multi-task fashion where at the start of each episode a random task (e.g. “Find the nearest place to buy a beach umbrella” and “Go to ebay.com, search for logitech 950” ) is drawn out of  a task pool containing 200 tasks.\n\n___\n\n### “If yes, do you think classical Multi-task learning or Multi-task reinforcement learning approaches would enhance the performance even more?”\nThis is a good idea! In fact, we believe that DigiRL is an effective starting point for building effective device control agents and researching RL algorithms for training agents. We are now ourselves building on DigiRL, to devise better RL approaches for training agents in this environment. Your suggestion is valuable for our exploration and we will study multi-task RL approaches such as PC-Grad [1], Task Grouping [2], etc to further enhance performance of DigiRL.  \n\n___\n\n### “Although the POMDP definition sounds correct when defining the problem, a contextual POMDP is even more appropriate for such a problem”\n\nThanks for pointing out this definition! We will post the device control problem in Section 3 as a contextual POMDP as it is indeed more appropriate. Note that this does not change any of the training objectives as it is largely a notational change. We will include an additional distribution over contexts $c$ when we define the POMDP. \n\n[1] Yu, Tianhe, et al. ‘Gradient Surgery for Multi-Task Learning’. arXiv [Cs.LG], 2020, http://arxiv.org/abs/2001.06782. arXiv.\n\n[2] Fifty, Christopher, et al. ‘Efficiently Identifying Task Groupings for Multi-Task Learning’. arXiv [Cs.LG], 2021, http://arxiv.org/abs/2109.04617. arXiv.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all the reviewers for their feedback. We are glad that the Reviewer Xhdc thinks that there is  “no major weakness” in the paper and that Reviewer zDzF thinks that “the usage of RL for designing a successful agent for device control tasks is fascinating”. 

At the outset, we would like to clarify that our goal is to show that autonomous RL can be used to build SOTA device control agents that outperform the dominant approaches in the device-control community including prompting proprietary models (Gemini/GPT-4) and fine-tuning with human demonstrations. Rather than claiming novelty of the algorithm, the main contribution of this work is the efficacy of our effective RL system / approach including the environment setup, usage of autonomous evaluator, and particular design choices specific to the real-world challenges of this environment. On the RL side, our methodological contribution involves identifying and designing the right RL objective to be able to do so (robust advantages + curriculum + cross-entropy loss + AWR). (see Fig.7 in paper and one-page PDF).

In the rebuttal period, we have run additional experiments and plot the results in the one-page pdf, including:
- Table 1: Ablation on Q-function based advantage estimations (Reviewer JuUv, Xhdc, and 251x)
- Table 2: Hyperparameter profiling for step-level advantage estimation (Reviewer 251x)
- Table 3: Comparison with search-based method with autonomous evaluator (Reviewer Xhdc)
- Figure 1: Study on the effect of stochasticity (Reviewer JuUv and 251x)
- Figure 2: Algorithm diagram (Reviewer zDzF)
- Figure 3: Comparison with pure-online setting (Reviewer 251x)
- Figure 4: Comparison with pure-online PPO (Reviewer JuUv)
- Figure 5: Studies on auto-curriculum (Reviewer Xhdc)

We thank the reviewers in advance for paying attention to our new results and clarifications. We look forward to discussions and hope that our responses and the discussion will convince the reviewers that our work is valuable.

Meta-review: Thank you for the rebuttal. After reviewing the reviewers' comments and the authors' responses, I acknowledge the contributions of this paper in advancing the use of Reinforcement Learning (RL) for digital agent control through Graphical User Interfaces (GUIs). The introduction of DigiRL, a novel autonomous RL approach combining offline training on static demonstrations with an in-the-wild, offline-to-online RL stage, offers a compelling solution to real-world stochasticity in device control tasks. I appreciate the authors addressing the major concerns during the rebuttal and recommend accepting this paper.
Decision: Accept (poster)