DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

Authors: hao  bai
yifei  zhou
mert  cemri
jiayi  pan
alane  suhr
sergey  levine
aviral  kumar
u c berkeley
google  deepmind
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 00:02:22

Abstract: Pre-trained vision language models (VLMs), though powerful, typically lack training on decision-centric data, rendering them sub-optimal for decision-making tasks such as in-the-wild device control through Graphical User Interfaces (GUIs) when used off-the-shelf. While training with static demonstrations has shown some promise, we show that such methods fall short when controlling real GUIs due to their failure to deal with real world stochasticity and dynamism not captured in static observational data. This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents through fine-tuning a pre-trained VLM in two stages: offline and offline-to-online RL. We first build a scalable and parallelizable Android learning environment equipped with a VLM-based general-purpose evaluator and then identify the key design choices for simple and effective RL in this domain. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our 1.5B VLM trained with RL achieves a 49.5\% absolute improvement -- from 17.7 to 67.2\% success rate -- over supervised fine-tuning with static human demonstration data. It is worth noting that such improvement is achieved without any additional supervision or demonstration data. These results significantly surpass not only the prior best agents, including AppAgent with GPT-4V (8.3\% success rate) and the 17B CogAgent trained with AitW data (14.4\%), but also our implementation of prior best autonomous RL approach based on filtered behavior cloning (57.8\%), thereby establishing a new state-of-the-art for digital agents for in-the-wild device control.

Main paper: 
 introduction: 
 Advances in vision-language models (VLMs), especially in regards to their remarkable commonsense, reasoning, and generalization abilities imply that realizing a fully autonomous digital AI assistant, that can simplify human life by automating day-to-day activities on computer devices via natural language interfaces, is no longer a distant aspiration [16, 45, 56 ]. An effective device-control AI assistant should be able to complete tasks in-the-wild through Graphical User Interfaces (GUIs) on digital devices: make travel plans; experiment with presentation designs; and operate a mobile device autonomously, all while running amidst stochasticity and distractors on the device, the Internet, and the tools it interacts with. However, enhanced reasoning or common-sense abilities do not directly transfer to intelligent assistant behavior: ultimately we want AI assistants to accomplish human demonstrations can easily get stuck in out-of-distribution states while GPT-4V often get on a wrong goal (searched "logitech g933bestbuy.com logitech g933" in Google instead of bestbuy.com). In contrast, DigiRL can recover from such states and complete complex instruction as requested. maximal learning signal, which is inspired by prioritized replay methods [11, 32, 23] , and (ii) another step-level value function trained via effective cross-entropy loss [17, 5] to extract low-variance and less-biased learning signal amidst stochasticity and diverse tasks. This RL approach allows us to fine-tune VLMs on their own experience.We evaluate our agent trained with DigiRL in carrying out diverse instructions from Android in the Wild dataset [31] on real Android device emulators and find that our agent can achieve a 28.7% improvement over the existing state-of-the-art agents (from 38.5% to 67.2% success rate) 18B CogAgent [9] , and over 9% improvement over the prior best autonomous learning approach based on Filtered Behavior Cloning [18, 26] . The performance of our agent also significantly surpasses wrappers on top of state-of-the-art proprietary VLMs such as GPT-4V [24] and Gemini 1.5 Pro [7] (17.7% success rate), despite using a significantly smaller model (with 1.3B parameters). To our knowledge, this is the first work to successfully build an autonomous offline-to-online RL approach to enable state-of-the-art performance on device-control problems. 
related work: 
 Multi-modal digital agents. In contrast to language-only agents that largely interact with both text or code inputs and outputs [33, 49, 3, 30, 46, 20, 13] , training multi-modal agents capable of controlling devices presents different challenges: first, device control is done directly at the pixellevel and in a coordinate-based action space, instead of natural language [31, 44] that LLM is most familiar with, and second, the ecosystem of a device and the Internet tends to be quite stochastic and unpredictable, which is absent with high-level planning in language only. To handle these challenges, prior work largely builds on strong proprietary VLMs [24, 7], and designs complex rule-based wrappers [47, 51, 45, 52] to enhance the visual grounding capabilities of VLMs in GUI interfaces and convert text output into pixel interactions. However, without any form of fine-tuning, this limits the room for possible performance improvement [44, 47, 49, 3, 50] , especially when pre-training corpora only present limited action-labeled data. A separate line of work fine-tunes VLMs with demonstration data [19, 15, 9, 53] via imitation learning, but maximizing single-step accuracy from stale demonstrations without accounting for consequences of these actions in subsequent steps may lead to poor solutions amidst stochasticity [26] , as agents trained in such ways will struggle to recover from out-of-distribution states not included in the demonstration data [8, 12] . The third category, and perhaps the closest to us, are works that run filtered imitation learning on autonomously-collected data to directly maximize the episode success rate [26, 18] . In contrast, ours is the first work to scale autonomous, offline-to-online RL for device control, producing an agent that outperforms prior agents built via imitation. Even when compared to prior work running on-policy RL in simplified web navigation settings (MiniWob++ [37, 10] ), our approach is 1000x more sample efficient (around 1e3 trajectories compared to around 1e6 trajectories), and operates in real-world GUI navigation tasks.Environments for device control agents. Recent works have introduced simulated environments for building device control agents [48, 56, 16, 54, 4, 44] . However, these environments are primarily designed for evaluation, and present only a limited range of tasks within fully deterministic and stationary settings, infeasible for acquiring a diverse repertoire of skills needed for device control.Alternatively, other works use environments with a greater diversity of tasks [48, 37] , but these environments often oversimplify the task complexity, thus failing to transfer to in-the-wild settings. Coversely, our training environment utilizes autonomous evaluation [26] with Gemini 1.5 Pro [7] to support diverse, open-ended tasks on parallel actual Android devices, at full scale unlike prior environments. This also contrasts other prior works that use single-threaded Android emulators [26, 39, 19] and thus inefficient for support online RL at scale.Reinforcement learning for LLM/VLMs. The majority of prior research employing RL for foundation models concentrates on tasks that must be solved in a single turn, such as preference optimization [25, 58, 2] or reasoning [27] . However, optimizing for single-turn interaction from expert demonstrations may result in sub-optimal strategies for multi-step problems [57, 38, 42] , especially amidst a high degree of stochasticity or non-stationarity. Therefore, we focus on building multi-turn RL algorithms that can learn from sub-optimal, online interaction data in this work. While prior works have developed value-based RL algorithms for LLMs [42, 38, 1, 57, 50] , they typically require maintaining multiple models such as Q-networks, value-networks, and policy networks, along with their delayed target counterparts, and can be subjective to slow convergence and sensitivity to choices of hyper-parameters. In contrast, we focus on identifying the key design choices for instantiating a simple yet effective RL algorithm for practitioners to incorporate to substantially improve full-scale Android device control. Our approach can serve as a base model for future research.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Limitations are discussed in the last section of the paper. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper point out any strong assumptions and how robust the results violations of these (e.g., independence assumptions, noiseless settings, model asymptotic approximations holding locally). The should reflect on these assumptions might be violated in practice and the implications be. • The authors should reflect on the scope the made, e.g., if the only a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be Justification: The research conducted in the paper conform, in every respect, with the NeuIPS code of Etics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
problem setup and preliminaries: 
 Problem formulation. We are interested in pixel-based interaction with virtual devices. We scope our study in the control of Android devices: this is already significantly more challenging and more general than previous learning-based environments that focus solely on web navigation [16, 56, 4] , where the web browser itself is merely one application within our broader environment, and link-based device controls [47, 51] are inadequate for tasks like games that do not support link inputs.Each episode begins with the emulator initialized to the home screen. Subsequently, a task is selected from a predefined set of language instructions, some examples of which are shown in Appendix A.1.An agent is then tasked with manipulating the emulator to fulfill this instruction. At each time step, the agent receives a screenshot of the current screen as the observation. Following the action space in prior literature [31] , the available actions include tapping and sliding based on normalized (x, y) coordinates (ranging from 0 to 1 relative to the screen dimensions), typing text strings of variable length, and pressing special buttons such as HOME, BACK, and ENTER, as illustrated in Figure 3 .Our train and test instructions comes from General and Web Shopping subsets in AitW [31] . These tasks consist of information-gathering tasks like "What's on the menu of In-n-Out?", and shopping tasks on the web like "Go to newegg.com, search for razer kraken, and select the first entry".Challenges of stochasticity. Real-world device contrl presents unique challenges of stochasticity absent in simulated environments [56, 37] such as: (1) the non-stationarity of websites and applications, which undergo frequent updates, causing the online observations to be different from stale offline data, (2) various unpredictable distractors such as pop-up advertisements, login requests, and the stochastic order of search results. (3) technical challenges and glitches such as incomplete webpage loading or temporary access restrictions to certain sites. Examples of scenarios with such stochasticity from our experiments are shown in Figure 3 . We observe that these stochastic elements pose significant challenges for pre-trained VLMs, including even those fine-tuned on device control data. As a concrete example, Figure 4 shows an experiment result that illustrates the necessity of continuously adapting the models to the non-stationarity of websites and applications. After obtaining a good checkpoint using our approach (DigiRL), that we will introduce in the next section, with autonomous data from June.1 to June.3, we compare the performance of a frozen policy and a continuously updating policy using fresh autonomous data from June.7 to June.11. We find that indeed the the performance of the frozen policy gradually degrades over time due to the changes on websites and applications, while continuous online updates plays a key role in preventing this degradation. Setup for reliable and scalable online RL. As autonomous RL interleaves data collection and training, to maximize learning amidst stochasticity, it is crucial to have a real-time data collection pipeline to collect enough experience for gradient updates. While this is not possible in single-thread Android emulator environments [26, 39] due to latency, we parallelize our Android emulator using appropriate error handling as discussed in Appendix A.1. In addition, the environment must provide a reward signal by judging whether the current observation indicates the agent has successfully completed the task. To generalize our evaluator to support a wide range of tasks, we extend Pan et al. [26] 's end-to-end autonomous evaluator that does not require accessing the internal states of the emulator or human-written rules for each task. This contrasts previous works that manually write execution functions to verify the functional completeness of each task [16, 48, 37, 44] . We adopt Gemini 1.5 Pro [6, 7] as the backbone of the autonomous evaluator. We seed this model with few-shot rollouts and the associated human-labeled success indicators to guide evaluation of novel queries. This pipeline enables a single evaluator that can evaluate all AiTW tasks. The evaluator is highly aligned with human annotations (average error rate 2.8%), validated in Figure 8 . 
digirl: autonomous rl for building a strong device-control agent: 
 We now present our autonomous RL framework for training device agents. We pose the device control problem as a Markov decision process (MDP) and develop RL methods for this MDP. The core of our approach is based on a simple and scalable off-policy RL method, advantage-weighted regression (AWR) [29] , but we make crucial modifications to handle stochasticity and highly-variable task difficulty, through the use of value functions trained with appropriate losses, and an automatic curriculum, induced by an instruction-level value function to maximize learning.Device control and GUI navigation as a MDP. We conceptualize device control guided by natural language instructions as a finite horizon Markov Decision Process (MDP) represented by M = {S, A, T , µ 0 , R, H} and run policy gradient to solve this MDP. At the beginning, an initial state s 0 and a natural language instruction c are sampled from the initial state distribution µ 0 . A reward of 1 is given at the end if the agent successfully fulfills the task per the evaluator, otherwise a reward of 0 is given. The trajectory terminates either when the agent accomplishes the task or when the maximum allowed number of interactions H is exceeded. States are represented using the last two screenshots. To explain our approach in detail, we also include several standard definitions used in reinforcement learning (RL). The Q function for a policy π represents the expected longterm return from taking a specific action at the current step and then following policy π thereafter:Q π (s h , a h , c) = E π H t=h r(s t , a t , c) . The value function V π (s h , c) is calculated by averaging the Q-value, Q π (s h , a h , c), over actions a h drawn from the policy π. The advantage A π (s h , a h , c) for a state-action pair is computed by subtracting the state's value under the policy from its Q-value:A π (s h , a h , c) = Q π (s h , a h , c) -V π (s h , c).The starting point we choose to build our approach on is the advantage-weighted regression (AWR) algorithm [29] , which says that we can improve the policy reliably by regressing the policy towards exponentiated advantages induced by the reward function, as a proxy for optimizing the policy gradient while staying close to the previous policy [14, 35, 34] :EQUATION) for some positive parameter β and the distribution of past experience ν, and A(s, a, c) denotes the advantage of a state-action pair (s, a) given a context c. To avoid tuning the hyperparameter β, we consider an alternative that does "hard filtering" on the advantages instead of computing exp(A), similar to prior works [22, 43] . This leads to the following loss function for fine-tuning the model:L(π) = -E filter(ν) [log π(a|s, c)]. (4.2) Typically, these advantages are computed by running Monte-Carlo (MC) rollouts in the environment to estimate the value of a given state-action pair, and subtracting from it an estimate of the value of the state given by a learned value estimator alone. However, this approach is likely to produce high-variance advantages given the stochasticity of the device eco-system that affects MC rollouts.To reliably identify advantageous actions given significant environment stochasticity, we construct a per-step advantage estimator, inspired by doubly-robust estimators [40, 36] :A step (s h , a h , c) := λ H-h r(s H , a H , c) + (1 -λ H-h r(s H , a H , c))(V step (s h+1 , c) + r(s h , a h , c) -V step (s h , c)), (4.3)where λ is a weighting hyper-parameter. This construction of the advantage estimator is a simplified version of Generalized Advantage Estimation (GAE) [36] using only the next-step advantage estimator and final-step advantage estimator as there are no intermediate rewards in our problem. This construction balances an advantage estimator with higher variance Monte-Carlo estimates λ H-h r(s H , a H , c) (due to stochasticity) and an estimator with higher bias V step (s h+1 , c) + r(s h , a h , c) -V step (s h , c) (due to imperfect fitting of the value function). We observed that combining both high-variance and high-bias estimators gave us a sweet-spot in terms of performance. To implement the step-level hard filtering, we simply threshold this doubly robust estimator as A step (s h , a h , c) > 1/H to decide which actions progress towards the goal.While the AWR update (Equation 4.1) coupled with a robust advantage estimator (Equation 4.3) is likely sufficient on standard RL tasks, we did not find it to be effective enough for device control in preliminary experiments. Often this was the case because the task set presents tasks with highlyvariable difficulties that collecting more data on tasks that the agent was already proficient at affected sample efficieny negatively. In contrast, maximal learning signal can be derived by experiencing the most informative tasks for the agent during training. To this end, we design an instruction-level value function V instruct (c) to evaluate if a given rollout can provide an effective learning signal:EQUATIONwhereH t=h r(s t , a t , c) is a Monte-Carlo estimator of Q(s h , a h , c). The equality holds because the MDP formulation only provides rewards at the end of a rollout. Intuitively, if a rollout attains a high value of A instruct (s h , a h , c), it means the value function V instruct is small. Therefore, this rollout represents a valuable experience of the agent accomplishing a difficult task, and thus should be prioritized, akin to ideas pertaining to prioritized experience [32] or level replay [11] . When training the actor with a buffer of historical off-policy data, we first perform a filtering step to identify the top-p datapoints with highest A instruct (s h , a h , c). Then, we use it for AWR (Equation 4.1) with the doubly-robust advantage estimator (Equation 4.3).Implementation details. Inspired by the findings in some recent works [5, 17] that modern deep learning architectures like transformers [41] are better trained with cross-entropy losses instead of mean-squared losses, we utilize a cross-entropy objective based on the Monte-Carlo estimate of the trajectory reward for training both of our value functions: EQUATIONL(V step ) = -Eν [r(sH , aH , c) log V step (s h , a h , c) + (1 -r(sH , aH , c)) log(1 -V step (s h , a h , c))]. 
experimental evaluation: 
 The goal of our experiments is to evaluate the performance of DigiRL on challenging Android device control problems. Specifically, we are interested in understanding if DigiRL can produce agents that can effectively learn from autonomous interaction, while still being able to utilize offline data for learning. To this end, we perform a comparative analysis of DigiRL against several prior approaches, including state-of-the-art agents in Section 5.1. We also perform several ablation experiments to understand the necessity and sufficiency of various components of our approach in Section 5.2.Baselines and comparisons. We compare DigiRL with: (a) state-of-the-art agents built around proprietary VLMs, with the use of several prompting and retrieval-style techniques; (b) running imitation learning on static human demonstrations with the same instruction distribution, and (c)a filtered BC approach [26] . For proprietary VLMs, we evaluate GPT-4V [24] and Gemini 1.5 Pro [7] both zero-shot and when augmented with carefully-designed prompts. For the zero-shot setting, we use the prompt from Yang et al. [47] and augment the observation with Set-of-Marks [55] . Set-of-Marks overlays a number for each interactable element over the screenshot, so that a VLM can directly output the number of the element to interact with in plain text instead of attempting to calculate pixel coordinates, which is typically significantly harder. We also compare with AppAgent [47] , which first prompts the VLM to explore the environment, and appends the experience collected to the test-time prompt. We also compare with two state-of-the-art fine-tuning methods for Android device control: AutoUI (specifically AutoUI-Base [53] ) and CogAgent [9] . AutoUI-Base uses an LM with 200M parameters, and a a vision encoder with 1.1B parameters. CogAgent has 11B parameters for its vision encoder and 7B for its LM. The supervised training corpus for both AutoUI-Base and CogAgent contains AitW, including the instruction set and the emulator configuration we use.Base VLM and offline dataset. Both Filtered BC and DigiRL use trained AutoUI-Base checkpoints with the image encoder frozen. The instruction and step-level value functions for DigiRL employ this same frozen image encoder. The visual features output from the encoder are concatenated with instruction features derived from RoBERTa [21] . A two-layer MLP is then used to predict the value function. In the offline phase, the offline dataset is collected by rolling out the initial AutoUI-Base supervised trained checkpoint as policy. For fair comparisons, we keep the number of offline data collected in the pure offline training roughly the same as the total number of data collected in the offline-to-online training. Due to the dynamic nature of the Internet-device eco-system, our offline data was stale by the time we were able to run our offline-to-online experiments, and this presented additional challenge in offline-to-online learning. In both General and Web Shopping subsets, offline experiments make use of around 1500 trajectories while offline-to-online experiments start with around 500 offline trajectories and update with another 1000 online trajectories. In the offline phase, DigiRL skips instruction-level filtering and instead trains the actor with all successful trajectories to make full use of the offline data. See a detailed breakdown of our dataset in Appendix A.1.Our main results are summarized in Table 1 and Figure 6 . We find that on both AitW General and AitW Web Shopping subsets, the agent trained via DigiRL significantly outperforms prior state-of-the-art methods based on prompting and retrieval (AppAgent + GPT-4V/Gemini 1.5 Pro) or training on static demonstrations (CogAgent and AutoUI), by a large margin with more than 49.5% absolute improvement (from 17.7% to 71.9% on the General subset and from 17.7% to 67.2% on the Web Shopping subset). Notably, this improvement from DigiRL is realized fully autonomously without making use of human supervision (e.g. manually labeled rollouts or hand-written verifiers).Are inference-time prompting and retrieval techniques or supervised training enough for device control? Delving into Table 1 , we observe that off-the-shelf proprietary VLMs, even when supplemented with the set-of-marks mechanism, do not attain satisfactory performance: both GPT-4V and Gemini 1.5 Pro achieve success rates under 20%. One possible cause could be the underrepresentation of Android device data in the pre-training data. Moreover, inference-time adaptation strategies such as AppAgent [47] show minimal improvement, with gains not exceeding 5% for either model. All this evidence suggests a limited scope for improvement without fine-tuning of some sort. As illustrated in Figure 7 , the primary failures of these VLMs stem from hallucinatory reasoning that lead the VLMs to land on a relevant but wrong page. This suggests that while state-of-the-art VLMs excel at reasoning problems in code and math, their reliability in less-familiar domains, such as device control, remains inadequate. For example, for the instruction "Go to newegg.com, search for alienware area 51, and select the first entry", a GPT-4V based agent erroneously searched "alien area 51 ebay" in Google.com and decided that it had made progress towards the task (Figure 17 ).Training on domain-specific human demonstrations, however, does boost performance, allowing the smaller, specialized VLM, AutoUI with 1.5 billion parameters, to match or surpass the larger, generalist VLMs like GPT-4V and Gemini 1.5 Pro. Nonetheless, this supervised imitation learning approach still fall short, with success rates on both subsets remaining below 20%. This shortcoming is not fundamentally addressed via enhancements in model scale or architecture, as evidenced by CogAgent [9] , with 18 billion parameters still achieving performances below 40% success rate. As depicted in Figure 7 , a predominant failure mode for these agents is an inability to rectify their own errors. An example trajectory that we observed is that for the instruction "what's on the menu of In-n-Out", the agent accidentally activated the voice input button, and failed to quit that page until the step limit. In contrast, DigiRL is able to recover from the errors more efficiently( Appendix C.2).Comparison of different RL approaches. In Table 1 and Figure 6 , we present a comparative analysis of various autonomous approaches. Notably, both offline and offline-to-online configurations demonstrate that our RL approach, when augmented with a continuous stream of autonomous interaction data and reward feedback, substantially improves performance. This improvement is evident from an increase in the success rate from under 20% to over 40%, as the agent learns to adapt to stochastic and non-stationary device interfaces. Moreover, although the total sample sizes for offline and offline-to-online settings are equivalent, the top-performing offline-to-online algorithm markedly surpasses its offline counterpart (75% versus 62.8% on the General subset). This highlights the efficacy of autonomous environment interaction, and establishes the efficacy of DigiRL in learning from such uncurated, sub-optimal data. Lastly, DigiRL consistently outperforms the state-of-the-art alternative, Filtered BC, across both the General and Web Shopping subsets, improving from 61.5% to 71.9% and 57.8% to 61.4%, respectively, highlighting DigiRL's performance and efficiency.Failure modes analysis. We conduct an additional user study to annotate the failure modes for each agent as shown in Figure 7 , and a more fine-grained breakdown can be found in Appendix D. At a high level, we classify the major failure modes of all agents into the following three categories: (1) Failure to recover from mistakes refers to the scenario where the agent made a mistake that led it to states from which it failed to quickly recover and resume the task, such as a wrong search page. (2) Getting stuck midway refers to the failure mode where the agent gets distracted on the right track to completing the instruction and as a result fails to accomplish the task. For example, failing to click on the right link or failing to search after typing the key words.(3) Arriving at wrong goal refers to the failure mode where the agent arrives at a wrong page and mistakenly thinks that it had completed the task. For e.g, the agent finds a macbook on costco.com instead of finding a macbook on ebay.com.While all the types of failure modes benefit from offline and offline-to-online RL training as shown in Figure 7 , the most consistent and significant reduction is probably for the failure mode of failing to recover from mistakes. This is because while pre-trained models, generating plausible future tokens, can get distracted by the dynamic nature of the environment and, as a result, encounter at never-before-seen states. With no clue of how to escape such states, these methods are unable to recover and fail to solve the task. In contrast, by training on autonomously-collected rollouts, our agent DigiRL is able to learn from its own mistakes and reduces failures to recover over training. Ablation study of each component in DigiRL.We conduct an ablation study on different components of DigiRL in Figure 9 . We find that all the components used by our approach are necessary: ( Additionally, we also observe no degradation in performance as a result of "hard-filtering", as show by nearly comparable performance of our approach and the best run of exponential filtering obtained via an extensive tuning of the temperature hyperparameter τ in naïve AWR (comparing Ours and Ours w/ vanilla AWR reweighting), despite simplicity of implementation in the hard filtering approach. Putting together, these choices result in a new state-of-the-art RL approach for device control.Evaluation of our autonomous evaluator. In Figure 8 , we present the findings from a user study aimed at assessing the accuracy of our autonomous evaluator. Our results indicate that the success rates reported by our automatic evaluator are remarkably consistent with those assessed by human evaluators across almost all models, with differences less than 3%. Furthermore, we observed that evaluations on the Web Shopping subset are more precise compared to those on the General subset. This increased accuracy likely stems from the fact that tasks in the General subset are formulated in free-form language, which can introduce ambiguity, whereas the Web Shopping subset features a narrower range of language expressions, reducing potential variability. 
discussion and limitations: 
 In this paper, we propose a novel autonomous RL approach, DigiRL, for training in-the-wild, multimodal, device-control agents that establish a new state-of-the-art performance on a number of Android control tasks from Android-in-the-Wild dataset [31] . To achieve this, we first build a scalable and parallelizable Android environment with a robust VLM-based general-purpose evaluator that supports fast online data collection. We then develop a system for offline RL pre-training, followed by autonomous RL fine-tuning to learn via interaction, admist the stochasticity of the real-world Internet and device eco-system. Our agent achieves a 280% improvement over the previous state-of-the-art agents (from 17.7% to 68.2% in terms of task success rate), including AppAgent based on GPT-4V and Gemini 1.5 Pro, and supervised trained models such as AutoUI and CogAgent.Due to computational limitations, and despite the fact that the parallel emulator and autonomous evaluator can be easily extended to complicated tasks, our agent is trained only with tasks from AitW instead of a all possible tasks on the device. Our design of the DigiRL algorithm aims for maximal implementation simplicity, so we hope that our approach to serve as a base algorithm for future research to build on, including algorithmic research as well as expanding the space of tasks.A.1 Post-processing of AitWThe Android in the Wild (AiTW) task set is a large-scale dataset for android device control, containing five subsets: GoogleApps, Install, Web Shopping, General, and Single, where we select the General and Web Shopping subsets. Single subset is not considered here because all tasks in Single can be completed within one step and thus this subset fails to examine the multi-step challenges that we are interested in this paper. Install and GoogleApps are not considered due to security reasons as those tasks require an active Google account and parallel emulations can flag security concerns.General. The General set focuses on searching for information and basic application usage. For example, it contains searching for latest news in Chile, search for flights from NYC to Sydney, opening Gmail, etc. We use all 545 tasks in the training set for training and the first 96 tasks in the test set for testing due to computational and budget constraints. The maximum allowed number of steps for this subset is 10. Offline data is collected by rolling our the initial AutoUI policy with tasks from the training set. The offline data used for the offline-to-online setting contains 608 trajectories while the offline data used for the offline setting contains 1552 trajectories. Some task examples are shown in Table 3 .How do I get to the nearest Verizon Store? How much does a 2 bedroom apartment rent for in Denver? Search for flights from Barcelona to Boston What's a good restaurant in New York? What's on the menu at Burger King? Table 2 : Examples of task descriptions in the AiTW General task set.Web Shopping. The Web Shopping subset comprises search instructions on various shopping websites, like searching for razer blader on ebay. As some websites (e.g. Amazon) and operations (e.g. adding items to cart) frequently require captcha verifications, we post-process the Web Shopping subset to exclude such operations and websites and also make the task easy to evaluate for our autonomous evaluator. The resulting task set involves navigating through five websites (costco.com, bestbuy.com, target.com, walmart.com, newegg.com) and three basic operations (go to website, search in the website, and select items from the searched results). Our post-processed training set contains 438 tasks and our testing set contains 96 tasks. Example tasks after post-processing can be found in Table 3 . The maximum allowed number of steps for this subset is 20. Offline data is collected by rolling our the initial AutoUI policy with tasks from the training set. The offline data used for the offline-to-online setting contains 528 trajectories while the offline data used for the offline setting contains 1296 trajectories.Go to costco.com Go to walmart.com 2 Go to costco.com, search for "bose soundsport free" Go to walmart.com, search for "logitech g910" 3 Go to costco.com, search for "bose soundsport free" and select the first entry Go to walmart.com, search for "logitech g910" and select the first entry When running experiments on the AitW Web Shopping subset, we find solving easier tasks helps improve solving harder tasks, where the difficulty is identified in Table 3 . By specifying the difficulty DigiRL-Run1 in Figure 6 , we empirically show the success rates of each difficulty across the online learning process in Figure 12 , we observe that a significant increase of success rate of tasks of difficulty 1 leads to increasing success rate of difficulty 2, and the same pattern for difficulty 2 and 3, demonstrating effective curriculum learning. We ablate on the learning method, i.e. online learning or offline-to-online learning. We find that offline-to-online learning converges faster than online learning, and is not necessarily worse than online learning in terms of final performance, as shown in Figure 11 .We investigate the horizon limit of filtered BC and DigiRL on the AitW General subset. As most tasks can be effectively solved within 10 steps, we specify two horizon limits: a sufficient horizont H = 10, and a redundant horizon H = 20. Results in Figure 12 show that a redundant horizon introduces significantly faster learning speed for both filtered BC and DigiRL, presumbaly because longer horizon means more opportunity to try in a single trajectory. In both horizon settings, we observe the DigiRL offers a significant speedup of around 100 trajectories over Filtered BC.We investigate the rollout length of DigiRL compared to filtered BC. Results in subsets. This observation holds true whether considering all rollouts for computing this correlation or only investigating this correlation on rollouts that eventually succeed. This indicates the capability of DigiRL to solve tasks in a more efficient and directed manner. Qualitative examples can be found in Figure 16 .In Figures 13 and 14 , we provide trajectories of DigiRL, AutoUI, and GPT-4V randomly sampled from our test set to offer a qualitative understanding of the agents' performance. As shown in these examples, DigiRLcan efficiently carry out in-the-wild device control tasks and less likely to get stuck or get to a wrong page compared to AutoUI and GPT-4V.We observe that DigiRL is able to recover from its own mistakes. As shown in Figure 15 , we find that DigiRL explores ways to get back to the original screen in order to perform a search. As a comparison, AutoUI fails to reset to the original screen and gets stuck at the diverged screen. Under the hood, we find DigiRL trying to maximize the state value, which usually induces it to reset to the original screen (that has a large value to success). Qualitative example on the number of steps in trajectories of DigiRL and filtered BC are shown in Figure 16 . We find consistent cases where DigiRL has shorter trajectory length than filtere BC.The performance of GPT-4V failed on AiTW tasks predominantly due to not being able to carry out control actions as it plans on a high level, and then not being able to recover from these mistakes. Moreover, one of the main reasons why it is not able to recover from a mistake is that it might hallucinate and make itself believe that it is a wrong app or website. Indeed, GPT-4V constructs a plan of further actions when provided a task from either Web Shopping or General dataset of AiTW. Then, when it makes a misclick and fails to successfully proceed in an intermediate step, it might think that it actually solved that intermediate step and is in the correct app or website to execute further actions, the overall trajectory to fail. An example of this is provided in Figure 17 . Here, we ask the model to search for an item in a webshopping website, in particular in "newegg.com". However, the model fails to proceed to that website due to not being able to precisely locating the search button. Then, instead of trying to go to that website again, the model thinks it is already in that webshopping website, and mistakes the search bar of Google with the search bar of "newegg.com". Hence, the rest of the trajectory also fails. Another slightly different phenomenon is illustrated in Figure 18 . Here, the model is able to proceed to the correct website and search for an item, but this time it fails to tap on the search button on the website and clicks to an advertisement instead. Consequently, the model fools itself to think it successfully searched the item, and scrolls the page hoping to find that item, but it cannot do so because in reality it views the results of the advertisement. The primary reason of these failures is the challenge of grounding the control actions in GUI interfaces to realize the intermediary goals laid out by GPT-4V model's thoughts. As an example, we provide an illustration of trying to set up an alarm task in Figure 19 . Here, in the last frame, it fails to execute the precise movements in the necessary amount of rounds to correctly set up the alarm to the desired time, and in the last frame we see that the action taken does not align with the thought process of the model. In Figure 20 , we present a more fine-grained breakdown for all six failure modes provided in the user study. Those failure modes include:• Failure to recover from mistakes refers to the scenario where the agent made a mistake that led it to states from which it failed to quickly recover and resume the task, such as a wrong google search page.• Failure to click on the right link or failure to click refers to the failure mode where the agent either fails to locate the element that it tries to click on and keeps clicking on the nearby region, or fails to start typing in the string when it is supposed to do so.• Failure to take reasonable attempts at all refers to the failure mode where there is no clear reason that the agent fails to complete the task and does not seem to be on the right track throughout the trajectory.• Quit or press HOME early refers to the failure mode where the agent decided to finish the task or press HOME to start over before the task is actually finished.• Stops at wrong but relevant page refers to the failure mode where the agent arrives at a wrong page and mistakenly thinks that it had completed the task. For example, the agent finds a macbook on costco.com while the instruction asked it to find a macbook on ebay.com.• Technical issues refer to the failure mode that either the task is impossible (e.g. the tasks asks to open Amazon app but this app is not installed) or the agent is temporarily blocked from a certain website due to frequent visits.The translation between fine-grained failure modes and coarse-grained failure modes is presented in Table 5 .Go to newegg.com, search for "alienware area 51" and select the first entry. Action: tap (10) Figure 17 : Failure of GPT-4V, with its thoughts and link-based actions given. A typical cause of failure is that it cannot tap on the correct "search" button after entering a query and mistakenly tapped onto the "x" symbol in the search bar as the "search" button. Here the goal is: Go to newegg.com, search for "alienware area 51" and select the first entry. As seen in emboldened actions, it fails to press search button and deletes the query instead. Also, as seen in red highlighted parts in thoughts, it thinks it is in "newegg.com" website even though it is not. Figure 19 : Failure of GPT-4V, with an example task on the AiTW general test set. The task is "Set an alarm for 4pm". Here, GPT-4V able to successfully navigate to the clock app, and the alarm settings of that app. However, cannot take the correct precise actions to set the alarm quickly enough, and it fails due to maximum rounds reached. In the last round, notice that the action of tap(1) contradict with its own thought process of setting minutes to "00". Set-Of-Marks GPT4V Set-Of-Marks Gemini-1.5-Pro AppAgent GPT4V AppAgent Gemini-1.5-Pro AutoUI CogAgent Filtered BC Offline DigiRL Offline Filtered BC Online DigiRL Online Set-Of-Marks GPT4V Set-Of-Marks Gemini-1.5-Pro AppAgent GPT4V AppAgent Gemini-1.5-Our main experiments are conducted on VM instances from Google Cloud Platform. Each VM instance comes with 1x Tesla T4 GPU and 16x Intel(R) Xeon(R) CPU.Running multiple emulators in parallel can be challenging due to the inefficiency in thread synchronization and frequent fault propagation when one emulator runs into an unknown error. To address this challenge, we set up a server-client system where all emulator processes are running in independent server processes. Each emulator process communicates with the main training process through different UIAutomotor servers. The main training process sends high-level instructions to UIAutomotor servers (such as reset and step), while UIAutomotor servers parse high-level instructions into low-level UI commands (such as typing a character and tapping at a coordinate) and such UI commands are executed by the emulator processes. When an exception is thrown in the emulator, the UIAutomotor examines if it is recoverable (e.g. an UI command takes too long to execute in the emulator) and reset emulator process if it When an exception is thrown in the UIAutomotor server, the training process stops and the UIAutomotor server to ensure data correctness. This design can easily be scaled up to a multi-machine setting. As illustrated in Figure 21 , one host machine equipped with GPU accelerator has a local copy of the current policy π t , and distributes the policy to all worker machines equipped with only one GPU and multiple CPUs. Each worker machine will then collect trajectories of different tasks using π t . After all collection processes are synchronized, the host machine gathers all the trajectories together to update the policy to π t+1 . This process keeps iterating until the policy converges.Speedup emulation parallel. performance boost with respect to the number of worker machines is nearly linear, as demonstrated in Figure 22 (right), where we conduct experiments that examine the scaling performance of our parallel emulator. Our distributed emulator that emulations across multiple servers can reliably collect data with up to 64 parallel emulators on 128 CPUs with near-linear speedup. In contrast, a naive baseline that runs all parallel emulations on the same server achieves much inferior performance (0.74 compared to 1.74 trajs/min using 64 CPUs).Our autonomous evaluator gives a reward to each observation we get. The observation is composed of the current screenshot of device and the task. The evaluator gives a reward of 1 if the screenshot shows a completion of the task, and will terminate the POMDP as a result result.The optimized prompt is shown in Figure 23 and Figure 24 for General and Web Shopping subsets respectively. Figure 25 shows the prompt that we used for testing the Set-of-Marks performance for GPT-4V and Gemini 1.5 Pro. This prompt is directly taken from Yang et al. [47] .Hyperparameters for both Filtered BC and DigiRL are carefully tuned through binary search on the training set of General and Web Shopping subsets. The final choice of hyperparameters for both methods can be found in Table 6 • It is fine to include aspirational goals as motivation as as it is clear that these goals are not attained by the paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The positive societal impacts are discussed in the Introduction while the negative societal impacts are discussed in Section 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The capability of the model that we will be releasing is limited to simple tasks in Android in the Wild dataset, and therefore does not have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.We use external versions of these models as of June 11, 2024. Experiments with GPT and Gemini models were performed entirely by Hao Bai, Yifei Zhou, Mert Cemri, and Jiayi Pan. 
 
 Appendix:  


Decision: Thank you for the rebuttal. After reviewing the reviewers' comments and the authors' responses, I acknowledge the contributions of this paper in advancing the use of Reinforcement Learning (RL) for digital agent control through Graphical User Interfaces (GUIs). The introduction of DigiRL, a novel autonomous RL approach combining offline training on static demonstrations with an in-the-wild, offline-to-online RL stage, offers a compelling solution to real-world stochasticity in device control tasks. I appreciate the authors addressing the major concerns during the rebuttal and recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: This paper tackles AI agent training for controlling digital devices (e.g., web navigation). The proposed framework, named DigiRL, is a 3-stage training process consisting of model pre-training, offline fine-tuning (offline RL), and online fine-tuning (online RL). To achieve this goal, the authors first build a parallelizable Android learning environment that enables fast online interactions for policy learning; they then adopt a VLM-based evaluator to provide reward signals for the agents; finally, they perform ablation studies to examine several key design choices in typical policy-based RL methods for the third stage. Compared to larger models trained without this stage, the proposed approach enjoys significant performance enhancement due to the online fine-tuning stage.
strength_weakness: strength_weakness: The main issue is the limited comparison with online RL methods for fine-tuning LLM-based agents. The only RL method compared in the experiments is Filtered BC (besides vanilla AWR, which the proposed method is based on). Filtered BC is strictly speaking not an online RL method. Admittedly, AI agent training for device control is a relatively under-explored new area, and the authors claim that theirs is the first successful offline-to-online RL approach for device control AI agents, I believe more experiments with other online RL baselines not originally designed for device control is required to justify DigiRL's advantages. For example, the classic on-policy methods such as REINFORCE and PPO, or the more recent ones that are more sample-efficient, such as [1, 2] (which might be considered more-or-less recurrent work, though). Further comparisons also help to provide more insight into the unique challenges of the device control problem for digital agents.

[1] Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
[2] REBEL: Reinforcement Learning via Regressing Relative Rewards
questions: questions: - Will the proposed method further scale well with more online interactions?
- While multi-turn interactions are a challenge of the device control problem, is there any component in the proposed framework that specifically helps to tackle it?

I am willing to adjust my ratings after seeing the authors' responses.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes an autonomous RL approach, RL for digital agent (DigiRL), to finetune a pretrained VLM as an in-the-wild device control agent through GUI. The authors build a parallelizable Android learning environment with VLM-based evaluator to identify the key design choices for RL. The training include two stages an offline RL phase on existing data, then followed by an online RL phase by interacting with real-world graphical user interfaces using the Android learning environment. The proposed method with only 1.5B model size outperforms other state-of-the-art models such as GPT4-V or 17B CogAgent in the Android-in-the-Wild (AitW) dataset.
strength_weakness: strength_weakness: - Overall there is no major weakness. There are only several questions and potentially interesting empirical studies to look at. Check more details in the question sections.
questions: questions: - Could the authors compare more advanced LLM reasoning or planning algorithms like Chain of Thoughts (CoT), Tree of Thoughts (ToT), Reasoning as Planning (RaP), etc.?
- Following the previous question, is it possible to compare with those planning/search-based methods with the autonomous evaluator or the trained value model of DigiRL as value functions?
- What is the training time and compute requirement for online training?
- In Figure 7, what does the AWR reweighting mean? Is it simply AWR?
- With the auto-curriculum setup, it may be interesting to look at what types of data/replay are critical throughout the online learning process; with simple categorization like failure mode in Figure 5 or whatever characterization can be interesting.
As detailed in the "Challenges of stochasticity" paragraph in section 3, could the authors provide some studies on unpredictable distractor and technical glitches?
limitations: limitations: Limitations are discussed in the paper.
paper_summary: paper_summary: This paper introduces a novel autonomous reinforcement learning (RL) approach, DigiRL, for training in-the-wild device control agents. DigiRL first employs offline RL to fine-tune a pre-trained vision-language model (VLM as the agent) using stale task-specific data, and then further refines the agent through online RL by continuously interacting with parallelized emulators. DigiRL achieves a 49.5% absolute improvement in task success rate over existing state-of-the-art agents, establishing a new benchmark for digital agents in device control.
strength_weakness: strength_weakness: Major Points:

1. From a ML methodological point of view, the novelty/contribution of the paper is limited. To perform offline and off-to-on RL, the paper adopts a number of existing techniques such as AWR, doubly-robust estimators with little customization (e.g.,  hard filtering on the advantages instead of computing $\exp(A)$, which is mainly indended for easier implementation), all well-known to the community. The only thing seems "new" is training value functions with cross-entropy losses, also directly taken from [1], and the equations in line 250-251 seems to be qustionable (see my comments in the Questions section). Moreover, no theoretical insight is provided to elucidate why these specific designs are chosen.

[1] Stop regressing: Training value functions via classification for scalable deep rl, 2024.

2. Limited Scope. The entire paper focuses on a very specific domain (autonomous device control). The scope of the proposed method might be too narrow to be of general interest to the ML/RL community.

Minor Points:

- In section 4.2, how to properly balance the two estimators, one with higher variance and one with higher bias to achieve the optimal result? What's the hyperparameter profile of the combined estimator? Have you tried any alternative designs and can you give theoretical insight to justify this specific design choice?

- Regarding the offline and off-to-on RL setting: to my knowledge, the main point of offline RL is to leverage a large body of stale data to safely and efficiently pretrain a RL agent. Therefore, for off-to-on RL, where online RL operates as the fine-tuning stage, one should use data far less than the offline pretrained dataset to ensure the setting is meaningful. The fact that authors intentionally use the same amount of data for both offline and online stage, which assumes access to a large amount of online data might make the offline pretraining unecessary. To see this, I recommend the authors to directly perform online RL on the combined dataset and it's highly possible that such "purely online" agent outperforms its off-to-on counterpart. 

-  The authors spend quite a few words discussing the challenges of stochasticity and device control as a POMDP. However, I do not see any specific design or techincal contribution targeting such problems.
questions: questions: 1. In line 250-251, the CE loss pairs r with log V and (1-r) with log (1-V). Intuitively, this means one would like to make the distribution of r and V as close as possible (when r-> 1, V -> 1 and vice versa). This seems to contradict the claim in line 240-241 that "Intuitively, if a rollout attains a high value of A(sh, ah, c), it means the value function V is small".

2. How do you perform the train/test task split, if not random split? It is odd to see in Table 1 that almost all testing performance clearly surpass the training performance (normally should be the opposite), which suggest that the testing tasks are in general easier than the training tasks and not i.i.d?
limitations: limitations: N/A
paper_summary: paper_summary: This paper shows the potential of Reinforcement Learning (RL) for designing an effective digital agent for in-the-wild control through Graphical User Interfaces (GUIs). The proposed approach relies on the advantage of the pre-trained visual language models (VLMs) while tackling real-world stochasticity by training an RL agent that interacts with an environment instead of relying on static demonstrations. Accordingly, this work proposes a novel autonomous RL approach, namely DigiRL, for training device control agents, which consists of two stages: an offline RL phase where the agent is trained on static demonstrations, followed by an in-the-wild, offline-to-online RL stage for training the agent through interacting with an environment. Consequently, this work also introduces a scalable and parallelizable Android learning environment with a reward model (evaluator) based on a robust VLM-based model. To show the effectiveness of the proposed method, an evaluation of different tasks given diverse instructions is carried out from the Android in the Wild dataset on real Android device emulators. The results show a significant improvement of DigiRL compared to the existing state-of-the-art agents, including wrapped proprietary VLMs such as GPT-4V and Gemini 1.5 pro. The paper claims to be the first to succeed in developing an autonomous offline-to-online RL approach to enable state-of-the-art performance on device control problems.
strength_weakness: strength_weakness: - Although the POMDP definition sounds correct when defining the problem, a contextual POMDP is even more appropriate for such a problem [1].
- A pseudo-code or an illustrative diagram should be added to facilitate understanding the method.
- It is not clear how the policy and the value network are conditioned given the context $c$. (implementation-wise)

[1] Hallak, Assaf, Dotan Di Castro, and Shie Mannor. "Contextual markov decision processes." arXiv preprint arXiv:1502.02259 (2015).
questions: questions: - How are the policy and the value network conditioned given the context $c$?
- Is the RL agent trained in a multi-task fashion (which I believe is true)? I mean the agent is trained with more than one task concurrently accessing fully or partially the same models.
- If yes, do you think classical Multi-task learning or Multi-task reinforcement learning approaches would enhance the performance even more?
limitations: limitations: I believe the authors discussed the method's limitations in the final section. I agree with the authors regarding the impact of such application on the economy, society, and privacy, and that needs careful review in the future to limit any harm.

Comments: title: Raised my score
Rebuttal
Thanks for the rebuttal
Discussion period ends soon
Link to annonymous code
references
references
comment: Thanks for the rebuttal. I have carefully read all the responses and reviews. I will raise my score. With that being said, I will leave it up to the AC to decide whether the contribution of developing agents is strong enough for the paper's acceptance. I also lowered my confidence as my expertise mostly lies in RL.
We thank the reviewer for the response. We appreciate your score increase!
Thank the reviewer for reading our rebuttal We are glad that our rebuttal has solved most of your concerns. We appreciate your score raise and voting for acceptance!
Dear Authors,

Thanks a lot for answering my questions and addressing my concerns! 

Given the authors' responses to my questions and those of other reviewers, I will increase my score from 6->7 while keeping my confidence level, as it is similar to Reviewer 251X; my background is RL.
Thank you for the rebuttal and I appreciate the efforts of providing many new experiments in the PDF, please do add these to the revised version. I think most of my concerns are properly addressed. Regarding the technical novelty, since my expertise mainly comes from RL algorithm research, and much less from developing AI agents/systems, I might not be in the best position to make the judgement. 

Nevertheless, given all information provided, I will raise my score 4->5 but lower my confidence 4->3, and vote for acceptance.
We thank the reviewer for recognizing our efforts in the rebuttal and additional new results to address your previous concerns, and are glad that the additional results address the concerns. Since there is still one more day, we are also wondering if there would be some other discussion or evidence that we can provide in this period to help improve your score of our paper further. Please let us know. We would be very grateful if you are willing to upgrade your score. Thanks a lot!
Thanks for the ethics review and the recognition that the benefits of our work outweigh the risks. We will definitely add additional sentences in the paper to discuss these points about ethical concerns as suggested.
The rebuttal addressed most of my previous concern. Thanks!
Dear reviewer 251x,

Apologies for bothering you! Since the discussion period will end in two days, we would be grateful and would sincerely appreciate if you could respond to our rebuttal, leaving us enough time to address any remaining questions.

Thanks, Authors
Dear reviewers,

Apologies for bothering you! Since we are getting close to the end of the discussion period, we would be grateful and would sincerely appreciate if you could respond to our rebuttal, leaving us enough time to address any remaining questions.

Thanks, Authors
https://anonymous.4open.science/r/digirl-anonymous-7ED0/
Here is the link to our code promised for Reviewer zDzF
[1] ‘WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents’. NeurIPS 2022.

[2] Mind2Web: Towards a Generalist Agent for the Web’. NeurIPS 2023.

[3] GPT-4V(Ision) Is a Generalist Web Agent, If Grounded. 2024, ICML 2024.

[4] ‘TravelPlanner: A Benchmark for Real-World Planning with Language Agents’. ICML 2024.

[5] Data-Driven Offline Optimization For Architecting Hardware Accelerators’. ICLR 2022.

[6] Chip Placement with Deep Reinforcement Learning’. ICML 2022.

[7] A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis. ICLR 2024 (Oral).

[8] Schulman, John, et al. ‘High-Dimensional Continuous Control Using Generalized Advantage Estimation’.
[1] Zhang, Chi, et al. ‘AppAgent: Multimodal Agents as Smartphone Users’. 

[2] Shinn, Noah, et al. ‘Reflexion: Language Agents with Verbal Reinforcement Learning’.