DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agent
Authors: Taiyi Wang
Zhihao Wu
Jianheng Liu
Jianye HAO
Jun Wang
Kun Shao
Venue: ICLR 2025 Poster (2025)
Keywords: Mobile Agent
LLM
Reinforcement Learning
Fine Tuning
Distributed Training
Primary area: applications to robotics, autonomy, planning
TL;DR: We present an asynchronous distributed RL fine-tuning framework for mobile agents completing on-device control tasks. Using live interactions and an advanced off-policy RL approach, we ensure stable, scalable online learning for on-device agents.

Abstract: On-device control agents, especially on mobile devices, are responsible for operating mobile devices to fulfill users' requests, enabling seamless and intuitive interactions. Integrating Multimodal Large Language Models (MLLMs) into these agents enhances their ability to understand and execute complex commands, thereby improving user experience. However, fine-tuning MLLMs for on-device control presents significant challenges due to limited data availability and inefficient online training processes. This paper introduces DistRL, a novel framework designed to enhance the efficiency of online RL fine-tuning for mobile device control agents. DistRL employs centralized training and decentralized data acquisition to ensure efficient fine-tuning in the context of dynamic online interactions. Additionally, the framework is backed by our tailor-made RL algorithm, which effectively balances exploration with the prioritized utilization of collected data to ensure stable and robust training. Our experiments show that, on average, DistRL delivers a 3$\times$ improvement in training efficiency and enables training data collection 2.4$\times$ faster than the leading synchronous multi-machine methods. Notably, after training, DistRL achieves a 20\% relative improvement in success rate compared to state-of-the-art methods on general Android tasks from an open benchmark, significantly outperforming existing approaches while maintaining the same training time. These results validate DistRL as a scalable and efficient solution, offering substantial improvements in both training efficiency and agent performance for real-world, in-the-wild device control tasks.

Published as a conference paper at ICLR 2025
DISTRL: AN ASYNCHRONOUS DISTRIBUTED RE- INFORCEMENT LEARNING FRAMEWORK FOR ON- DEVICE CONTROL AGENTS
Taiyi Wang1.2+1, Zhihao Wu Jianheng Liu', Jianye Hao Jun WangS, Kun Shao University of Cambridge, Powersense Technology Limited, Huawei Noah's Ark Lab, Univeersity College London, AI Centre, University College London, Tianjin University
ABSTRACT
On-device control agents, especially mobile devices, are responsible for op- crating mobile devices to fulfill users' requests, enabling seamless and intuitive interactions. Integrating Multimodal Large Language Models (MLLMs) into these agents enhances their ability to understand and execute complex commands, thereby improving user experience. However, fine-tuning MILMs for on-device control presents significant challenges due to limited data availability and ineffi cient online training processes. This paper introduces DISTRL, novel frame- work designed to enhance the efficiency of online RL fine- tine-tuning for mobile de- vice control agents. DISTRL employs centralized training and decentralized data acquisition to ensure efficient fine-tuning in the context of dynamic online inter- actions. Additionally, the framework backed by our tailor-made RL algorithm, which effectively balances exploration with the prioritized utilization of collected data to ensure stable and robust training. Our experiments show that, on average, DISTRL delivers 3x improvement in training efficiency and enables training data collection 2.4 faster than the leading synchronous multi-machine methods. Notably, after training, DISTRL achieves 20% relative improvement in suc- cess rate compared to state-of-the-arr methods on general Android tasks from an open benchmark, significantly outperforming existing approaches while maintain- ing the same training time. These results validate DISTRL as scalable and ef- ficient solution, offering substantial improvements in both training efficiency and agent performance for real-world, in-the-wild device control tasks. The code is available htttps::/ com/ - 20t./220000/////eeeeeee
INTRODUCTION
The integration of Large Language Models (LLMs) into agents capable of complex tasks has gained momentum with initiatives like AutoGPT (Yang et al. 2023a), HuggingGPT (Shen et al. 2024), and MetaGPT (Hong al. (023), etc. These agents extend beyond language processing to per- form sophisticated functions, leveraging their reasoning abilities to interact with and manipulate environments effectively.
One of the key factors driving this trend is the advent of Multimodal Large Language Models (MLLMs), which can process diverse inputs such as text, images, audio, and video, thereby sig- nificantly expanding the scope of LLM applications (Alayrac al. 2022] Achiam al.) 2023 Zheng et al. 2024 Li et al. 2023). This versatility also enables MLLM-based on-device control agents-ing inttl---itellieenn systems embedded within mobile devices that manage and operate applications to execute user commands seamlessly
interact more naturally and efficiently with their sur- roundings, completing more complex tasks that require a deeper understanding of context and the ability to learn from interactions. For instance, agents designed operate smartphone applications can interpret screenshots from the operating system, demonstrating flexibility and adaptability that make them valuable tools in wide range of scenarios (Yang et al. 2023b; Wang al.) [024a] Chris- tianos et al. 2024 Papoudakis et al. 2025). These agents are essential for tasks such as automating
"Equal Contribution.
1 Corresponning Email: Taiyi. Www.@el..am.......o.o shaokun2 huawei.com

---

Published as a conference paper at ICLR 2025
On-dovice Control Taak
Request
Host Leamner Centralized Training
User: "Update Google Map Playstore"
GPUs
Auto Sustutors
Training Support
Prontized Trqgectryy
Algenitm
H
Update Policy
A-RIDE
Sequence
Workers: Decentralized Execution
0
DistRL: Scalable
Trajecsory Data
Agents
Distributed Fine- Tuning Framework
Mobile Devicesi
Emulatora
Figure 1: Overview of On-device LLM control with DISTRL.
app interactions, managing settings, and enhancing user productivity by providing intuitive control over device funetionalities.
Nonetheless, gap remains between LLMs' general reasoning capabilities and their effectiveness in GUI-based device control. While LLMs can process information, they struggle with rational behav- ior and error recovery in real-world environments. Consequently, prior work in building on-device agents often relies on constructing complex wrappers, combining them with prompting, search, or tool use. Without updating the model's weights, the effectiveness of these agents remains herently limited by the capabilities of the base model (Bai et al. 2022). Morcover, MLLMs prone to confidently generating incorrect content that deviates from desired outputs (Ouyang et al.) 2022 Ziegler al. 2019 Bai et al. 2022). To address these challenges, introducing reinforce- (RL)-based based inn-tuning methods, such as Reinforcement Learning from Al Feedback (RLAIF) (Yu al.] 2024 Lee et al. 2023), which evolved from Reinforcement Learning from Hu- man Feedback becomes essential. RLAIF leverages Al-generated feedback to align model outputs with intended behaviors and performance criteria. By incorporating RL-based methods, models can learn to optimize policies that not only perform tasks effectively but also adhere to specific expectations. tasks.
ment learning (RL)
One significant challenge in RL-based fine-tuning of mobile agents is the lack of support for effi- cient online fine-tuning. Existing tine-tuning approaches over mobile agents primarily rely on static offline datasets like AitW (Rawles al., 2024b) and AndroidControl (Li et al. (0224), which fail to capture the dynamic and evolving nature of mobile applications. Frequent updates and new elements like advertisements cause distribution shifts that offline-trained agents struggle handle, leading to failures in real-world deployments.
The second challenge the need for Reinforcement Learning (RL) algorithms that can operate efficiently within distributed framework. Asynchronous data collection introduces algorithmic difficulties: non-stationary data distributions hinder convergence, and delays between policy updates and data collection can cause agents to act outdated policies, degrading performance.
These challenges motivate us to develop DISTRI., as illustrated in Figure T DISTRL. is novel and scalable reinforcement learning (RI.) fine-tuning pipeline specifically designed for on-device mobile control agents on Android, featuring Centralized Training and Decentralized Data Acquisitions. Our main contributions are:
1. Scalable and Asynchronous Data Acquisition Architecture: DISTRL introduces decou- pled and asynchronous framework that deploys RL agents across heterogeneous worker devices for remote data collection, enhancing training efficiency and scalability ($
2. Advanced RL Algorithm for Centralized Training: We develop A-RIDE, a novel off-policy reinforcement learning algorithm tailored for distributed and asynchronous data utilization, which prioritizes significant experiences to improve sample efficiency while encouraging exploration ($5).
In practice we validate our framework using a T5- based multimodal generation architecture with 1.3B parameters (details in Appendix I.5.I) efficiently handle both vision and language inputs. To the best our knowledge, DISTRL is the first deployable and scalable autonomous RL fine- tuning system for online mobile device control tasks in distributed environment.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORKS
2.1 MULATI-MODAL. ON-DEVICE CONTROL AGENTS
Recent advancements in pre-trained Large Language Models (LLMs) and Multimodal LLMs (MI.MMs) have revolutionized on-device control agents, moving beyond early methods like be- havioral cloning or reinforcement learning (Osa ct al. 2018 Mnih et al.] 20155: Shi et al. 2017). Early agents simulated mouse clicks and typing (Shi et al. 2017 Humphreys et al. [022) but faced scalability and adaptability challenges.
Modern approaches use pre- -reitrained models with zero or few-shot prompting and fine-tuning for enhanced capabilities. WebGPT (Nakano et al.] 202I) employ fine-tuned models for web browsing, while WebAgent (Gur al., 2023) generates web code using T5. AppAgent (Yang et al. 2023b) and MobileAgent (Wang et al. 2024a) act as drivers, enabling the L.MSs to explore and act on mobile device environments. Training multimodal device control agents poses challenges like pixel- level interactions and variability in device ecosystems. Many rely on proprictary Vision-Languee Models (VLMs) and wrappers for GUI visual grounding (Driess al.
2023 Reid al. 2024). but
without fine-tuning, they are limited by the base models (Driess et al. 2023).
Several works fine-tune Vision-Languaen Language Models (VLMs) using demonstration data, such as AutoUI and CogAgent (Kapoor et al.) [024] Zhan & Zhang, [023); however, models trained on static datasets often struggle with the variability of real-world environments (Jiang et al., 2023 Oth- ers employ filtered imitation learning with autonomously collected data (Pan et al. 2024 Lai
jet al.) 2024). While DigiRL (Bai et al. (2024) supports on-device reinforcement learning (RL) fine-tuning, it encounters significant inefficiencies in parallel environments. Specifically, DigiRL's multi-machine setup relies on a fully synchronous data acquisition process, causing faster workers to idle while waiting for slower ones. This approach is impractical in real-world scenarios where task durations can vary by up to 100 times, ranging from seconds to over ten minutes.
Our extensive case studies reveal significant limitations in these prior works: advanced MLLMs like GPT-4V (Achiam et al., (023), SFT-based agents like AutoUl (Zhan & 'Zhang| 2023), and state-of-the-art mobile control agents like DigiRL. (Bai et al., (024) exhibit numerous failure modes (detailed analysis in Appendix I..I. In particular, DigiRL's lack of efficient distributed learning algorithms severely limits its scalability in dynamic, parallel settings. To address these limitations, we introduce DISTRL, a scalable and asynchronous RL fine-tuning pipeline designed for efficient
distributed mobile control agent training.
2.2 REINFORCEMENT LEARNING FOR ON-DEVICE AGENT FINE-TUNING
Reinforcement Learning from Human Feedback (RLHF) is widely used fine-tune LLMs to align with human preferences (Stiennon et al.) 2020 Ouyang et al.) (2022). In device control tasks, similar approaches use imitation learning from human-labeled evaluations, but RLHF is labor-intensive due to the need for human annotations (Ouyang et al. 2022 Bai et al. 2022). Recent advances in MLLMs (Alayrac al. 2022. Reed et al. [022 et al. 2023) show impressive multimodal capabilities but often produce incorrect outputs that deviate from human preferences (Ouyang et al.,
2022
Ziegler et al. 2019 Bai et al. 2022, Stiennon al., 2020). Reinforcement Learning from
Feedback (RLAIF)), using Al labelers as proxies, offers an alternative (Yu et al. 2024 Lee et al.,
2023 For on-device tasks, Al evaluators assess task completion using prompts and screenshots (Bai
al. 2024
Lee et al. 2023 Yu et al. 2024 Chen et al. 2024h.
Previous RL research focused on single-turn tasks, limiting their effectiveness for multi-step prob- lems (Reed et al. 2022; Liang et al. 2023). To address this, we developed a simplified off-policy multi-turn RL algorithm, A-RIDE, which learns from suboptimal online interactions, reducing com- plexity and accelerating convergence compared to previous value-based methods (Driess et al.) 2023 Yao et al. 2023). This approach is effective for large-scale applications like Android device control.
2.3 SCALABLE AND DISTRIBUTED RL FRAMEWORK
Scalable reinforcement learning frameworks like Ray RLlib (Liang et al. 2018) enable distributed training by parallelizing policy learning across CPUs and GPUs. RLlib supports various algorithms and efficiently manages neural network training, but it assumes that data collection can be simulated or parallelized within the same infrastructure, limiting their applicability to real-world on-device control tasks involving heterogeneous mobile devices with varying task durations and network con- ditions.
3

---

Published as a conference paper at ICLR 2025
Moreover, IMPALA (Espeholt et al. 2018 and IMPACT (uoo et al. 2019) are influential distributed reinforcement learning algorithms, but we are not able to adopt them directly as baselines for our distributed RL system in mobile device control due to their limitations in addressing the unique challenges of this environment. Specifically, these algorithms inadequately handle the fluctuating online experiences inherent in real-world mobile interactions, lack efficient buffer management for distributed cases, offer limited support for oof-po reinforcement learning models, and do not provide the necessary scalability and system optimizations required for distributed mobile control. Implementing them directly would necessitate substantial modifications to manage communication, schedule worker roles, and handle queues and replay buffers effectively. Therefore, we developed our own approach that builds upon the foundational concepts of IMPALA but extends them to meet the specific requirements of on-device control mobile environments.
3 PROBLEM SETUP AND PRELIMINARIES
As presented in Figure 2 we model the on- device control problem as a finite-horizon Markov Decision Process (MIP) M {S, A, No. H). Here, S denotes the set of GUI states, represented by screenshots or visual observations of the device screen. A represents the set of actions available the agent, such as touch events at specific screen coordinates. The state transition function T S x S - [0, 1] defines the probability of transitioning from one state to another given an action. The reward function R S x A + I provides sparse re- wards, typically positive upon task completion. Mo is the initial state distribution, and H is the finite horizon of the episode.
SSho motio video to Chane search abki
Thaining Taoks
Take Acdions
@xxcutton paraltel
Interaction
Feeebaok
Tra 2
Collection
Annotated trajectories
Rewards sasigned by Evalustor
Figure 2: Reinforcement Learning dynamics and auto evaluation for fine-tuning the on-device agent.
At each timestep i, the mobile agent observes state 8; E S, selects an action a, E A according to its policy π (a,) 8t), receives reward r. R(st, 0z), and transitions to the next state 8t+1. The agent's objective is to maximize the expected cumulative reward E, Ei= Tr over the episode. Given the asynchronous nature of distributed data generation in DISTRL, trajectories are collected under behavior policies and used to optimize a target policy This setup requires robust off-policy learning algorithms to correct for discrepancies between my and x.
A critical component of our RL framework is the ability to obtain reliable reward signals in real- time. To achieve this, we utilize Gemini-1.5-pro (Reid et al., (2024) as an autonomous evaluator to assess whether the agent has successfully completed the task at each state. The evaluator receives the current observation, composed of the task description and a screenshot of the device, and outputs reward signal. Specifically, the evaluator assigns reward rı 1 if the screenshot indicates suc- cessful task completion and = O otherwise. Details of how we implemented the auto-evaluation can be found in Appendix A.2
4 SYSTEM DESIGN
DISTRL is an asynchronous distributed reinforcement learning framework for scalable and efficient training of mobile agents. By decoupling trajectory collection from policy learning and doing both in parallel, it leverages distributed working machines for CPU-intense egent-environmmnt interac- tions and GPU servers for policy training. This separation optimizes efficiency, scalability, and resource utilization by aligning tasks with appropriate hardware. Moreover, such decoupled and asynchronous design offers several key advantages: it improves scalability as data collection scales with more working machines providing the mobile environment, even if there are large performance gaps between them, optimizes resource utilization by assigning tasks to suitable hardware, and it improves policy quality through richer, more diverse datasets from multiple devices, enhancing ro- bustness and generalization capabilities. The details of our system are presented as follows:
DISTRL employs a host-worker architecture consisting of a central Host Learner (Left side in Figure 3 and multiple Workers (Right side in Figure 3 which can be het-
As illustrated in Figure 3
4

---

Published as a conference paper at ICLR 2025
Hest Leurner (with GPU4
Fourne
Chilee
Clestar
Larp
Rople
Toaks
Reeasto Barkers
Bette greeen
Learner
Host
Figure 3: Illustration of the high-level workflow of DISTRL System.
erogencous devices: i.c machines of various specifications, running android emulators being connected with mobile devices, providing the android interaction environments. These components work together train agents through asynchronous data collection and distributed policy updates.
Host Learner orchestrates the policy training process using powerful GPUs. It maintains a Circular Replay Buffer (details in Appendix (A.4) that stores the trajectories collected from the workers. The training loop processes this data by applying reinforcement learning al- gorithms to update the policy. To manage incoming data efficiently, a FIFO Trajectory Queue receives experiences from the workers and organizes them for training.
Host Learner:
The host learner updates the policy using tailored regularization to promote worker exploration and priority-based sampling to efficiently utilize diverse data. Additionally, to maximize the use of the large-scale and diverse data collected, and to avoid excessive learning on redundant or similar data, the learner employs priority-based sampling techniques. Updated policies are then distributed workers, creating continuous cycle of experience collection and policy refinement. Detailed algorithmic design is presented in Section
Workers: Workers operate in parallel, each managing its own Android environments with Android Emulators or actual Android devices through multi-threading. Each thread in the workers executes the policy received from the host learner and interacts with the environment through an Agent. The agent queries the environment, receives observations, and generates actions based on the current policy. Each worker collects seeectorresssqqueneeee of actions, observations, and rewards-during during its interaction with the emulator.
To facilitate efficient simulation, workers use Environment Snapshots, allowing them to reset the emulator to specific states. The result trajectories from the collecting threads are asynchronously sent back to the host learner to be added to the replay buffer for training. The asynchronous architec- ture enables diverse worker machines to efficiently collect data independently without interference, maximizing their individual contributions to the system.
On the whole, DISTRL employs asynchronous RL to address the challenges of online RL in dy- namic, real-world settings. Each thread in workers operates independently. executing tasks and generating learning trajectories at its own pace, which accommodates variability in task durations and system latencies. Data produced by the working threads is queued and processed by the host learner, which updates the global policy based on the collected trajectories. The updated policy is asynchronously distributed back to the workers, allowing for independent and non-blocking policy updates. Further details are elaborated in Appendix A.3.2
5 METHODOLOGY
In this section, we introduce A-RIDE, the core reinforcement learning algorithm employed in DIs- TRL to fine-tune agents for device control tasks in distributed environments characterized by lim- ited on-device resources, asynchronous data generation, and distributed constraints, Traditional on- policy algorithms like Proximal Policy Optimization (PPO) (Schulman et al, (017)) and Advantage Actor-Critic (A2C) (Mnih) (016)) are inefficient in these settings due to their reliance on synchronous data collection and policy updates, leading to sample inefficiency and delayed learning. Unlike ex- isting off-the-shelf RL models that have been widely applied various domains (Wang & Yoneki, 2024 Wang & Shi 2021 Wang et al. 2024b 2025 Sun & Wang 2022 Sun et al. 2022), our approach, A-RIDE Advantage-basee Retrace Improved by Distributed Prioritized Experience Re- play), addresses these challenges through two key novel components: Retrace 5.3 and DPER ($ (.2)). enhancing exploration efficiency, maintaining policy robustness, and improving training efficiency through robust exploratory behavior and prioritization of informative experiences. This
5

---

Published as a conference paper at ICLR 2025
enables DISTRL to achieve stable and efficient learning in real-world device control tasks (see Ap- pendix A.4 for more methodology details).
5.1 A-RIDE: THE BACKBONE OF DISTRL
In general, our approach enhances policy gradient updates by extending the Generalized Advantage Estimation (GAE) framework (Schulman et al. 2015) to better suit asynchronous, distributed envi- ronments common in device control tasks. Learning from GAE to estimate the advantage function directly, we introduce another stable and smart way to estimate advantages by maintaining two sep- arate networks: one for the trajectory-lever value estimation (Ving), which serves as the labeler, and one for the state value function (s)).
Trajectory-Leee Level Value Estimation We maintain a trajectory-level value estimator Vimi. parame- terized by neural network 0. This estimator serves as labeler, assigning rewards to trajectories and filtering the replay buffer to retain only high-value trajectories. The trajectory-level value estimator is trained using maximum likelihood estimation (MLE) loss function:
min ((Vin)) = BB [r(* aH) log Ving( dg) + (1 r(5 (()) log (1 (sH aæ))]
where a() is the reward at the time horizon step H, and denotes the distribution over trajectories. This estimator helps identify and retain valuable trajectories for more efficient training.
State-Value Function Estimation We also maintain a separate value network V(s;; o). which estimates the state value function (ss)), representing the expected return from state 8;. Instead of directly regressing on scalar returns, we train the value network to predict the probability that the Monte Carlo return G, is positive, transforming value estimation into a binary classification task.
The value network is trained with the following loss function: L(V) = E T(, > 0] log ((s:; () (1 - GG, > 0)) log(1 V(s d))], where V(s;; o) is the predicted probability that G, > O, G. = Ekme> k is the Monte Carlo return from timestep I, and II[G, > 0] is an indicator function that equals if G2 and 0 otherwise. The value network parameters 6 are optimized by:
6" = arg ming (V)
Ad Advantage Computation With the trajectory-level rewards and state value estimates obtained, we compute the advantage function A(st, az) using one-step estimation: A(s,, at) Q(5,, a.) V(s)) = r(s,, a,) + +V(4))))) - V(s)), which correctly represents the advantage function as per the policy gradient theorem. Here, r(s,, uz) includes signals of immediate rewards (see A.4.2). The advantage and value functions are further modified by off-policy corrections, which will be claborated in Section 5.2
Policy Optimization with Robust Regularization Finally, the policy network optimizes the fol- lowing loss:
C = E, Ap:A(st, at) log 7 (ac [s)) - -8 E, H (TT(a)()))) + +A E | Pinvatia (ax))
Update direction and fine tuning loss
Regularization
Action Penalty
is the importance sampling ratio between the target policy and the behavior policy 14, HH is the entropy term for prevention of overfitting (see 5.3). Pinaaia (ar) imposes penalty on actions deemed invalid based on task-specific criteria, 3 controls the strength of entropy regularization, and X modulates the penalty's influence. The penalty is assigned using validation through pre-trained LLMs like Gemini (Reid al. 2024)), ensuring that inappropriate actions are penalized. This formulation encourages the agent to explore a diverse set of actions while constraining it to generate valid and meaningful commands, thereby enhancing both exploration and policy robustness when dealing with online sottttation.issss.
where Pt =
This formulation enhances our method over traditional GAE and DigiRL (Bai et al., 2024 by: In- corporating Importance Sampling: The term Pt ensures that the policy updates remain stable even when learning from off-policy data, which is common in asynchronous, distributed environments. Adding Entropy Regularization: The entropy term H(T((a))s)) encourages more explorations. Penalizing Invalid Actions: The penalty term Pinvaia (ax) discourages the selection of inappropriate or nonsensical actions. Details of achieving the penalties be found in Appendix A.4.2

---

Published as a conference paper at ICLR 2025
Push Policy
Worker Process
Host I Lerrer with GPUs
Replay Buffer
Diveributte Pritccitiio Experieaas Repllay (DPER)
Update
query
Priorities
Collect Trajectories
Stable Oef Ort-Pallyy Learning with Retrace(A)
Prioritized Sampling
Centralizu Trainer
Mx Ensinter
Stupsher
Figure 4: Backbone of DISTRL: A-RIDE Reinforcement Learning- based Fine-Tuning
5.2 OFF-POLICY CORRECTION: RETRACE
To enhance the estimation of the value function V(s;) in off- off-policy and asynchronous settings, we apply Retrace(A) A) corrections directly to (st)). The Retrace algorithm extends the TD(A) method off-policy learning by incorporating importance sampling ratios and trace decay parameter X. Specifically, we update (s*)) using the correction term 5, computed as: ((+)) < V(+)) + 5z, 5, =
Amn (1, pi) with À [0, 1] being the trace decay parameter, and ps is the importance sampling ratio as mentioned before. This correction term effectively adjusts the value estimates using future rewards and importance sampling, enabling off- teflpolicy learning while mitigating variance due to importance weights. By applying Retrace(A), improve the estimation of V(s+) in off-policy settings, enhancing the stability and convergence of the value network.
2 c:) [r) ( (sx)) where
5.3 DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY (DPER)
To improve sample efficiency, we employ Distributed Prioritized Experience Replay (DPER). For each trajectory т {(s, ar Tr 1++ we compute the priority p(T) as: p(r) w/ oo| + w2p + H, where |8| is the average absolute temporal-differene (TD) error over the trajectory, calculated as & Ft + (V{+++1) V(s)); p is the average importance sampling ratio pt; and H the average policy entropy, H, = - log iT (arp(8)) encouraging exploration by encouraging pol- icy uncertainty, thus avoiding early convergence to suboptimal policies during training in dynamic environments. The weights Wi, W2, and w/ balance the contributions of each component, which selected by grid-search (See Appendix [A.43)). Trajectories with higher priorities more frequently, focusing learning on the most informative experiences. Priorities are periodically updated based on the latest policy, recalculating them focus learning on the most informative experiences, ensuring continual adaptation to evolving behavior policies.
replayed
EXPERIMENTS
To evaluate the performance of DISTRL. on challenging Android device control tasks, we conducted extensive experiments. Our primary goal is to determine whether I. can produce agents that effectively learn from autonomous online interaction. We present the experimental environment in Section 6.1 the baseline and benchmarks in Section 6.2 validation of our evaluator in Section 6.3 training performance Section 6.44 and on-device task evaluations in Section 6.5 present ablation studies on approach's 's components in Section 6.6
Additionally, we
6.1 EVALUATION ENVIRONMENT
Our evaluation environment consists of host learner with 4 NVIDIA V100 GPUs for intensive policy training and two worker machines with 8 NVIDIA Tesla T4 GPUs and 96 vCPUs each, sup- porting parallel emulation. This setup leverages 192 vCPUs to run multiple emulators concurrently, enabling scalable distributed reinforcement learning experiments.
6.2 BENCHMARKS AND BASELINE METHODS
To comprehensively validate our approach, we utilize both the General and web shopping tasks for training and testing. Specifically, our training set is derived from enhanced online task instructions, which are composed of AitW (Rawles et al. 2024b), AndroidWorld (Rawles et al. 2024a). and expert-curated task sets. We fine-tune our model on this combined training set and evaluate per- formance on the corresponding test subsets derived from AitW. Our analysis focuses on training efficiency using the General Tasks, which include fundamental application operations and informa-
7

---

Published as a conference paper at ICLR 2025
tion retrieval tasks. Additionally, assess the agent's performance both General Tasks and Web Shopping Tasks to evaluate its capability in handling domain-speciffe instructions, addressing the significant task distribution gap. Detailed descriptions of the datasets used are provided in Ap- pendix A.5 Our baseline methods
inethudsSinclude: include:
DigiRL (Bai et al. 2024): The state-of-the-ar framework prior to work, which integrates RL fine-tuning with visual language models (VLMs) and provides a reproducible training process.
We consider its both single and multi-machine settings online mode.
AutoUl (Zhan & Zhang
2023):
A simple explorative mobile agent equipped with VLMS under
supervised fine-tuning.
GPT-4V (OpenAl et al.
2024) and Gemini 1.5 Pro (Reid et al. 2024): Equipped with explo-
ration drives- driyes-AppAgent
Yang al. 2023b) facilitate learning from the environments.
6.3 VALIDATION OF OUR EVALUATOR
In our experiments, used the last screenshot along with the last two actions input the VLM Evaluator (Gemini) for prompting. This approach provides concise yet informative context for the evaluator. As shown in Figure 5 when testing different policy models on General subsets from AiTW, incorporating additional context, such as longer trajectory information, negatively impacts evaluation accuracy. Specifically, the discrepancy between the evaluator's 's output and human assess- ments was less than 2% under our setting. On one hand, our algorithm leverages score assignments based on final steps and states, achieving a balance between computational efficiency and evaluation accuracy, on the other hand providing more context led to performance drops, significantly higher computational costs, and increased usage of evaluator LLMs, thereby straining our budget.
Success Rate (%)
Gemin 62.1
Human
53.7
25.5
66.3
46.2
44.7
.8.2
X=1
X-2
X=3
Figure 5: Performance correlation between automated and human evaluation across different agents, using the last X images (validated on the whole General subsets from AiTW)
In fact, empirical tests show that large models like Gemini tend classify tasks successful by identifying evidence success in provided screenshots. The more information Gemini receives, the higher the probability it judges the task as successful. In few-shot scenarios with seven eight images, adding more tokens causes token explosion, leading to hallucinations. These are reasons for choice to limit context (instead of longer or even entire trajectories).
6.4 TRAINING PERFORMANCE
Training efficiency is crucial in reinforcement learning, particularly in complex environments, and is measured by the rate of improvement over time. We compared DISTRL with the existing Di- giRL framework. Our results show that DISTRL significantly boosts training efficiency with its distributed, asynchronous design, leveraging multiple machines and GPUs.
In subfigure (a), by 6k seconds, DISTRL achieves a success rate 30% and 40% higher than DigiRL in multi- and single-machine settings, respectively. Even compared with DigiRL that is enhanced with our asynchronous framework (by integrating the DigiRL algorithm into the DISTRL. framework to isolate the framework's 's benefits-named as DigiRL-DistRL Async), DISTRL achieves 10% higher result with faster convergence speed. Subfigure (b) illustrates the proportions of success rates above 60% and 80% in different training phases, representing key stages in the learning curve. DISTRL maintains higher proportions of success rates above these thresholds compared to DigiRL, showing a faster convergence and higher stability. These improvements are attributed to our asynchronous architecture and tailor-made algorithm for efficient data collection and sampling.
Subfigure (c) highlights DISTRL's 's superior data collection efficiency, accumulating 800 trajecto- ries in 6k seconds, compared to DigiRL's 300 in multi-machine setting, While subfigure (d)
A qualitative comparison among methods and details be found in Table 3 Appendix A.5.1
8

---

Published as a conference paper at ICLR 2025
Wall-Clock Time Comparison
Training Efficiency (32 Emulators)
Diuris (Dung
(Dtjn)-Ovviil) Azye
propertion Above 6PP, Success
Digfis DHutss
Aboue 80% Succeess
Digint (Sing)))
Above 40%
0.4
0.3
Time (Seconds)
0k-24
2k-41
[Seconds)
41-66
(o) Wall-Clock Time Comparison
(b) Training Efficiency Comparison
Trajectories Collection from 32 Emulators
Scalability Evaluation of Emulation Speed
800
DietAL (Ours)
Emuulation Speed (trajimin)
DistRL (Ours)
DigiAL (Mult))
DigiRL (Mults)
0020
DigiAL (Singla)
Ideal Upper Bound
To-
14
Time (Seconds)
128144
192
(c) Trajectories Collection Ability Comparison
Scalabiliity of Different Training Frameworks
Figure 6: Training performance (32 emulators) between the current state-of-the-art method (Di- giRL.) and DISTRI., highlighting the enhanced efficiency of DISTRL's distributed framework during online training. (a) Wall-clock time comparison (b) Training efficiency comparison. (c) Accumu- lated trajectories collection ability comparison. (d) Scalability of different training frameworks
demonstrates DISTRL's scalability. It achieves collection speed of approximately 7.7 trajecto- ries per minute with 192 CPUs, with nearly linear scalability, closely approaching the Ideal Upper Bound-perfect linear scalability with no overhead from communication or error handling. This ideal upper bound is determined by assuming each CPU operates independently and continuously with stable speed, profiled by measuring the collection speed when a single CPU handles the task.
Additionally, Table I also presents the final training performance at convergence or after extended training time budgets (will be explained in the subsequent subsection), demonstrating the superior long-term performance of DISTRL. compared to the baselines.
6.5 PERFORMANCE EVALCATION OF AGENTS TRAINED WITH DISTRL
We evaluate the end-to-end performance of agents trained with DISTRL against other frameworks, including on-device control agents, using subsets of both the AitW training and test sets. The pri- mary metric for evaluation the success rate across General and Web Shopping tasks. To ensure a fair comparison, we allocate extensive fine-tuning time for DigiRL in single-machine and syn- chronous multi-machine configurations, typically allowing 2 times the convergence time required by our asynchronous DISTRL multi-machine setup. Despite this generous tuning period, baseline methods often fail to achieve stable performance due to inherent inefficiencies in their synchronous designs, which hinder effective utilization of additional training time.
The results Table and Figure (a) demonstrate the superior performance of our DISTRL frame- work over other agents across all evaluated settings. In the General test set, DISTRI. achieves a success rate of 73.2%, showing a relative improvement of approximately 19.6% over DigiRI. (multi) and 2% over DigiRL (single). In the Web Shopping test set, DISTRL attains success rate of 68.5%, outperforming DigiRL (multi) by about 14.4% and DigiRL (single) by 14.9%. This significant enhancement is attributed to DISTRL's design for pure asynchronous task collection pro- cedures and its advanced algorithm for efficiently utilizing diverse incoming trajectories, leading to better generalization and higher success rates.
The prompting-ber erably lower success rates, not exceeding 45.3% in any test setting. These methods lack adaptive learning capabilities on real-time large-scale interaction data, leading to poorer performance and higher susceptibility task variability. AutoUI, another learning- pervised knowledge, also underperforms with success rates below 45% likely due to less efficient exploration strategies and inadequate handling of diverse user instructions.
methods, such as AppAgent combined with GPT-4v or Gemini, show consid-
basedd based agent fine-tuned by

---

Published as a conference paper at ICLR 2025
General
Web Shopping
Framework Type
Framework Name
Training
Training
Test
AppAgent + GPT-4v
414
43.0
31.2
35.2
Prompting
AppAgent + Gemini
39.1
45.3
30.5
32.0
AutoUII
38.3
40.6
42.2
44.5
DigiRL (single,online)
64.6
59.9 2.1
63.3 d: 1.5
59.6 4 3.1
Learning
DigiRL (multi)
67.7 +
61.2
64. 1.1
59.9 - 2.8
DistRL (Ours)
75.5 0.2
73.2
69.8 -:
68.5 +
Table Main comparisons regarding the success rate of different agents across various settings. Each experiment is repeated three times and the mean and standard deviation are reported. Results are evaluated with our autonomous evaluator with the 128 user instructions in the train and test set.
Agent Porformance Evaluated on the AitW Benchmark
Success Rate over Wall-Clock Time
N Sucoess Rale
Appilgrt
50
Dedi (Oun)
Shopping
Duttt (Ours)
Time (Sosonds)
Agent Performance Comparison
(b) Ablation Study
Figure 7: (a) Comparison of trained agent performance when evaluated on the AitW benchmark. Ablation Study of DistRL
6.6 ABLATION STUDIES
To understand the contributions different components DISTRL, we conduct ablation studies by systematically removing altering key elements of the algorithm, such the enhanced Re- trace algorithm and Distributed Prioritized Experience Replay (DPER). The results, summarized in Figure (b), demonstrate the significant impact of each component the task success rate.
Distributed Prioritized Experience Replay (DPER) is crucial for accelerating training conver- gence. Removing DPER results in an 8% decrease in the success rate, indicating that prioritizing trajectories with higher TD errors and smaller policy discrepancies enables faster and more efficient learning by focusing updates the most informative experiences. With the entropy term, the pri- oritization mechanism promotes exploration based on the evolving policy distribution, preventing stagnation during training.
Retrace Algorithm essential for maintaining training stability. Ablating the Retrace algorithm leads to a 6% drop in success rate and causes sharp decreases in performance during training. This instability arises because Retrace provides off off-po correction, ensuring stable updates even when the agent receives a large number of diverse trajectories.
Overall, the ablation results confirm that both DPER and the Retrace algorithm are integral to the efficiency and robustness of DISTRL.
CONCLUSION AND FUTURE WORK
In this paper, we introduce DISTRL, an efficient distributed reinforcement learning framework tai- lored for mobile-based agents tasked with user instructions. Our primary contribution the devel- opment of robust and scalable pipeline that seamlessly bridges the gap between real-time inter- actions on mobile devices or emulators and distributed training infrastructures, ensuring efficient and adaptive learning. For future work, aim to extend the generalization capabilities of DISTRL to a broader range of tasks, focusing on enhancing both the training pipeline and the underlying algorithmic architecture. Additionally, tegrating many more Multimodal Large Language Models (MLLMs), allowing for a wider range of applications and evaluations on diverse benchmarks.
envision evolving DISTRL into core backbone for in-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (Grant Nos. 62422605, 92370132), AWS cooperative funding supports and POWERSENSE TECHNOLOGY LIMITED.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report (2023). URL https://api. semanticscholar org/CorpusII), 257532815, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Paul Luc, Antoine Miech, lan Barr, Yana Hasson, Lauren Menschen, Sander Dieleman, Karen Simonyan, and Aaron den Oord. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204.14198 2022.
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Di- girl: Training in-the-will device-control agents with autonomous reinforcement learning. arXiv preprint ar arXivv2406. 1896, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Neal DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586,, 2022.
Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. In NeurlPS 2024 Workshop on Open-World Agents, 2024.
Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, and Kun Shao. Lightweight neural app control. arXiv preprint arXiv:2410.17883, 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.03377. 2023.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference machine learning, pp. 1407-1416. PMLR, 2018.
Ido Gur, Shay Mazor, Adi Jerbi, Amir Globerson, and Jonathan Berant. Learning to use the web for conversational question answering. In Proceedings of the 61 st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1459-1476, 2023.
Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, al. Metagpt: Meta programming for multi- agent collaborative framework. arXiv preprint arXiv:2308 0000:223800055,, 2023.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint 1/Xxv:8803.0093. 2018.
Peter C Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Adam Santoro, and Timothy Lillicrap. A data-driven approach for learning to control computers. In International Conference on Machine Learning, pp. 106999466999 PMLR, 2022.
Amy Zhang Jiang et al. Active reward learning from multiple teachers. 1xXi::3301.12445 2023,
arXiv preprint
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. arXiv preprint arXiv:2402. /7553, 2024.
11

---

Published as a conference paper at ICLR 2025
Hanyu Lai, Xiao Liu, lat Long long, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: Bootstrap and reinforce a large lan- guage model-based web navigating agent. arXiv preprint 0et::2444.03444 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune. Abhinav Rastogi, et al. Rlaif: Scaling reinforcement learnnnnnn from human feedback with ai feedback. arXiv preprini urXiv:2309.00267 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Li Fei-Fei. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12590) 2023.
Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Camphell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.036 03679, 2024.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gon- zalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning, pp. 3053-3062. PMLR, 2018.
Kensen Liang et al. Code Ilama: Open
foundation models for code.
arXiv preprint
arXiv:2308. 12950, 2023.
Michael Luo, Jiahao Yao, Richard Liaw, Eric Liang, and Ion Stoica. Impact: Importance weighted asynchronous architectures with clipped target networks. arXiv preprint arXiv: 1XXXv:1912.00167. 2019.
Volodymyr Mnih. Asynchronous methods for deep reinforcement learning.
arXiv preprint
1007:.16,..01783, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Chris O' O'Brien, Christina Kim. Christo- pher Hesse, Sandhini Agarwal, Jonas Schneider, William Clark, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv: 12.09332, 2021.
Achiam OpenAl, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkar, et al. Gpt-4 technical report. 2023. URL: https://arxiv. org/abs/2303.0877 000.0/07423000...... 2024.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2): 1-179, 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155: 2022.
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. In First Conference on Language Modeling, 2024.
Georgios Papoudakis, Thomas Coste, Zhihao Wu, Jianye Hao, Jun Wang, and Kun Shao. Appvlm: A lightweight vision language model for online app control. arXiv preprint arXiv:2502.06335, 2025.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint 14X5 2024a.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36, 2024b.
12

---

Published as a conference paper at ICLR 2025
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Quoe Le, Wojciech M Czarnecki, Tom Schaul, Trevor Cai, David Budden, Gabriel Barth-Man Martn, et al. generalist agent. arXiv preprint arXiv:2205.06175, 2022.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Tom Schaul. Prioritized experience replay. arXiv preprint arXiv:1511.0599 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- gpt: Solving tasks with chatgpt and its friends in hugging face. Advances Neural Information Processing Systems, 36, 2024.
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pp. 3135-3144. PMLR, 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M Ziegler, Ryan J Lowe, Caleb Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems, volume 33, pp. 3008-3021, 2020.
Hao Sun and Taiyi Wang. Toward causal-aware rl: State-wise action-refined temporal difference.
arXiv preprint arXiv:2201.000 220400034, 2022.
Hao Sun, Ziping Xu, Taiyi Wang, Meng Fang, and Bolei Zhou. Supervised q-learning for continuous control. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406.0100 201144006,11,, 2024a.
Taiyi Wang and Jiahao Shi. Solving maximal stable set problem via deep reinforcement learning. In ICAART (2), pp. 483-489, 2021.
Taiyi Wang and Eiko Yoneki. Ia2: Leveraging instance-aware index advisor with reinforcement learning for diverse workloads. Proceedings of the 4th Workshop on Machine Learning and Systems, pp. 10-17, 2024.
Taiyi Wang, Jianheng Liu, Bryan Lee, Zhihao Wu, and Yu Wu. Ocmdp: Observation--oonsraiined markov decision process. arXiv preprint mrXiv:2411.07087,
Taiyi Wang, Liang Liang, Guang Yang, Thomas Heinis, and Eiko Yoneki. A new paradigm in tuning learned indexes: reinforcement learning enhanced approach. arXiv preprint arXiv:2502.0500) 2025.
Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions. arXiv preprint arXiv:2306.02224, 2023a.
Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771 2023b.
Shunyu Yao. Howard Yu, Yuan Wu, Vasu Vinay, Yuan Cao, Karthik Narasimhan, et al. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 10601, 2023.
13

---

Published as a conference paper at ICLR 2025
Tianyu Yu, Haoye Zhang, Yuan Yao. Yunkai Dang. Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17222 2024.
Zhuosheng Zhan and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXiv:2309. //46,, 2023.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is generalist web agent, if grounded. arXiv preprint 44X1112001.01614, 2024.
Daniel Ziegler. Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, and Paul F Christiano. Fine-tuning language models from human preferences. arXiv preprint arXiv: 190900853, 2019.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1
CASE STUDY
Task: Update Google Map in Playstore
DistRL
DigiRL
APP
STUCK
AutoUI
EARLY TERMIRATION
GPT4-V
Figure 8: Case study general app operation tasks.
Figure illustrates a type of common case where baseline methods always fails, highlighting the challenges for these device-control agent in real-time app operation tasks.
DigiRL, which trained older versions of the system, fails due to discrepancies between its learned knowledge and the current environment. This mismatch in training data leads to a signifi- cant error: DigiRL mistakenly opens Google Photos instead of the Play Store. Since the two apps share similar icon features. After opening the wrong app. DigiRL. continues to operate within the incorrect environment, ultimately getting stuck in the settings menu of Google Photos. This reveals a significant limitation of offtine- when visual similarities between app icons lead misclassification. AutoUI shares similar issue where it struggles to correctly identify the target application. In this case, it opens Google Maps directly instead of navigating through the Play Store. Its lack of adaptability to new tasks or novel instructions results in failure.
agent in adapting to updated environments, especially
The AppAgent with GPT-4V takes an alternate route by resorting to web searching. which di- verges from the intended method of updating the app. Eventually, this leads to the agent becoming stuck within the Google Maps application itself, indicating that while GPT-4V was able explore different avenues to achieve the goal, it did not follow the expected approach due to the lack of app-specific knowledge.
While the DISTRL, which was actively trained on the real-time online-training, could conduct the intended operations accurately and successfully.
environment through
15

---

Published as a conference paper at ICLR 2025
Task: Start a new chat in message app.
DistRL
DigiRL
AEDEE
STUCK
AutoUl
GPT4-V
WRONG
P
APP
4
M
STUCK
Figure 9: Case study on general app operation tasks.
The case study in Figure 9 further illustrates the comparative performance on simpler general app operation task - starting new chat in the messaging app.
DISTRL successfully completes the task by efficiently navigating through the app's interface. opens the correct messaging app and enters the "New Conversation" screen without getting stuck.
In contrast, DigiRL. manages to open the correct messaging app but fails to proceed, getting stuck when attempting start the new chat. This is due to DigiRL's reliance outdated training data, as it was trained on an older version of the app's interface. In the outdated UI, the intended action (starting a chat) involved interacting with elements in a different layout, and DigiRL cannot adapt to the updated version. This demonstrates the pitfalls of relying primarily on offline data for train- ing without sufficient online fine-tuning to adapt new UI changes, as seen in modern apps that frequently update their designs.
AutoUI, on the other hand, fails immediately by selecting the wrong app. It opens the Contacts app instead of the messaging app, leading to failure in completing the task from the very beginning. This reflects a limitation in AutoUT's task understanding and its inability to differentiate between similar apps, further highlighting the weakness of frameworks that lack robust decision-making process or real-time adaptability.
GPT-4V, though not specifically trained for app-specific tasks, performs well in this scenario due to its generalization capability. It opens the correct messaging app and navigates to the "New Con- versation" " screen successfully. GPT-4V is more flexible and suitable for simpler, general-purpose tasks. However, this general-purpose approach may not scale well for more complex tasks where app-specific expertise and interaction nuances are required.
The real cases in the figures emphasizes the critical importance of efficient real-time online learning.
Figure 10 shows case study the web shopping task. DISTRL demonstrates relatively smooth and fluid operations, progressing through the steps without hesitation. While it ultimately encounters early termination due to reaching the step limit (horizon), it performs each step with clear transi- tions and effectively navigates through the sequence. DisTRL shows strong task comprehension and adaptation throughout, but its misunderstanding on the task requirement prevents it from fully com-
16

---

Published as a conference paper at ICLR 2025
Task: Go to ebay.com, search for "usb-c to usb-a", and select the first entry
DistRL
SKIPPD
EARLY TERNINATION
DigiRL
STUCK FEW
AutoU!
1:03
IN
UNIELLTTED
FOR FEW
STUCK
GPT4-V
Figure 10: Case study on web shopping tasks.
pleting the task. This behavior emphasizes the efficiency of DISTRL's operations and its capacity to generalize across unseen web shopping tasks, even though the task is terminated early.
In contrast, DigiRL faces several challenges during the task. It frequently steps back and forth between pages, struggling with the microphone input. These actions result in unnecessary delays and inefficiencies, which eventually lead it to reach its step budget without successfully completing the task. The back- behavior indicates a lack of robust policy adaptation, which causes to get stuck in a loop, unable to make meaningful progress.
AutoUI, on the other hand, wanders into unrelated apps before eventually returning to the task. The lack of focus results in it spending multiple steps outside the task's scope, which ultimately con- tributes to its failure. This signifies weaknesses in both task planning and execution, as it struggles with distractions and incorrect app selections.
GPT-4V follows similarly smooth approach DISTRL, but it becomes stuck after selecting wrong entry into eBay, which triggers the cookie settings of the website. Although GPT-4V suc- cessfully navigates through several steps, it ultimately fails to get around the emergent pop-up, highlighting its limitation in handling web-specific tasks that require precision and app-specific un- derstanding.
In summary, while DISTRL and GPT-4V demonstrate smoother task execution, only DISTRL man- ages to maintain a consistently structured progression, even though it faces early termination. Mean- while, DigiRL struggles significantly, exhibiting inefficient operations that lead to step budget haustion without meaningful progress. This case study emphasizes the importance of the ability to adapt policies in dynamic environments to complete tasks successfully within budgeted steps.
17

---

Published as a conference paper at ICLR 2025
A.2 AUTO REWARD LABELING
A.2.1 EVALUATION WITH
To generalize the evaluator across a wide range tasks without manual rule definitions, we leverage a pre-trained LLM with appropriate prompting. The prompt is designed to instruct the LLM to act as expert evaluator, i.e., Gemini-1.S-Pro (Reid et al. 2024), in our practice. An example of such a prompt is provided below:
You't re an expert in evaluating whethez the Screenshot auccessfully completes the Taak. Exemexxapeee
Screenshot: (train_l.pag)
Task: Send message Evelyn.
Q: what should expect to see en the screenshet if i've sent a message to Evelyn? should expect see an open maasaging app with conversation window showing measage sent to "Evelyn. * The screenshot, however, shows the messaging app's contact List, but no message has been sent. Status: failure
In this prompt, the evaluator compares the expected outcome of the task with the actual screenshot. By analyzing the visual content and reasoning about the task, the LLM determines task completion.
A.2.2 REWARD PENALTY
To capture long-term dependencies in the device control setting, Monte-Cario (MC) rollouts were employed to compute cumulative returns, which are then propagated backward to inform updates across each transition. However, during the experiments, we observed frequent repeated nonsense actions even in successful trajectories when doing roll-out with AutoUI agent, which sometimes causes unstable convergence during the asynchronous online learning. Thus, we further deployed a reward penalty unexpected behaviors: accumulative penalty on repetitions and hard penalty on invalid actions.
A.3 SYSTEM DESIGN
A.3.1 DETAILED SYSTEM DESCRIPTION
DISTRL is distributed reinforcement learning framework designed for scalable, efficient training of mobile agents. It decouples trajectory collection from policy learning. utilizing working ma- chines for event-environment interactions and GPU servers for policy training. The working devices handle inference and CPU-intense data collection, transmitting data asynchronously to GPU servers for training large language models (LLMss. This separation optimizes efficiency, scalability, and resource utilization by aligning tasks with appropriate hardware.
This decoupled design offers several key advantages. First, enhances efficiency by preventing re- source contention: mobile devices focus on interaction tasks without being slowed by training, and GPUs dedicated to training perform updates without interruption. Notably, different types of GPUs used; lightweight GPUs CPUs handle inference and data collection, while nigh-performance GPUs are employed for intensive training computations. Second, scalability improves as more mo- bile devices added, allowing data collection scale naturally without single-machine hardware limitations. Third, resource utilization is optimized by aligning tasks with suitable hardware, maxi- mizing performance. Dedicated training resources achieve faster convergence by efficiently process- ing larger batches and complex models. Cost efficiency is enhanced by leveraging existing devices for data collection and appropriately allocating GPU resources based on task requirements, reducing unnecessary hardware investments. Finally, the quality of learned policies improves due to the richer and more diverse dataset collected from multiple devices, enhancing robustness and generalization capabilities.
A.3.2 COMMUNICATION BETWEEN HOST LEARNER AND WORKERS
In our DISTRL framework, communication between the Host Learner and Workers is crucial for synchronizing policy updates and collecting trajectories. We have opted to use SCP (Secure Copy Protocol) over SSH to transfer LoRA weights between the Host Leamer and Workers. This choice is based on several practical considerations related to bandwidth, overhead, and deployment flexibility.
Within the AWS environment, network bandwidth between instances can exceed 500 Mbps. As illustrated in Table transferring the LoRA weights (approximately 100 MB) using SCP takes less than seconds on average. This communication overhead is negligible compared to the time required for trajectory collection and policy training.
18

---

Published as a conference paper at ICLR 2025
Table Communication Time for Transferring LoRA Weights via SCP
Network Bandwidth
LoRA Weight Size
Transfer Time
500 Mbps
100 MB
1.6 seconds
Gbps
100 MB
0.8 seconds
Each Worker thread completes approximately 6-10 trajectories per minute. Concurrently, training the policy on the Host Learner, even utilizing 4 V100 GPUs, takes around 120 seconds to perform single model update. Consequently, the communication time of few seconds for transferring weights significantly lower than both the data collection and training durations, introducing no noticeable bottlenecks.
Alternative high-pens performerrmeee communication options like InfiniBand (IB) or RDMA over Con- verged Ethernet (RoCE) were considered. However, these technologies require specialized hardware and configurations, which are not always available or pracical-especiallle when Workers (mobile devices or emulators) are dispersed across different physical locations or data centers. SCP over SSH offers a flexible and widely supported solution that operates effectively across diverse environments.
In summary, the minimal communication overhead introduced by using SCP over SSH does not adversely affect the overall performance of the DIsTRL framework. The simplicity, reliability, and broad compatibility of this approach make it a reasonable and efficient choice for our distributed reinforcement learning system.
A.4 METHODOLOGY DETAILS
A.4.1 LIMITATIONS TRADITIONAL METHODS
On-policy algorithms such as Proximal Policy Optimization (PPO (Schulman et al. (017)) and Ad- vantage Actor-Critic (A2C (Mnih) 2016)) require synchronous data collection and updates, leading to inefficiencies in distributed and large-scale environments due to low sample efficiency and syn- chronization delays.
Standard off-policy methods like V-trace have been popular in distributed RL frameworks (c.g., IMPALA (Espeholt et al. 2018)) but be suboptimal when the divergence between the behavior policy μ and the target policy æ is either too small or too large due to clipping mechanisms.
These weights were empirically validated to provide robust trade-off between bias and variance, enhancing the overall learning efficiency and stability of the reinforcement learning agent our distributed, asynchronous setting.
A.4.2 IMPLEMENTATION DETAILS
Circular Replay Buffer We utilize Circular Replay Buffer with fixed capacity N store experience tuples (St, Tr, S1+1. 04+1). When the buffer is full, new experiences overwrite the oldest ones, ensuring that the buffer contains the most recent experiences, which is effective in non-stationary environments.
The buffer index is updated as:
i (i+ mod N.
(2)
Enhanced Retrace Algorithm Retrace(A) adjusts the importance sampling corrections based on policy divergence. When policies are similar, it fully exploits trajectories through X-returns. When they differ significantly, it truncates importance sampling ratios control variance, ensuring stable and unbiased updates.
Temporal-Differreee Difference Error Calculation culated as:
The temporal-difference (TD) error for each step is cal-
6, = (8 - V(s)),
which represents the discrepancy between predicted and actual rewards, guiding the learning up- dates.
19

---

Published as a conference paper at ICLR 2025
Priority-Based Sampling in DPER In Distributed Prioritized Experience Replay (DPER), tra- jectories are sampled based on their computed priority to focus on the most informative experiences. The probability of sampling a trajectory is proportional to its priority p(T), caleulated as:
p(T)
P(T) = E()(T,)
where a 0.5 controls the extent to which prioritization is applied. A value a = 0.5 provides a balance between uniform sampling (when 0) and full prioritization (when 1). allowing the model to benefit from both the prioritization of informative trajectories and a degree of randomness. This balance ensures that less prioritized but potentially useful experiences still have an opportunity to be replayed, helping to prevent overfitting to a narrow subset of the replay buffer.
DPER improves sample efficiency by prioritizing high-value transitions, such those with signif- icant temporal-difference (TD) errors or high importance sampling ratios. By focusing on these informative transitions, the agent learns more effectively from previously explored states, reducing unnecessary exploration of less relevant areas. Additionally, by maintaining policy entropy, DPER balances exploration and exploitation, accelerating convergence with fewer samples.
Policy Update Mechanism The actor (policy network) is updated using gradients derived from the advantage estimates:
Vol -E, Apt, a,) Vo log no (ur(se)) By [log (a,(aa))))) XE (Pin)ana(Oe)): (4)
where 0 represents the parameters of the policy network, Pt corrects for off-policy data, (T))) en- courages exploration, and (anvalia(a)) penalizes invalid actions. In this formula, the invalid action penalty term Punaaldd penalizes actions that the agent cannot execute, addressing the challenge of en- tropy regularization, which encourages exploration but can lead to invalid actions. In mobile device control tasks, valid actions such as clicking and typing, while invalid commands, such as "rotate screen" in unsupported context or interacting with non-existent UI elements, waste resources and hinder learning.
We use Gemini-1.5-Pre to evaluate each action a₂ If action is invalid, Pinvalid = 1; otherwise, is O. The penalty term is integrated into the loss as Lpenalty = A E(Puvatidl, where À controls its impact. This approach ensures exploration remains within valid bounds, enhancing learning efficiency and robustness.
How is (ss, at) calculated? advantage estimation approach, as follows:
The advantage function A(st. at) is calculated using one-step
A(st, a.) r(st, 0z) +VV((++1)) Vsss)
Here, r(st, at) comprises two components: the Monte Carlo return from timestep + to the terminal state sy, which is determined by success failure signals, and hard-coded penalties for repeated actions and certain violations, serving as immediate reward signals. This combination provides long-term cumulative reward estimate. V(st) represents the estimated state value at timestep (($+1) is the estimated state value the next timestep t 1, and y is the discount factor.
The trajectory value estimation function Viraj The trajectory value function Vinj estimates the expected return from terminal state &и in sparse/delayed reward environments. It serves as a base- line by labeling reward signals collected trajectories (analogous RLHF reward models) and filters/riiorttiess trajectories for training. Through this filtering mechanism, Vinj maintains high- quality training data, leading to more effective policy learning.
A.4.3 HYPERPARAMETER TUNING FOR DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY
To enhance sample efficiency in our Distributed Prioritized Experience Replay (DPER) frame- work, it is crucial appropriately balance the contributions of the average cemporal-difference (TD) error (ő)), the average importance sampling ratio (p), and the average policy entropy (H). These components inherently operate on different scales, necessitating careful normalization to ensure that single component disproportionately influences the priority calculation.
20

---

Published as a conference paper at ICLR 2025
Each component contributing to the priority score is normalized to common scale based on their statistical properties observed during preliminary training runs. The normalization process is as follows:
Average Absolute TD Error (ōō): Normalized by dividing by the maximum absolute TD error observed across all trajectories in the training set. This scaling ensures that [o] ranges between 0 and 1.
Average Importance Sampling Ratio (p): As importance sampling ratios naturally fall within the range [0, I], no additional scaling is required.
Average Policy Entropy (H): Normalized by dividing by the maximum observed entropy value during training, ensuring that H also ranges between 0 and 1.
This normalization facilitates balanced contribution from each component when computing the overall priority, preventing any single factor from dominating the priority score.
The weights Wi. W2, and wy are critical in determining component on the priority calculation. To identify the optimal values for these weights, we employed a grid search strategy on a validation set, exploring the following empirical ranges: W1 E {0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0), W2 {0.01, 0.10.3, 0.5, 0.7, 1.0), W3 (0.01, 0.3, 0.5, 0.7, 1.0).
influence of each normalized
These ranges were selected based on insights from prior research (Schaul 2015 Horgan et al. 2018) and preliminary experiments that indicated effective performance within these intervals.
The chosen weights effectively balance the three components, ensuring that:
Learning from High-TD Error Trajectories: By assigning a higher weight to [0], the frame- work emphasizes replaying experiences where the model's predictions were significantly off, facilitating targeted learning and faster convergence.
Maintaining Exploration: The weight on policy entropy ensures that the agent continues to explore diverse actions, preventing premature convergence suboptimal policies.
Correcting for Distributional Shifts: The importance sampling ratio weight allows the algo- rithm to adjust for changes in the policy distribution, maintaining unbiased updates despite using prioritized replay.
A.5 EXPERIMENTAL DETAILS
A.5.1 BASELINE METHODS
We evaluate proprictary vision-language models (VLMs), GPT-4V (OpenAl et al., 2024) and Gem- ini 1.5 Pro/Reid et al. 2024), using the AppAgent framework. By applying the prompt from (Yang et al. 2023b), we enable these models to interact effectively with the environment. We assessed the AppAgent (Yang al. 2023b in augmented prompting setting, where the agent explores the environment and gathers experience ahead of the inference phase. This collected experience appended the test-time prompt, enhancing the model's decision-making capabilities. Unlike learning-baa based approaches, these methods rely on advanced prompting strategies to accomplish tasks without extensive training.
In our system's implementation. we maintained a consistent and conflict-free approach by exclu- sively utilizing the T5-based Multimodal Large Language Model (MLLM) architecture (not vanilla pretrained T5, T5-Base normally has 220M, Our Agent (AutoUU-driven)) has 1.3B). This encoder- decoder framework facilitates the seamless integration of various agents, enhancing the model's versatility and performance, which is aligned with AutoUI (Zhan & Zhang 2023) design.
Our T5-based MLLM architecture is inherently designed to support diverse decoder initializations. including the integration of pretrained decoder models. While previous works like AutoUI (Zhan) & Zhang 2023) have utilized their own model weights, we could not directly use AutoUT's model models because they were trained with ArTW knowledge. Incorporating these weights could introduce unintended biases specific to their tasks as well as prior knowledge to training and testing data set, which might reduce the credibility and confidence of system.
weights to initialize
21

---

Published as a conference paper at ICLR 2025
Meanwhile, our empirical analysis showed that decoder layers require more diverse initialization patterns than encoder layers to achieve optimal performance in Ul-specific tasks, GPT-2 help more than the pre-trained T5 weights. Therefore, the best approach was manually select pretrained decoder model to initialize the weights of our T5-based decoder. We chose to initialize the decoder with GPT-2 weights, this allowed us to leverage GPT-2's pre-trained language generation capa- bilities, providing a foundational understanding of language that we could further refine through fine-tuning tailored to our specific tasks.
We did not specify the decoder type/weights in the original explanation because we found that fine-tuning efficiency was already satisfactory with GPT-2. We acknowledge that there might be better options available; however, we adopted this approach simply by drawing inspiration from our basclinn DigiRL (Bai et al.] 2024), which also used GPT-2 to initialize their decoders (This is a trick here). By following structured and methodical approach from existing baselines, we ensured architectural compatibility and effective integration of the GPT-2 weights into our T5-based decoder.
In Table we compare DISTRL with other frameworks based on scalability, task diversity, and training efficiency.
Table 3: Comparison among on-device agents' frameworks based on scalability, task diversity, and training efficiency.
DISTRL (Ours)
DigiRL
AutoUI
AppAgent-MLLMs
Type Scalability Task Diversity Training Eff. Multi-GPUs Sup.
Async. ++
Sync.
N/A
N/A
N/A
N/A
General High
Limited
Limited
General N/A
Low
Low
x (offline only)
X
X
A.5.2 TRAINING AND TEST DATA
The dataset used in this work based on the Android in the Wild (AitW) (Rawles al. 2024b and Android (Rawles et al.] 2024a) task set, with enhancements for practical use in fine-tuning agents to control mobile devices and interact with real-world applications. We trained two separate models using two distinct subsets for General Tasks and Web Shopping Tasks. Each model was trained on its corresponding training subset and evaluated on the respective test subset drawn from AitW.
For training, we utilized the General Tasks subset from AitW, augmented with tasks selected from AndroidWorld and several expert curated ones, which includes tasks that require basic to complex application usage and information retrieval. For Web Shopping Tasks, we used the subset from W directly. Each training set consists of more than 400 tasks, allowing the agent learn from broad range of apps and websites operations and promote robust learning without overfitting to specific task types.
To avoid cold start issues in asynchronous reinforcement learning framework, we constructed a warmup trajectory dataset for each task type, each consists of 128 trajectories collected with an initial version of the AutoUT agent. These sets will be fed into the replay buffer at the beginning of the online training.
During testing, we evaluated the models on their respective test sets: 128 tasks for General Tasks and 128 tasks for Web Shopping Tasks, both sourced from AitW. This approach ensures that each model is assessed on the task domain it was trained on, addressing the task distribution gap between general user instructions and domain-specifie web shopping instructions.
General Tasks The General Tasks subset consists of tasks that involve basic application operations and information retrieval. Examples include searching for the latest news, retrieving information about locations, and interacting with mobile apps. To force the agent to operate more on the various applications instead of searching everything through web, we augmented the task set for training with several instructions from AndroidWorld and some expert curated tasks, which are typically more complex and 128 tasks from AitW, facilitating a robust evaluation of General Tasks performance. Each task
The training set contains 600 tasks, and the test set includes
22

---

Published as a conference paper at ICLR 2025
allows maximum of 15 steps to complete. Example tasks from the General Tasks subset shown in Table
Task Set
Task Example
What is the capital of Norway?
AitW
Play some music on YouTube. Run the stopwatch. Create a new contact for Jack. Their number is 0123456789.
AndroidWorld
Check today's 's events the calendar.
Expert Curated
Check if there is any app update in Playstore.
Table 4: Examples of task descriptions in the General Tasks subset.
Web Shopping Tasks The Web Shopping Tasks subset includes tasks that simulate real-world shopping activities such as searching for products, navigating e-commerce websites, and interacting with shopping carts. Task complexity ranges from simple web navigation involving product searching and browsing. The training set consists of 500 tasks, and the test set includes 128 tasks from AitW, enabling the evaluation of the agent's ability handle domain- specific instructions. Each task permits up to 20 steps to complete. Example tasks from the Web Shopping Tasks subset are presented in Table
multi-step operations
Difficulty
Task Example
1
Go to ebay.com
1
Go costco.com
2
Go to ebay.com, search for "asus zenbook"
2
Go walmart.com, search for "corsair k70"
3
Go to bestbuy.com, search for "dell xps", and select the first entry
3
Go to newegg.com, search for "bose soundlink mini", and select the first entry
Table 5: Examples of task descriptions in the Web Shopping Tasks subset.
A.5.3 DETAILED PERFORMANCE COMPARISON
Other methods struggle to achieve comparable performance. DigiRL, both in single and multi- machine settings, suffers from inefficiencies in data collection and utilization. The multi-machine version requires extensive collection time due to its low efficiency, hindering its ability to train effectively on diverse tasks, while the single-machine version struggles with scalability issues. These inefficiencies lead higher variance in performance, as evidenced by the higher standard deviations (up to #3.1 13) compared to DISTRL.
Overall, the low variance and high success rates of DISTRL demonstrate its robustness and effec- tiveness in generalizing across different tasks, emphasizing advantages of our distributed rein- forcement learning approach over existing methods, especially in large-scale, asynchronous settings.
A.6 ADDITIONAL QUANTITATIVE EXPERIMENTS
A.6.1 FAILURE MODES ANALYSIS
presents a comparative analysis of failure rates across different approaches the AitW General and Web Shopping subsets. Among the evaluated frameworks, DisTRL consistently ex- hibits the lowest failure rates across all failure categories, notably excelling in recovering from mistakes and achieving the correct goal.
Figure 11
For the General subset, DISTRL. demonstrates exceptional performance with failure rates as low as 4% in recovering from mistakes, 12% in getting stuck midway, and 2% in arriving at an incorrect goal. These rates are at least three times lower than those observed alternative approaches such as AutoUI and DigiRL. This significant reduction in failure rates can be attributed to DISTRL's robust asynchronous distributed reinforcement learning (RL) framework, which facilitates more nuanced and adaptive policy learning. The distributed nature of DISTRL allows for parallel exploration and exploitation of the state-aco action space, leading to a more comprehensive understanding of task dynamics and improved decision-male makinn accuracy.
23

---

Published as a conference paper at ICLR 2025
% Wiin All Trajectories
General
Apphgent
H
AppAgent Gemin
Fall recover from mistakes
Get stuck midway Failure Mode
Artive al wrong goal
AutoUI
% in All Trajectories
Web Shopping
DigiRL
(singla,oniine)
20
DigiRL (multi-mach.)
DistRL (Ours)
Fall recover from mistakes
Get stuck midway Failure Mode
Artive at wrong goal
Figure 11: Comparison of failure modes across different frameworks the AitW General and Web Shopping subsets.
Similarly, the Web Shopping subset, DISTRL maintains low failure rates of 3% in recovering from mistakes, 7% in encountering mid-task obstacles, and 4% goal misalignment. These figures represent least twofold improvement over competing frameworks, highlighting DISTRL's perior capability in managing complex and dynamic task environments. The ability to effectively handle task complexities is further reinforced by the asynchronous updates DISTRL, which miti- gate issues such as delayed feedback and non-stationary environments that often plague distributed learning systems.
In contrast, frameworks like AutoUI and DigiRL exhibit higher failure rates, which may stem from their less sophisticated policy learning mechanisms limited scalability in distributed settings. These higher failure rates suggest that these approaches may struggle with tasks that involve intricate dependencies or require rapid adaptation to changing conditions. The limitations observed in these frameworks underscore the importance of advanced distributed learning architectures in developing resilient and efficient agents capable of navigating complex, real-world environments.
Overall, the superior performance of DISTRL. across multiple failure modes underscores its effec- tiveness in building robust agents. This robustness is crucial for applications where reliability and precision paramount, such automated web interactions and general task execution. Future work may explore further enhancements to the distributed framework, such as incorporating more sophisticated exploration strategies or leveraging transfer learning extend capabilities to even more diverse task domains.
A.6.2 GENERALIZATION PERFORMANCE ON AITW SUBSETS
Generalization Performance to Unseen Tasks
Apphgent
+ GPT-4v
% Success Rate
AppAgant
Gemini
AutcUl
(single online)
DigiRL
Web Shopping
Instal
Google Apps
(muli-mach.)
Task
DiaRR (Ours)
Figure 12: Generalization performance across different AitW subsets. The agents trained on the General task set and evaluated on 128 tasks per subset.
Figure 12 illustrates the generalization performance of various frameworks across the Web Shopping, Install, and Google Apps subsets of the AitW dataset. The agents were trained exclusively on the General task set, and their ability to generalize was assessed on the first 128 tasks within each respective subset.
24

---

Published as a conference paper at ICLR 2025
DISTRL consistently outperforms its counterparts, achieving accuracies of 68.5% on Web Shopping, 73.5% on Install, and 70.2% on Google Apps. These results highlight DISTRL's ' superior general- ization capabilities, which can be largely attributed to its robust distributed learning approach. The asynchronous distributed RI. framework employed by DISTRL enables the agent to learn from a diverse set of experiences concurrently, fostering more versatile and adaptable policy that can transfer effectively across different task domains.
In contrast, frameworks such as DigiRL and AppAgent exhibit markedly lower generalization per- formance. DigiRL and AppAgent struggle particularly with adapting to Install and Google Apps subsets, where task structures and requirements may differ significantly from the training set. This limitation suggests that these frameworks may be overfitting to the General task set or lack- ing the necessary mechanisms capture the underlying transferable features essential for effective generalization.
The ability of DISTRL to generalize across diverse task subsets is a critical advantage, especially real-world applications where agents are often required operate varied and unforeseen en- vironments. This generalization strength is likely result of the extensive exploration and varied experiences facilitated by the distributed learning process, which allows DISTRL to build a more comprehensive and flexible policy.
These findings have significant implications for the development of autonomous agents. The demon- strated generalization capabilities of DISTRL suggest that distributed RL frameworks can be promising direction for creating agents that are not only proficient in specific tasks but also adaptable to wide range of scenarios without the need for extensive retraining. Future research could inves- tigate the integration of additional generalization techniques, such as meta-learning or multi-tad task learning, with distributed RL to further enhance performance across even more diverse and complex task domains.
A.6.3 DISTRL AGENT PERFORMANCE ON TEST SET
Our primary contribution is improved training performance, which evaluated using strictly sep- arated training and testing sets. While our main results are reported in Section 66.3 using the final trained agent, also tracked performance progression test during training. Figure the averaged results (3 trials) of periodic checkpoint evaluations on the "General" test set from AiTW, demonstrating continuous improvement throughout training.
Walltime
Success Rate on
75
74.2%
73,4%
73.2%
71.8%
Success Flats (%) on Test Sel
68.8
65.5%
65
65
55
50
60
80
Walltime (mins)
Figure 13: Continuous Evaluation the Test Sets during DistRL. fine-tuning process
25

---

Published as a conference paper at ICLR 2025
A.7 GENERALIZATIONS TO OTHER DOMAINS
Expanding to a Broader Range of Mobile Applications We conducted additional experiments on "GoogleApps" and "Install" tasks from AiTW datasets, included in the training set, assess generalization. Results in Appendix A.6.2 demonstrate that DistRL maintains high performance and effectively generalizes to new tasks. However, achieving high generalization remains challenging when facing substantial variations across different apps. We are also exploring more open-world benchmarks, and as future work, we plan to expand our exploration wider range of mobile applications.
Handling Significant UI Changes or App Updates DistLL's design accommodates different de- vice types, OS versions, UI changes, and app updates by using ADB and emulators for human-like interactions and system resets. The main challenge is the compatibility of AVD and app versions. For major UI overhauls or new devices, additional training or fine-tuning may be required. However, DistRL's architecture supports scalable data collection and training across diverse environments, en- hancing its robustness in real-world scenarios. We validate this robustness using our dedicated testing infrastructure, including proprietary OS environments and hardware.
Adapting the Framework for Other OS To extend DistRL to iOS, we can adapt our framework by replacing Apdroid-specific tools with Apple's counterparts (e.g., XCUITest, simetl) while main- taining our platform-agnostic core components. Our validation was facilitated by robust hardware and OS support that enabled efficient API resets and testing modules. The key challenge mi- grating such RL fine-tuning platforms lies in the ability to reset states for trial-and-error learning. While our framework can support iOS environments with necessary adjustments, Android remains our primary platform due to its developer- ecosystem.
26


Reviews:
[{"review_id": "EYPz789plQ", "reviewer": "Reviewer_vpis", "summary": "The paper presents DistRL, an asynchronous distributed reinforcement learning framework designed explicitly for on-device control agents in mobile environments. DistRL aims to optimize training efficiency and agent performance for complex tasks, leveraging centralized training with decentralized data acquisition. By introducing a unique off-policy algorithm, A-RIDE, DistRL effectively manages asynchronous data collection and prioritizes valuable interactions, addressing challenges like delayed updates and non-stationary data. Experiments demonstrate its significant performance and efficiency improvements over existing frameworks, especially for real-world mobile tasks, positioning DistRL as a scalable solution in distributed reinforcement learning for mobile devices.", "questions": "- Any results from INSTALL and GOOGLEAPPS of AITW? / Why not compare results in AndroidWorld tasks?\n- In real-world cases, many users (workers) have different types of devices and different OS versions. How does DistRL handle scenarios where the mobile device encounters significant UI changes or unexpected app updates during training?\n- What are the trade-offs between computational overhead and empirical performance (success rate) when using more environments (workers) and Distributed Prioritized Experience Replay (DPER)?\n- Could the framework be adapted for other operating systems, such as iOS, and if so, what modifications would be necessary?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- DistRL employs a decoupled architecture that enhances scalability and robustness by distributing data acquisition across heterogeneous mobile devices, making it efficient for large-scale deployment. Moreover, they propose an A-RIDE algorithm adapted for asynchronous environments, balancing exploration and stability, which improves sample efficiency and task success rates.\n- The paper provides comprehensive experiment results that significantly outperform baselines, showcasing superior success rates in complex tasks. Ablation studies highlight the importance of its components, like DPER and retrace correction.\n- The paper is well-written and easy to follow.", "weaknesses": "- Lots of hyperparameters are required. (e.g., Equation 1, 2, DPER). Do you have any tips or evidence for determining them, and are the values sensitive to different types of environments?\n- Although the experiments cover general tasks, expanding the testing to a broader range of mobile applications would offer more substantial evidence of DistRL’s adaptability. (from additional tasks from AITW and AndroidWorld)\n- Is VLM Evaluator (Gemini)'s performance in providing reward signals credible? Did you encounter any failure cases where VLM misleadingly provides success/failure signals? AndroidWorld provides manually defined dense rewards. Have you tried training DistRL with such a reward?\n- While the asynchronous design improves adaptability, more detail on DistRL's response to drastic changes in mobile environments or new, unseen tasks would strengthen the discussion of its robustness in dynamic settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jXSIjDiYOS", "reviewer": "Reviewer_fxPf", "summary": "This paper introduces DistRL, an asynchronous distributed reinforcement learning (RL) framework designed to fine-tune on-device control agents, particularly for mobile environments. The key contributions include a scalable architecture that decouples data acquisition from policy learning, improving training efficiency and robustness in dynamic environments. The paper claims significant improvements in training speed and success rates compared to state-of-the-art methods, validated through experiments on Android tasks.", "questions": "See weakness", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The decoupling of data acquisition from policy learning allows the framework to handle heterogeneous devices efficiently, which is crucial for real-world applications.\n2. The asynchronous nature of DistRL enables faster data collection and policy updates compared to synchronous methods, which is demonstrated by a claimed 3x improvement in training speed.\n3. The focus on mobile device control agents addresses a practical problem with significant industry relevance, particularly in automating complex user interactions on smartphones.", "weaknesses": "1.  There is a significant discrepancy between the model descriptions in different sections (T5-based multimodal model mentioned in Sec 1 vs GPT-2/ROBERTA in A.5.1). This inconsistency undermines the paper’s credibility and makes it difficult to understand what model was actually used. If different models were used for different tasks or components (e.g., policy learning vs critic evaluation), the authors should explain why and provide a rationale for these choices. If there were intentional changes in model usage (e.g., using GPT-2 for policy learning and ROBERTA for critic evaluation), the authors should explain why these variations were necessary and how they impacted performance.\n\n2. It is unclear how the proposed method differs from IMPALA from the methodology perspective. I understand that applying them to a new on-device setting is trivial, but it is confusing to me what are the major differences from the algorithm side. It seems that A-RIDE builds upon existing methods like Retrace and GAE but does not introduce substantial innovations beyond adapting these techniques to a distributed setting. Additionally, the author mentioned in Sec. 2.3 that:\n> Furthermore, we develop a novel reinforcement learning algorithm specifically designed to handle the stochasticity and scalability inherent in mobile device environments.\nHowever, I did not see any specific techniques and algorithm improvements that address the stochasticity and scalability issues.\nIf there are indeed major algorithm changes, it would be beneficial to add a baseline of IMPALA and IMPACT.\n\n3.  Additionally, the training and testing set used are not clearly described. I am a bit confused about the difference between the training and testing sets -- are there overlaps of tasks between them? How is the expert-curated and AndroidWorld training set selected? This questions the generalization issues of the experiments.\n\n4. The evaluation metrics and result plots are a bit confusing. It is unclear whether the plots in Fig. 6 are for the training task success rate or the testing task's. Additionally, adding the testing task success rate plots instead of just training one in Fig. 5 would be beneficial as well.\n\n5. The x-axis in Fig. 5 (a) is the time. From Fig. 5 (c) we can see there are significantly more trajectories collected than baselines. I acknowledge that the implementations of the proposed method is much more efficient than baselines, but I am also curious about what the plot looks like if the x-axis is the trajectory-num or the step-num, which is commonly reported in the RL literature. It would be super helpful to understand the upper bounds of the baseline algorithms given the same number of collected data. It is also interesting to know the synchronized algorithm performance trend.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XbmzonAj5R", "reviewer": "Reviewer_w8MY", "summary": "This paper provides a way to train device control agents using reinforcement learning. Specifically, it proposes a setup of asynchronous data collection using parallelized data collection on emulators (or real devices), combined with a centralized learning algorithm. The algorithm for training is V-trace, combined with prioritized replay buffer and an additional component encouraging exploration. The system runs on multiple workers and uses a single host learner.\n\nThe paper evaluates results on AitW using model-based evaluation. Compared to existing literature/SoTA the DistRL approach achieves a high success rate in less wall time by applying a distributed, asynchronous training approach, which results in faster emulation speed (traj/min). In the General test set, DistRL achieves a success rate a relative improvement of over 19% vs previous work (DigiRL) in the Web Shopping test set, DistRL attains a success rate  outperforming DigiRL by over 14%. The work conducts ablation analysis.", "questions": "“The evaluator receives the current observation, composed of the task description and a screenshot of the device, and outputs a reward signal.” – Have the authors validated that this does a good enough job of success validation, versus, say using the entire trajectory?\n\nTo me, “On-device End-to-End Agent Performance Evaluation” implies that the inference is done on the device, but this appears not to be the case. I understand it could be done in theory because the model is small enough, but it’s not actually on device as far as I can tell. Is there another way to describe it?\n\n## Comments\n(You don’t have to address these; they are intended to be helpful only.)\n\nFor future work, you may consider SeedRL to further scale the approach. https://arxiv.org/pdf/1910.06591#page=5.27\n\nFuture work could evaluate on online benchmarks such as AndroidWorld to ensure that results are easily comparable for future researchers, since model-based evaluation is harder to reproduce due to costs and because of differences in model outputs, even for the same base model.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The work achieves impressive results on a challenging task. It provides a path to scale up data collection and enable future work to tackle even more complex problems. Implementing this distributed setup and making it work is a valuable contribution. This work is particularly relevant to future researchers where researchers can easily spin up many workers on cloud platforms. \n\nThe paper is well-written, with technical depth and there are enough details describing the methodology and approach. The paper does a good job of comparing their results to existing works and agents.\n\nIt is good the authors run multiple experiments per and average the results to provide more stable results. It is also good that the work includes an ablation study.", "weaknesses": "In order for this work to be a major contribution, I feel it should be open-sourced since a core contribution is the engineering work around centralized training and decentralized data acquisition, which requires complex communication between different systems. Open-sourcing will greatly help the community to take advantage of the contributions here and with reproducibility.\n\nThe core RL techniques used are relatively standard. There's no fundamental advance in machine learning theory or methodology. However combining IMPALA with prioritized experience replay seems to be novel, and it’s implementation is highly non-trivial.\n\nIf both DigiRL (single) and DigiRL (multi) have the same amount of emulators, why are the performance numbers between them different? Ultimately, both consist of 32 emulators, if I understand correctly, operating with synchronous training. Please clarify how they are different and why it's relevant to include both. I also think the number of emulators should be displayed more prominently as a key experimental parameter (rather than only in Figure 5)\n\nSpelling/Writing: \"ensuring that only contextually appropriate actions are penalized\". I believe this should be “inappropriate actions are penalized”?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BqyFZTwh0L", "reviewer": "Reviewer_oSkG", "summary": "This paper focuses on the domain of on-device control agents and presents DistRL, an asynchronous distributed reinforcement learning framework for device control agents. This paper designs a centralized training and decentralized data acquisition system design to ensure the efficiency of online RL fine-tuning given potentially heterogenous hardware specifications. It also proposes a tailor-made RL algorithm in the context of distributed device control training. Both the system design and new RL algorithm shows improvements over the prior state-of-the-art DigiRL in a popular device-control benchmark AitW.", "questions": "Line 53-70 seems repetitive.\n\nLine 71 fine-tuning(type of light training) -> fine-tuning\n\nParagrapho one and two of intro seem to be a bit verbose in introducing the background.\n\nLine 78-85, it seems unnecessary to introduce the notion of RLHF here cuz this paper does not have human preferences.\n\nLine 86-90 is confusing, it seems to describe challenges of exisiting offline datasets such as AitW and Android-Control, but why is it relevant for online RL fine-tuning?\n\nLine 98-104 can be best put in the related works.\n\nThere are too many bolded text in the introduction that can tire the readers. I would suggest removing the bold font for related methods. Same applies to the experiment section.\n\nLine 121-122, te claim of \"DistRL is the first work to scale autonomous, online RL fine-tuning for mobile device control in a distributed environment.\" may be incorrect, DigiRL also supports distributed data collection. I understand that DistRL supports more distributed RL optimizations, but it might be good to update this sentence.\n\nLine 247-257, would be great to defer algorithm-related designs to the next section as this section mainly talks about system design.\n\nThere can be more details for the method. In particular, do you maintain a separate model for $Q$ and $\\rho$? If so, how are those models trained? How is $\\Pcal_{invalid}$ calculated? How is $A(s_t, a_t)$ calculated? More motivations can also be included. For example, why does DPER improve sample efficiency? 5.3 includes a training loss for the trajectory value function $V_{traj}$, where is it used?\n\nFigure 5 (b) needs more explanations, what does \"Proportion Above 60%\" mean?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper addresses a realistic concern of asynchronomous data collection in applying RL fine-tuning for device-control agents, so it is highly relevant. \n\nThe proposed system-level optimizations can be valuable to the open-source community. I hope the authors can clean up and release the infrastructure for their experiments.\n\nThe proposed RL algorithm and system design seems to make a difference on popular AitW benchmark.", "weaknesses": "The writing of the paper, in particular the introduction and methodology, can be improved. In fact, the current state of writing has caused me trouble for understanding the contributions of the paper. A significant amount of re-writing is required before the paper can be published in a top-tier venue like ICLR. Please see my questions for more details.\n\nThe organization of the section 5 methodology makes it hard to understand what parts are novel in the paper and what parts are adapted from existing literature. From the current writing, it seems to be a combination of existing components from the prior literature. Additional motivations are needed to explain what is novel in the tailor-made RL algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)