Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control
Authors: Georgios Papoudakis
Thomas Coste
Jianye HAO
Jun Wang
Kun Shao
Venue: NeurIPS 2025 poster ()
Keywords: App Control
Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Reinforcement learning (RL) using foundation models for policy approximations in multi-turn tasks remains challenging. We identify two main limitations related to sparse reward settings and policy gradient updates, based on which we formulate a key insight: updates from positive samples with high returns typically do not require policy regularisation, whereas updates from negative samples, reflecting undesirable behaviour, can harm model performance. This paper introduces Succeed or Learn Slowly (SoLS), a novel off-policy RL algorithm evaluated on mobile app control tasks. SoLS improves sample efficiency when fine-tuning foundation models for user interface navigation via a modified off-policy actor-critic approach, applying direct policy updates for positive samples and conservative, regularised updates for negative ones to prevent model degradation. We augment SoLS with Successful Transition Replay (STR), which prioritises learning from successful interactions, further improving sample efficiency. We evaluate SoLS on the AndroidWorld benchmark, where it significantly outperforms existing methods (at least 17\% relative increase), including prompt-engineering and RL approaches, while requiring substantially fewer computational resources than GPT-4o-based methods with 5-60x faster inference.

Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control
Georgios Papoudakis" Thomas Coste
Jianye Hao
Jun Wang
Kun Shao
Huawei Noah's Ark Lab, University College London
Abstract
Reinforcement learning (RL) using foundation models for policy approximations in multi-turn tasks remains challenging. We identify two main limitations related sparse reward settings and policy gradient updates, based which we formulate a key insight: updates from positive samples with high returns typically do not require policy regularisation, whereas updates from negative samples, reflecting undesirable behaviour, can harm model performance. This paper introduces Suc- ceed or Learn Slowly (SoLS), a novel off off-policy RL algorithm evaluated on mobile app control tasks. SoLS improves sample efficiency when fine-tuning foundation models for user interface navigation via modified off-policy actor-critic approach, applying direct policy updates for positive samples and conservative, regularised updates for negative ones to prevent model degradation. We augment SoL.S with Successful Transition Replay (STR), which prioritises learning from successful interactions, further improving sample efficiency. We evaluate SoLS on the An- drrldWorld benchmark, where it significantly outperforms existing methods (at least 17% relative increase), including prompt-engineering and RI. approaches, while requiring substantially fewer computational resources than GPT-4o-based methods with 5-60x faster inference.
Introduction
Mobile phones have become a central part of daily life, supporting communication, productivity, financial management, and entertainment, As mobile apps become more complex, there is growing need for automated systems that can understand and interact with mobile interfaces. Such systems could improve accessibility for people with disabilities, enhance automated testing, and act as assistants capable completing multi-step tasks on behalf of users.
Reinforcement learning (RL) offers promising approach to training agents for mobile app interaction [e.g.. 2). allowing learning through trial and error without requiring extensive manual labelling. However, applying RL. to mobile app control presents several unique challenges. First, like many open-ended tasks, these environments often provide sparse or no reward signals. Second, the action space is large and context dependent, requiring the agent to choose to interact from hundreds of possible elements and many different action types. Third, simulations in mobile environments are time and computationally expensive: several seconds to execute each step limits training interactions.
Foundation models have demonstrated remarkable capabilities in understanding text instructions and visual context, making them promising candidates for mobile app control. Recent efforts have shown that prompting LLMMs such as GPT-40 can yield agents capable of following instructions to complete However, these approaches typically require multiple
tasks in mobile environments [14 36 38
* Equal contribution. Correspondence to kun. shao @ huawei.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

API calls per step and extensive in in-context reasoning, resulting in high operational costs and slow inference times. Meanwhile, Supervised Fine-Tuning (SFT) approaches that directly train smaller models on human demonstrations struggle with generalisation to new and online tasks p 21 23 35
In this paper, we introduce Succeed Learn Slowly (SoLS] novel RI. algorithm that enables sample-efficient fine-tuning of foundation models. This work tackles the challenges of sparse rewards and limited samples through two complementary innovations. Our first contribution is dynamic policy update mechanism that aggressively reinforces successful actions while conservatively regularising unsuccessful ones, helping prevent performance degradation and forgetting during exploration. This pairs with our second innovation, Successful Transition Replay (STR), which selectively stores and prioritises successful state-action pairs from previous episodes. Together, these approaches create a sample-effi cumple-efficient learning system that extracts maximum value from limited generated samples, substantially improving the model's ability to learn effective control strategies even with minimal training data.
We evaluate SoLS on AndroidWorid [20). a benchmark of real-world mobile app control tasks across three difficulty levels. Our approach achieves an overall success rate of 51.3%, significantly outperforming both expensive GPT-4o-based approaches and other RL-based fine-tuning methods. Importantly, our method requires only a single pass through an 8B parameter model per step, resulting in inference times approximately 5-60x faster than ofte-of-thhearr prompting approaches.
Our contributions can be summarised as follows:
We introduce SoL.S, a novel off-policy RI algorithm specifically designed for sample-efficient fine-tuning of foundation models in sparse reward settings, which achieves the best performance, by at least 17% relative increase, and the fastest inference time of all baselines.
We propose Successful Transition Replay (STR), a technique that prioritises learning from successful interactions to maximise learning efficiency in environments with costly simulations.
We show that small language models, when fine-tuned with suitable RL techniques, can outperform much larger foundation models on mobile app control tasks, and offer comparative analysis of RL methods, highlighting the strengths and weaknesses of various methods.
This work helps narrow the gap between costly but effective large model prompting approaches and efficient but traditionally less performant fine-tuned models, providing potentially practical direction for mobile app control that balances performance, efficiency, and resource requirements.
2 Preliminaries
2.1 Problem Formulation
We formulate the Mobile App Control problem within Partially Observable Markov Decision Process (POMDP) framework, represented as A, O, R. P. 92. S denotes the state space, A the action space, and O the observation space. The reward function is binary, signalling successful episode completion. Functions P and Ω represent the state and observation transition processes, respectively. Episode length is bounded by a task-specific horizon H. An episode ends either with the completion of the goal or when // is reached without success. We define the return as the terminal reward and aim to learn parameterised policy To that maximises the expected re- turn across the task distribution, with g sampled from the task set G, and r the appisode-termina reward:
max Egeg, [r] &
Our approach leverages an offline dataset Doff containing human demonstrations, which we use to warm-start the policy To and familiarise it with the environment's action and observation space. Additionally, we highlight that D.ff is out-of-distribttion (OOD) relative to the target environment. Our training pipeline follows a two-phase procedure: (1) SFT using Dosy. followed by (2) RL fine-tuning to optimise the objective in Equation (I). Notation used in equations throughout can be
found summarised in Table 2
Name inspired from WoLF

---

2.2 Mobile App Control Data
Our work focuses on two open-source phone navigation datasets and benchmarks, namely Android- Control [20] and Android World [28). AndroidControl provides an extensive training dataset of more than 13k tasks spanning 833 Android applications, including task instructions, screenshots, and UI element trees, as well as human-selected action demonstrations. We use this dataset for initial fine-tuning, leveraging the fact that its action space has strong similarities with Android s. AndroidWorld is benchmark consisting of 116 tasks with a different app and goal distribution compared to AndroidControl. We specifically use an 80-task subset, omitting tasks such as Q&A and verification, due to our agents and action space, as well as evaluation process (see Appendix A.442).. Our final benchmark thus has harder overall difficulty distribution. AndroidWorld works by con- necting agents to Android phone emulator to run tasks online, in a realistic environment with ground- truth rewards. Agents are provided with the task goal, current screenshot, and UI tree infor- mation at every step, and required to provide actions to be executed in the environment. We use AndroidWorld as our RL environment and evaluation benchmark throughout our experiments.
It is important to mention that Worro is Control dataset used for SFT. While the task domains of both overlap. almost all AndroidWorld apps are unseen. Moreover, the phrasing of goals is different, with AndroidControl goals being quite wordy and AndroidWorld goals being more imperative. Finally, though the action spaces are similar, the distribution or use of actions in certain cases can be quite different. For example, the long long-press press action is extremely rare in AndroidControl, appearing only as 0.2% of the training set, while it an important action featured regularly in AndroidWorld, such as to clear text in a text field.
(OOD) compared to the Android-
3 Methodology
3.1 Observation and SET Step
An important design detail for experiments with AndroidWorld tasks is the construction of the observation. In this work, we use Llama-3-8B-Instruct (12) for approximating the policy. As input to the model, we use the textual goal, and process the UI tree into list of available UI elements with descriptions and relevant attributes. Text-only input has shown potential in achieving similar or even better results than visual input (28 and requires many fewer tokens, leading to faster inference times.
As a first step, we fine-tune our base model through SFT on AndroidControl, which shares observation and action space similarities with AndroidWorld More details on observations, action space, and output format can be found in Section 2.2 and Appendix [A.4] Using a similar input format and action space during SFT and RL training, the SFT step allows the model to adapt to action-predicte
in Android environments, with the required output format. The resulting SFT model works as a starting
point for the RL methods, such that some tasks can be solved in the initial phase of training.
3.2 Successful Transition Replay
Generating trajectories in open-world environments presents significant challenges, primarily due to the high computational cost involved. For instance, in Android emulators, each forward step in the simulation can take 4-5 seconds, which makes trajectory generation resource-intensive and inefficient. Moreover, the generated trajectories tend to be long and involve many steps. but only small fraction are successful. As a result, much of the computational effort invested in trajectory generation yields little in terms of meaningful learning outcomes.
In addition, while RL algorithms are commonly used in large-scale post-training of LLMs, most of these algorithms are designed for single-turn tasks, such as solving mathematical problems or question-answering, where reward models provide immediate feedback. However, our setting is more complex. The agent seldom solves new tasks due to significant distribution shift from the original SFT phase, which makes traditional RL approaches less effective in our context.
To address these challenges, we propose Successful Transition Replay (STR), an experience replay 22 that stores individual successful timesteps. Figure S shows how STR integrates into our pipeline. It focuses on preserving only those transitions from episodes where the agent takes successful actions, ensuring that learning incorporates proven effective steps rather than failed or non-meaningful ones. By maintaining pool of these successful timesteps, the agent can revisit and build on past successes,
3

---

SoLS
Successful Transition Replay
Sampled timesteps
Sample is positive negative?
Successful
Training Data
trajectories
A 0
<
Environment
On-policy data
Full update
Conservative update
Tasks
Updated model
Figure 1: Illustration of the SoLS and STR methods.
ultimately leading to more efficient learning rather than relying exclusively on-policy data. STR thus creates a bridge between the SFT and target distributions, by bootstrapping from rare successes to build repository of successful interactions specific to the environment.
STR uses hash table to map each task a list of successful individual timesteps. During training, we sample a specific number of timesteps from each task and combine them with on-policy data. Each task-spec list stores the 50 most recent successful timesteps for that task. Furthermore, the training pipeline employs data-paraliel architecture, with each parallel process maintaining its own instance of STR. Let DSTr represent STR, n denote the number of sampled timesteps per task, and Don be the on-policy sampled data. The training dataset 'D for each update is generated as follows:
D
sample(DsIR(t))
Don
EEtasks
With STR, focus training proven, effective actions, reducing the need for inefficient trajectory generation and improving exploration efficiency in complex, open-world environments.
3.3 Succeed or Learn Slowly
We highlight two critical observations about RL training for foundation models.
First, standard RL from human feedback (RLHF) techniques use policy regularisation prevent reward model hacking. However, this concern diminishes in environments with struetured action formats. Unlike free-form generation, where models might produce deceptive but incoherent text, structured environments naturally constrain actions through predefined formats. These constraints ensure outputs remain coherent and valid within the environment's action space.
Second, the actor-critic policy loss function -A; log (a; st) creates unintended consequences when fine-tuning models with large output vocabularies. When the advantage negative (A, < 0), the policy decreases the probabilities of tokens that formed poor actions. This increases probabilities for other tokens across the model's vocabulary; tokens that may be semantically related but inappropriate for the specific action space. This redistribution disrupts the model's learned representations. This explains the rise of rejection sampling methods in multi-turn RL literature, such as DigiRL for app control, as they avoid destructive negative updates while still improving policy performance.
We start with the off-policy actor-critic algorithm [1]. assuming data is sampled from a behavioural policy my, used to generate the training data. We write the objective with the state distribution d (s) under the policy Th. subject to the learning policy being close to the base policy:
max Σ 1 (s)
subject to I [
0) KL.(mol ([Tol|m) TT) Se
(3)
Following the work of Degris et al.
(ao(als)
VJ(I) Σ d (s)
Σ d (s)
(s,
(4)
T((as)

---

We also subtract the state value from the state-action value to reduce variance. The state value is computed under the behavioural policy Tb, which does not change with actions and therefore does not introduce bias into the policy gradient.
ViT((als)
VJ(0) Σ d
Σ
Tb(als) (Q V (s))
(5)
Additionally, replace the state-action value with the Monte Carlo return R in Equation (6) to reduce the bias of the estimator, especially early on when the value function is randomly initialised. This aligns with the work of Degris et al. [11], where the state-action value is replaced with X-returns under the behavioural policy.
To ensure consistency with the objective in Equation T we apply PPO-like (2) cut-off in the policy gradient update when the advantage is negative. This restricts policy updates when the advantage is negative and the importance sampling ratio falls outside the allowed range, preventing the policy from deviating too far from the policy that generated the action. The gradient of the loss function for the actor, with A(s, a) R - (s)(s), is therefore:
E MAND
[A-
VTa(ala) ma(als)
A 0 or (1-Cs
(als) (a(s) S c
Vao
(6)
otherwise
In contrast to the original PPO objective, which restricts policy updates when A < 0 and only when the importance sampling ratio falls below 1- E, our approach introduces symmetric constraint by also restricting updates when the importance sampling ratio exceeds 1 +
The baseline critic is trained using the same off-policy data. The value function is represented by adding an affine layer followed by a sigmoid activation on top of the final hidden layer of the transformer. We use Monte Carlo targets to avoid the need for separate target network, which would significantly reduce the computational efficiency of SoLSS. The value network parameters are updated by minimising the squared TD-loss:
Vi (s))² Cor = E R. (R- V2-
(7)
We refer to this algorithm as Succeed or Learn Slowly, or SoLS for short, which aims to update the policy following the off-policy actor-critic objective when the advantage is positive, while it follows PPO-like regularisation in the policy updates when the advantage is negative. A hige-level illustration of SoLS shown in Figure [ SoLS updates of positive samples due to small or large values in the importance sampling. Finally, we optimise the joint loss function by adding the two losses:
be combined with STR without pruning the
L = Lac + A Cer
(8)
4 Experiments
4.1
Evaluation Baselines
4.1.1 Prompting and fine-tuned methods
First, we compare SoL.S with three baseline agents that exclu- sively leverage large foundation models, such as GPT-4o. T3A and M3A (28) both employ a two-step prompting process: the agent summarises the previous observation, and then proposes action based on this summary and the current observation. The primary distinction between T3A and M3A lies in the input format: T3A relies only on the UI accessibility tree, while M3A also uses screenshot, annotated with bounding boxes around each UI element. Additionally, we evaluate SeeAct [38], which also performs two-step prompting, first generating high-level output, then a grounded action.
GPT-4o pure prompting methods:
We evaluate two of the UGround-VI models, UGround-V] 2B and 7B. UGround combines aspects of the planner-grounde SeeAct framework (38) and the actor-summariser M3A 28 along with a fine-tuned grounding model. The agent prompts GPT-4o for high-level action, grounds the action with the UGround model, and then creates summary by prompting GPT-4o.
UGround 14]
5

---

AriaUl [36]
AriaÜl follows a similar three-step grounder-summarisee architecture to UGround, with fine-tuned 24.9B mixture-of-expert Aria [19] model (3.9B active parameters) as the grounder and GPT-4o as the planner and summariser.
We use the OS-Atlas-Pro-77 variant of OS-Atlas, trained greater number of datasets. It has a two-stage training approach: first, GUI grounding pre-training on a custom corpus of 2.3 million screenshots, and then action fine-tuning on agent datasets such AndroidControl. OS-Arlas-Pro- Prolaa------BB uses Qwen2-VL-7B [33 as a backbone, and takes as input the current goal, previous actions, and the current screenshot. A large prompt also describes the current task and action space, including custom action descriptions tailored to each dataset.
OS-Atlas [5
4.1.2 RL methods
PPO: Proximal Policy Optimisation (PPO) [29] is an RL algorithm that uses a heuristic clipping mechanism to prevent the training policy from deviating too much from the prior distribution. PPO can perform multiple consecutive epochs of updates using the same batch of generated data, effectively extracting more learning signal from each environment interaction while the clipping mechanism ensures the policy does not change too drastically between updates.
A2C-STR: Advantage Cetor-Critic (A2C) [26) is on-policy actor-critic algorithm. We augment with STR by incorporating transitions from previously successful episodes into training updates. Since A2C is designed for on-policy learning, using off-policy data from the STR buffer requires correction. Following Degris et al. [IT], we apply importance sampling to properly weight these off-policy transitions and maintain unbiased gradient estimates. Notably, A2C-STR is equivalent to SoLS when transition has positive advantage. However, unlike SoLS, A2C-STR continues to fully update parameters when the advantage is negative, rather than clipping these updates.
DigiRL-STR: DigiRL 2 is an off- pfl-cc variant of rejection sampling [[5] that evaluates transitions based their advantage rather than their total return. We augment DigiRL with STR to leverage previously collected successful experiences, instead of the original prioritised replay buffer that was used in the original work, ensure consistency. Further details can be found in Appendix A.3
4.2 Results
Table I presents the success rates of SoLS and the baseline methods. For models that do not require proprictary access, we average results over three evaluation runs and report two standard errors for the overall success rate. Our results show that SoL.S achieves the highest overall success rate on AndroidWorld, outperforming all baseline methods.
Table 1: Success rates of different agent methods in the AndroidWorld environment, across task difficulty levels. Overall results for non-GPT-4o methods show average success rates with two standard emrors of the mean across three
Success Rate 1
Overall Success Rate t
Method
Input Type
Easy
Medium
Hard
GPT-4o
SeeAct
screen + UI tree
36.1
17.9
0.0
22.5
T3A
UI tree
66.7
21.4
12.5
40.0
M3A
screen UI tree
61.1
21.4
6.3
36.3
GPT-4o UGround-2B
screen
63.9
25.0
6.3
38.8
FT Mixed
GPT-4o + UGround-7B
screen
69.4
28.6
12.5
43.8
GPT-40 AriaUI
screen + UI tree
66.7
28.6
6.3
41.3
OS-Atlas-Pro
screen
40.7
11.9
6.3
23.8 + 1.2
SFT
UI
38.9
9.5
4.2
22.1 + 2.7
A2C-STR
UI tree
52.8
17.9
2.1
32.1 0.7
PPO
UI tree
53.7
8.3
6.3
28.3 0.7
DigiRL-STR
UI tree
55.6
32.1
12.5
38.8 0.0
SoLS-STR
UI tree
68.5
40.5
16.6
51.3 1.2

---

As method simply fine-tuned on AndroidControl, SFT performs quite poorly, solving mostly easy tasks that are closely related its training dataset, for example, those related to system apps, such as turning on/off Bluetooth, Wifi, etc. OS-Atlas achieves slightly better performance, benefitting from GUI- -Ullggrunding pre-training phase and larger range of fine-tuning datasets. Nevertheless, success rate remains limited and much lower than other methods which are able to have better understanding of the task environment, either through larger priors or better domain learning.
In general, we observe that most methods using GPT-4o tend to perform similarly, in the 36%- 44% range, with methods that use a grounding model achieving the highest performance. Methods using GPT-4o seem to perform best on easy tasks, while struggling more with medium and hard tasks compared with the strongest RL methods. This is likely because GPT-4o's strong prior and generalisation abilities allow it to understand and solve easier tasks well, while harder tasks usually require more in-domain and specific knowledge, where GPT-4o's ' assumptions and reasoning might not be as useful or sufficient. We also note that while UGround-2B performs worse than UGround-7B still performs quite well, leveraging GPT-4o's capabilities.
Among RL methods, DigiRL-STR achieves the highest performance among existing approaches with 38.8% overall success rate. However, our proposed SoLS-STR significantly outperforms all baseline methods, achieving 51.3% overall success rate. This represents 32.5% relative improvement over DigiRL-STR and significantly exceeds even the best GPT-4o-basse methods. The improvement is particularly notable on medium difficulty tasks, where SoLS S-R achieves 40.5% compared to DigiRL-STR's s 32.1% Finally, PPO achieves the lowest success rate overall among the RL algorithms, which highlights the need for STR, as is the only algorithm that is trained on on-policy data.
To further highlight the contribution of asymmetric update mechanism,
compare SoL.S with standard A2C using identical STR parameters. The results show that SoLS-STR improves upon A2C-STR by approximately 60% relative improvement (51.3% 32.1%). This dramatic improvement validates our core hypothesis that conservative updates for negative-advantage actions prevent performance degradation while maintaining aggressive learning from positive experiences.
4.3 Additional Studies
First, evaluate the effect of STR the success rate of different RL algorithms. For this comparison, we augment PPO with STR and, as such, enable the reuse of successful trajectories, transforming our implementation into an off- off-po variant. This off- off-policy nature occasionally results in gradient clipping for samples retrieved from the STR buffer due to the policy divergence constraints. Figure 2a presents the success rates of PPO. DigiRL, and SoLS with and without STR. Our results show that STR significantly improves the success rate for both algorithms. We also observe that SoL. outperforms
Vanilla
60
UGround-78
with STR
Inference Time (Seconds)
50
Success Rate (%)
UGround
40
30
30
SeeAct
20
20
AriaUl
M3A
10
10
OS-Atlas-Pro-78
SoLS-STR
PPO
DigiRL
SoLS
25
30
35
45
50
Agent
Success Rate (%)
(a) Success Rate with/without STR
(b) Success Rate vs Inference Time
Figure 2: Left: Bar plot presenting the average success rate and two standard errors of the mean for PPO. DigiRL. and SOLS with and without STR. Right: Scatter plot illustrating the trade-off between success rate and inference time. The most desirable location the bottom-right, demonstrating strong success rate and low inference time, which SoLS occupies.
7

---

PPO and DigiRL even when all three are trained exclusively on on-policy data, confirming our hypothesis that the asymmetric policy updates lead to higher success rate on AndroidWorid. Notably, Sol.S outperforms even PPO-STR, and is on par with DigiRI.-STR.
Figure 2h illustrates the trade-off between success rate and inference time across different models. SoLS occupies the best position in the bottom-right quadrant, achieving the highest success rate (51.3%) while maintaining the fastest inference time (~ 0.9 seconds). This represents approximately a 60x speedup compared to UGround-7B, the next-best performing agent, which requires 40-60 seconds per step due to its multi-step prompting pipeline and slow grounding model. The efficiency gain is crucial for practical deployment in real-world scenarios where response time is critical.
we show how the success rate of SoLS evolves between the first part of training and the end. For every task category, the success rate at the end of training is higher than near the beginning, showing the importance of RL training. This dif-
In Figure 3]
First 1000 episodes
0.8
1000 episodes
Sucress Rate (9)
ference in success rate is less noticeable for certain task categories, such as Audio or System, where the early success rate is already high. This is be- cause the base SFT model already performs quite well in these categories due to similarities with the training set, and early training steps also re- inforce this behaviour, such that further training only marginally improves performance. In other categories, particularly Files, Maps, and Markor, the increase in success rate is very large, as the agent learns to solve tasks from these categories during exploration and reinforces them throughout training, especially with the help of STR.
0.4
0.2
0.0
Audio
Cateroler
Camera
Contacts
Expersee
files
Maps
Markor
Messages
Recipe
Syatem
Task Category
Figure 3: SoLS success rafe comparison the beginning and end of training, by task category.
4.4 Failure Case Analysis
Most of the tasks that SoL.S consistently fails to solve can be grouped into four categories: tasks requiring memory, tasks requiring visual input, tasks requiring unseen interactions, and long-horizon tasks. We provide further explanation below, as well as some case studies in Appendix B.2
Some tasks require the agent to remember information from previous steps, such as numbers or text displayed in earlier observations. Although SoLS is provided with history of actions, history of observations is not, due to context-length restrictions and concerns about information overload. In addition to memory-related challenges, other tasks might depend on visual input, either directly through images or because the textual representation of certain items, such as mazes, is inadequate. Furthermore, few tasks involve using phone features like the clipboard, which the agent has never encountered and is unable to intuitively understand. Lastly, some tasks demand very long sequences of interactions, with several requiring 15 60 steps for an "optimal" solution. These tasks are inherently difficult, and the likelihood of the agent making a mistake, particularly without a rich context of past observations, increases significantly. These failure cases provide avenues for future work, such as incorporating a better form of memory into agents alongside SoLS, or using base models with visual input and broader fine-tuning data.
5 Related Work
5.1 Reinforcement Learning with Foundation Models
There has been a growing focus on applying a final RL step to LLMs to improve alignment with human values, a process known as RL from Human Feedback (RLHF) 7 40). In RLHF, a reward model is trained on human-labelled data, then the LLM is trained maximise predicted rewards. To
prevent "reward hacking" [10 [3] [8] the policy is regularised using algorithms like PPO 29 RL
also used fine-tune models for specialised tasks like math and coding [16 30
Several works explored L.MSs for multi-turn tasks. Abdulhai et al. created tasks using the OpenAl Gym interface to evaluate various RL algorithms. While numerous studies apply RL to multi-turn
8

---

tasks (4,33)9 39], many focus on small-scale environments such as AlfWorld [3] or BabyAl [6]. Our work aims to apply RL with LLM-based policies real-world environments, where tasks are more complex and simulation speeds are slow. DigiRI. [2 attempts online RI. but mostly evaluates on tasks with small distribution shift between SFT and RI. steps.
Recently, there has been a turn towards critic-free RL algorithms. Following DeepSeek-RI success []6. several works use Group Relative Policy Optimization (GRPO) [3), which estimates the advantage function by generating multiple responses per prompt and using their average reward as baseline. Extensions have been proposed [5] [24] to improve GRPO. However, GRPO is expensive, requiring typically 64 responses per prompt, while our SoLS generates a single response per prompt.
5.2 Mobile App Agents
Many recent works focus mobile app agents for Android smartphones. Most current agents rely on prompting large proprietary models like GPT-4o. Complex prompting agents achieve strong performance through exploration, planning, and reflection [32 34 37,, sometimes with replanning and multiple passes
[4. However, this intricate prompting requires numerous costly API calls.
Some agents two-step prompting methods, plamning planning-grounding (38 acting- suaiing-smmmiiiiingg [28] In subsequent work, agents were developed with fine-tuned grounding models integrated into the pipeline, combining GPT-4o and fine-tuned models into planning grrunddnggg grounding-summarrin summarising [1] 36.. While achieving some of the hest online app control results to date, they remain slow and costly, requiring two proprietary API calls per step. Other agents explored using exclusively fine-tuned Several employ two-stage training, using GUI grounding datasets prior to action generation fine-tuning [21 23 35). However, fine-tuned models struggle to match large proprictary LLMs' performance, and some works are limited to offline evaluation
models 9 20 21 27 35
O 20 35
DigiRL [2] is the most closely related, using SFT followed by RL fine-tuning for mobile app control. They performed SFT on a VLM using small AitW subsets, then conducted RL on similar tasks. Our evaluation differs significantly by focusing RL in tasks that are OOD compared to the initial SFT dataset, substantially increasing difficulty.
Conclusion
In this paper, we introduced Succeed Learn Slowly (SoLS), a novel off-policy RI. algorithm for mobile app control tasks. SoLS addresses the challenges of sparse rewards and high simulation costs through asymmetric policy updates. The core innovation is enabling aggressive learning from suc- cessful experiences while applying conservative regularisation to unsuccessful ones. Combined with Successful Transition Replay (STR), SoLS achieves a 51.3% overall success rate on AndroidWorldd outperforming both state-of-the -frt GPT-4o- bo-ddaedd and alternative RL methods. Moreover, this is done with 5-60x faster inference time than the best approaches.
Despite its strong empirical performance, SoLS has several limitations. The asymmetric constraint mechanism can potentially lead premature convergence to local optima, especially when actions receive positive advantages early in training due to noise or limited exploration. SoL also depends heavily on the quality of the initial SFT policy; if that policy has significant deficiencies in certain action types or task domains, SoLS may struggle due to its conservative updates for negative- advantage actions. In highly stochastic environments, where the same action may succeed or fail inconsistently, SoLS can suffer from inconsistent feedback, potentially causing policy oscillations. Lastly, although STR aids knowledge transfer. SoLSS fundamentally cannot learn to succeed in tasks requiring interaction patterns entirely absent from both the initial policy and exploration distribution.
While this work focuses on technical advancements, the development of mobile app control agents raises important societal concerns as these agents develop further. Such systems could eventually be used to compromise user privacy through automated data access, enable new forms of digital surveillance, and create security vulnerabilities if misused for unauthorised access to personal devices.
Our work demonstrates that well-designed RL algorithms outperform much larger foundation models on specialised tasks, with important implications for conotreecon environments. Future work could explore shaped rewards, improved exploration strategies, and theoretical properties of selective constraint mechanisms.
enable smaller language models to

---

References
[1] Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models. arXiv preprint arXiv:231 18232, 2023.
[2] Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. arXiv preprint arXiv:2406.1 2024.
[3] Michael Bowling and Manuela Veloso. Multiagent learning using a variable learning rate. Artificial intelligence, 136(2):215-255 2002.
[4] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference Machine Learning, pages 3676-3713. PMLR, 2023.
[5] Kevin Chen, Marco Towner, Brody Huval, Aleksei Petrenko, Jackson Hamburger, Vladlen Koltun, and Philipp Krähenbühl. Reinforcement learning for long-horizon interactive IIm agents. arXiv preprint arXiv:2502.01600 2025.
[6] Maxime Chevalier-Boiswe Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. arXiv preprint arXiv: 18000882,, 2018.
[7] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[8] Filippos Christianos, Georgios Papoudakis, Matthicu Zimmer, Thomas Coste, Zhihao Wu, Jingx- uan Chen, Khyati Khandelwal, James Doran, Xidong Feng, Jiacheng Liu, al. Pangu-agent fine-tunable generalist agent with structured reasoning. arXiv preprint arXiv: 14878, 2023.
[9] Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, and Kun Shao. Lightweight neural app control. International Conference on Learning Representations, 2024.
[10] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.0274 2023.
[11] Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic, arXiv preprint arXiv:/205.4883, 2012.
[12] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[13] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[14] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024.
[15] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahem, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08990 2023.
[16] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 2025.
10

---

[17] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint vrXiv22/66009685 2021.
[18] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving. Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
[19] Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen. Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang. Yanpeng Li, al. Aria: An open multimodal native mixture-of-experts model. arXiv preprint arXiv:2410.05993, 2024.
[20] Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Camphell-Ajala Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv preprint arXiv:2406.0334 03679, 2024.
[21] Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang. Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision- aaiionlanguage-acce model for gui visual agent. arXiv preprint 1rXiv:2411.174000 2024.
[22] Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8:293-321, 1992.
[23] Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection. arXiv preprint arXiv:2501.0455 2025.
[24] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078. 2025.
[25] Ilya Loshchilov, Frank Hutter, al. Fixing weight decay regularization adam. arXiv preprint arXiv:171..05101, 2017.
[26] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. International conference on machine learning, pages 1937-1937. PmLR, 2016.
[27] Georgios Papoudakis, Thomas Coste, Zhihao Wu, Jianye Hao, Jun Wang, and Kun Shao. Appvlm: A lightweight vision language model for online app control. arXiv preprint 25K2::2502.06395, 2025.
[28] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405. 145733 2024.
[29] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1ee::700700444 2017.
[30] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033300 2024.
[31] Mohit Shridhar, Xingdi Yuan, Mlee-Alex Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint 1rX0v20010.03768, 2020.
[32] Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406.0100 01010 2024.
11

---

[33] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[34] Xiaoqiang Wang and Bang Liu. Oscar: Operating system control via state-aware reasoning and re-planning. arXiv preprint arXiv:2410.1890 2024.
[35] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprini arXiv:2410.2222
2024.
[36] Yuhao Yang. Yue Wang. Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256. 2024.
[37] Zhao Yang. Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023.
[38] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is generalist web agent, if grounded. arXiv preprint 44X1v:2401.01614, 2024.
[39] Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402. 19446, 2024.
[40] Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. URL https://arxiv. org/abs, page 14, 1909.
12

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: The main claim of the paper is around the asymmetric policy updates, based on how the performed action is evaluated by the advantage function. Our results validate the main hypothesis of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are mentioned in the Conclusion (Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
The paper does not introduce any theoretical claims or results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The implementation details described in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [No]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodemsssseeeeee 7/nips.com http///bbb////uuuuoooooo//aaaaaa www.onicccccccicccoooooociiicipi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The details are presented Appendix A.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Standard enrors of the mean provided in experiments. Details provided Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://wwwuppp.cccpo//tccccccca
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper focuses on developing smart assistants, with positive impacts discussed in the Introduction (Section)) I and negative impacts addressed in the Conclusion (Section (6.. This work is primarily focused machine learning research and its current applicability to real-world implementations remains limited.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No models or data are released as part of this work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Information is included throughout the paper and in Appendix A.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not provide any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper does not an LLM to generate ideas suggestions around the paper methodology.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
19

---

A
Experimental Details
A.1 Notation
Table 2: Notation used equations.
Symbol
Description
Action taken by the agent
8
Environment state
F.
Episode-terminaa reward
Clipping parameter controlling trust region around importance ratio
R
Monte Carlo return
A
Advantage function
Value function under policy Th
Q"*
Action-value function under policy mi
me
Parametric policy with parameters
Th
Behavioural policy used to generate training data
(s)
State distribution under policy ny
Training dataset
DSTR
STR dataset
Don
On-policy dataset
Doff
Off-policy dataset
I(0)
Objective function of policy parameters 6
Lac
Actor function
Ler
Critic (value regression) loss
Combined actor-critic SoLS loss function
A.2 Implementation Details
We use Llama- (ama--B-Iin IB-Instruct as our base model. We fine-tune the model using LoRA adapters to minimise computational requirements. The AdamW optimiser (2 is used across all experiments.
During SFT, we update the model parameters for three epochs using the AndroidControl training set. We apply learning rate of 10-1 that linearly decays to O. We use an effective batch size of 64. The LoRA adapters are configured with 64 dimensions, lora-c of 32, and dropout rate 0.05.
During RL fine-tuning, we train the model for 15K episodes, equivalent to approximately 200K transitions. This transition count varies between algorithms since those with higher success rates complete episodes more quickly, resulting in fewer overall transitions. For training, we implement data parallelism with 8 concurrent processes. Each process asynchronously interacts with emulators and collects data until accumulating 100 on on-policy transitions. These 100 timesteps, combined with 50 transitions sampled from the STR, are used for learning. We perform mini-batch gradient updates with batch size 64 and set the hyperparameter to 0.2. The learning rate for RL fine-tuning remains constant at 10-5 throughout training. We conduct a single training epoch for all algorithms except PPO, for which we perform two epochs on the training data.
For value function estimation, we add extra affine value head on top of the policy's last hidden layer. We allow value loss gradients to propagate through all trainable model parameters. To balance the policy and value loss functions, set 0.5.
A.3 DigiRL Details
In our implementation of DigiRL-STR we use the STR to augment the training algorithm and not the original prioritised experience replay used by the authors. We used this modification to ensure consistency among training algorithms. DigiRL also performs step-level filtering, where each transition is filtered using step-based advantage estimation (Equation 4.3 of the original paper), presented below and adjusted to our notations:
A ar) X "-rr(s an) + (1 =- (V) + r(s,, ar, c) (st)) (9)
where is the horizon, and value of is set to 0.5. Transitions with advantage larger than 0.05 are used for training, and the rest are discarded. Note that even though DigiRI, is off-policy, it does not use importance sampling, instead opting for other measures. We keep this consistent with the original algorithm.
20

---

A.4 Data and Benchmark
A.4.1 Action and observation space
In this section, we describe the action space and observation inputs used by our SoLS agent, as well as other RL methods. For the remaining baselines, we adopt their respective action spaces and observation formats, using provided code with only minor modifications where applicable.
The action space of our agent adheres to standardised JSON-style format, specifying both the action type and any optional parameters: ("action-type" <type>>, "action-extze" <extra>) The list of available action-type and action-exter options is detailed in Table and is consistent across both the AndroidControl and AndroidWorld environments. Standardising the action space across the SFT dataset and the RL environment is critical to ensure effective transfer of training. Therefore, actions are consistently converted to and from this format during training and evaluation in the click and Long-press actions operate on target UI elements rather than on explicit (z, screen coordinates. This design choice simplifies action generation and increases the likelihood of generating robust, executable actions, especially in the absence of GUI-gre BUI-grounding training. The referenced target element can be translated into an (z, y) coordinate at execution time using its bounding box.
both environments. As in prior work [e.g.. 2 20 28 38
Input
Action type
Action extras
"Take photo"
open-app
<app name?>
Goal: <goal description>
input-text
<text>
click
<arrgt element>
/'tert: '2r', 'dass!:
long-press
<target element>>
'android widget "content description': Zoom In!, 1:f'clase::
wait
Observation:
scroll-up
<List of UI elements>
widge imageee imageView!
scroll-down
ccntent description!: Shutter, 2:..
scroll-left
scroll-right
Previous Actions:
navigate-hom
<List of past actions>
l'action type': 'open apo', app__naee 'Camera')
navigate-back
Table 3: Our action space.
Figure 4: Input to SoLS and other RL methods.
As described in Sections 2.2 and 3.1 both AndroidControl and AndroidWorld provide the task goal, the UI accessibility tree, and screenshot of the phone interface each step. Since SoLS and the other RL baselines we evaluate are based on Llama-3-8B and are text-only models, the screenshot is omitted. This significantly reduces the number of input tokens, thereby decreasing the model's inference time. The UT tree is transformed into a list of UI elements, each represented in a JSON-like format that includes any relevant text metadata provided by the tree. This is concatenated with the textual goal description to form the first part of the model's 's input. The final part of the input the history of actions, which is recorded the agent progresses through each task and is appended the model's 's input. A visualisation of the overall observation and input structure is shown in Figure 4
A.4.2 Evaluation and benchmark set details
As discussed in Section 2.2 the AndroidWorld benchmark is used for evaluation throughout our experiments. Although the original benchmark consists of 116 tasks, we employ a subset of 80 tasks. Q&& tasks are excluded, as they represent a distinct category not supported by our agents, action space, or by AndroidControl. Verification tasks are also removed, since our evaluation procedure checks for task success at every step, rendering these tasks trivially solvable. Lastly, we exclude tasks that require free-form actions, such as drawing, as these are incompatible with the agents' current action spaces. Notably, most of the omitted tasks fall under the "easy" difficulty category, resulting in a task distribution that more challenging than that of the full benchmark suite, as shown by the task distribution charts in Figure 5
21

---

Hard
Hard
19 (16.4%)
(20.0%)
Easy
36 (45.0%)
Easy
61 (52.6%)
36 (31.0%)
Medium
(35.0%)
Medium
(a) Full benchmark
(b) Our subset
Figure 5: Pie charts comparing task difficulty distribution between the full AndroidWorld benchmark, and the task subset used in this work.
B
Additional Results
B.1 Inference Time and Model Size
Figure 2b illustrates the trade-off between success rate and inference time across different agents, demonstrating that SoLS not only outperforms the baselines in terms of success rate but also achieves significantly faster inference times. In Figure 6 we further examine the model sizes and inference times of the various agents side-by-side, offering insight into the memory, compute, and latency requirements of each approach. Purely fine-tuned methods, such as OS-Atlas-Pro and our implementations, are the least demanding in terms of both model size and inference time. Note that we report the model size of AriaUl as 24.9B, reflecting the total number of parameters, even though only 3.9B are active during inference.
60
10a.
GPT-4o Estimated Size
GILL
Inference Time
Inference ce Time (Seconds)
1011
-30
Model
101e
-20
-0
SeeAct
T3A
M3A
UGround UGround-78 Agent Name
AriUUl
05 Ar-attssPPro
Figure 6: Model size and inference time of different agents. GPT-4o size is estimated at trillion parameters and is shown on top of the fine-tuned grounding model size for mixed prompting -ine--ng--iin- tuned agents.
B.2
Failure Modes and Case Studies
The main failure modes and analysis of the types of tasks that SoL.S repeatedly fails solve introduced in Section 4.4 This section provides further explanation, examples, and illustrations of these failure cases. A representative example for each failure category included in Table 4
Lack of memory. Some tasks require the agent to retain information across multiple steps. While SoL.S receives a history of past actions as part of its input, this provides only limited information about prior states. Due context-length limitations and computational considerations, the history of
22

---

Table 4: Example failed tasks for each failure category.
Failure Mode
Example
Memory
Open the file task.html in Downloads in the file manager; when prompted open with Chrome. Then click the button 5 times, re- member the numbers displayed, and enter their product in the form.
Visual Input
Add the recipes from recipes.jpg in Simple Gallery Pro to the Broccoli recipe app.
Unseen Interactions
Copy the following text to the clipboard: {clipboard_contenl)
Long Tasks
Save track with waypoints Ruggell, Liechtenstein. Bendern, Liecht- enstein in the OsmAnd maps app in the same order as listed.
observations not included. Consequently, SoLS struggles with tasks that depend on remembering past information, as illustrated by the example in Table A possible mitigation strategy in future work is the integration of a state summarisation mechanism at each step, which can be fed into subsequent model inputs. A common approach in related works [e.g.. 28 36) is to use GPT-4o as an external summarisation module. However, this incurs the cost of querying GPT-4o at every step.
Lack of visual input. SoL.S relies solely on text-based inputs, which leads to two primary chal- lenges. First, although UI accessibility trees typically provide high-quality textual representations of elements, they can be inadequate for image- based elements. This is exemplified in Figure 77a where the maze's layout and the agent's position are not described textually, leaving the agent to take essentially random actions. Second, some tasks require extracting information from images, capabilities that text-only model cannot support. One such case shown in Table Mitigating this failure mode may require transitioning to a Vision- Dision-Language Model (VLM). Alternatively, one could apply Optical Character Recognition (OCR) extract textual information from image components of the UI element list and incorporate it into the input.
Unseen interactions. Tasks in AndroidWorid are -ittt--diitribution (OOD) relative to the An- droidControl SFT data. Although many tasks can be addressed by generalising new applications through similar interaction patterns, some require the use of previously unseen device features or unfamiliar actions. For instance, the clipboard functionality, illustrated in Table 4 has never been encountered by SoLS. Given the rarity of the long long-press press action in the training data (only 0.2%), the agent unlikely to discover and reinforce correct usage through exploration. A similar issue is presented in Figure 7b) where the agent attempts to append text to pre-filled field without first clearing it, again requiring a rarely used long-press action. potential mitigation to expand the SFT dataset to include broader variety of tasks and interactions, or to inject exploratory behaviours from other models during RL. training.
Long-horizon tasks. Certain tasks in AndroidWorld involve lengthy action sequences. Specifically, nine tasks have optimal trajectories exceeding 20 steps, and three exceed 30. One particularly difficult example in Table 4 requires 60-step optimal solution. Such tasks are inherently difficult for any agent, as the probability of making critical error increases with sequence length. This is especially problematic in sparse reward settings, where the agent only receives a positive reward upon full task completion. In such scenarios, RI. agents can only reinforce successful strategies after discovering a full solution, which is highly improbable given the length of these tasks. While difficult to fully mitigate, incorporating memory mechanisms (as discussed in the first failure mode) could help improve performance on these long-horizon tasks.
Combined failure modes. Many of the most challenging tasks fall under multiple failure categories simultaneously. This is particularly true for tasks involving visual input, which often also require memory to retain image-derived information over multiple steps. fact, many such tasks are additionally morizon, making them exceptionally difficult under current system limitations.
23

---

Image
Ul element list
Image
Goal
Record on sudio cli and save with name "2023, 07 04, meeating, miaa" Audio Recorder
Auda
('teet: X, 'closs: aandroid Texteee
Up,,
wigget Buttoon)
(Tweet: 'Down, landroid wridger Butioe!)
10: (Teett: Left', 'class: lnndoodd widget Button
11: Preath: Hight" 'cawel: 'android wi diget Button))
"2023_07, _0__0_00_,,,_00 _t__________geeeeeee mite")
Correct Action
("Ection type "Ton press" "targe: diemen"" "2"
(a) Image and textual representation of
(b) Failure to clear text before adding filename
Figure 7: Two examples illustrating the cause of failure cases. (left) The textual representation of the maze, highlighted in red, does not describe the content of the maze visible in the image. (right) Having never seen pre-filled text fields, the agent tries to input the filename, instead trying to clear it first, with the long-press press action.
B.3 Training Curve
Figure 8
presents the performance of training run of SoLS-STR, as success rate across episodes. Final training success rate slightly lower than the evaluation value in Table due higher sampling temperatures used during training.
Success Rate Time
0.5
0.4
Success Rate
0.3
0.2
0.1
0.0
3k
6k
9k
12k
15k
Number Episodes
Figure 8: SoLSS-TR success rate throughout training.
24


Reviews:
[{"review_id": "2rb0rZTkK6", "reviewer": "Reviewer_WCy2", "summary": "This paper addresses the challenge of sample-efficient RL for mobile app control tasks, particularly when fine-tuning foundation models in sparse reward settings. The authors identify that standard RL updates from failed actions can destabilize and degrade a model's performance. They introduce Succeed or Learn Slowly, a novel off-policy actor-critic algorithm. The core idea of SoLS is an asymmetric policy update mechanism: it applies direct, aggressive updates for successful actions, but uses conservative, regularized updates for unsuccessful ones to prevent catastrophic forgetting. To further improve sample efficiency, SoLS is augmented with Successful Transition Replay, a technique that prioritizes learning from a stored buffer of previously successful interactions. Evaluated on the AndroidWorld benchmark, the proposed SoLS-STR method significantly outperforms existing approaches, including those using the much larger GPT-4o model, while also being 5-60x faster at inference.", "questions": "1. SoLS vs. Rejection Sampling (DigiRL): Your motivation for SoLS's conservative negative updates is to avoid the destructive policy changes that can occur with standard actor-critic losses. This is similar to the motivation for rejection sampling methods like DigiRL, which avoids this problem by simply ignoring negative-advantage samples entirely. Your results show SoLS-STR dramatically outperforms DigiRL-STR (51.3% vs 38.8% success rate).  Could you elaborate on why \"learning slowly\" from failures is so much more effective than not learning from them at all in this context? What is the key mechanism that accounts for this significant performance gap?\n\n2. The paper argues that the SoLS algorithm is designed to be more stable than standard RL approaches in this context. However, the authors also acknowledge potential limitations where SoLS could face instability. can you present the training curves of multiple seeds?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper's central insight—that updates from negative samples should be handled differently from positive ones in this domain—is both simple and effective.\n\nWeaknesses:\nWhile the overall SoLS algorithm is novel, its core components are inspired by existing concepts. STR is a specialized form of experience replay , and the conservative update mechanism is conceptually related to the clipping in PPO or the filtering in rejection sampling methods like DigiRL.", "was_revised": true, "final_justification": "The rebuttal successfully clarified that the novelty of SoLS lies in the specific and effective composition of existing RL concepts for the mobile app control domain. In summary, the paper's empirical results are strong enough. However, this updated score is conditional upon addressing the two points above to ensure the scientific integrity and clarity of the final publication.", "rebuttal": "We thank the reviewer for their valuable feedback and comments. We address the main concerns below and clarify key points raised in the review.\n\n**Weaknesses:**\n\n> While the overall SoLS algorithm is novel, its core components are inspired by existing concepts. STR is a specialized form of experience replay , and the conservative update mechanism is conceptually related to the clipping in PPO or the filtering in rejection sampling methods like DigiRL.\n\nWhile SoLS draws on established ideas such as experience replay and advantage-based updates, its contributions lie in how it adapts and composes these for the unique challenges of mobile app control with foundation models.\n\nWhile STR shares the general principle of reusing past experiences, it differs fundamentally from standard replay by storing only successful timesteps rather than all experiences, which is crucial in sparse reward environments where the vast majority of experiences are failures. It also stores individual transitions and uses task-specific buffers for a more balanced task distribution.\n\nOur asymmetric update mechanism goes beyond PPO and DigiRL by:\n\n- Applying importance sampling in off-policy data.\n- Conservatively updating on negative advantages instead of discarding them, as in DigiRL. The difference with DigiRL is further elaborated in the following section.\n- Using an advantage-based policy gradient update, in contrast to DigiRL which only maximises the log likelihood.\n\nThese design choices enable SoLS to outperform all baselines, achieving >32% relative improvement over DigiRL and PPO. Our contribution lies in finding the right balance between aggressive learning from positive experiences and conservative but non-zero learning from negative experiences, combined with task-specific experience replay, a combination that proves more effective than existing approaches in our challenging mobile app control setting.\n\n**Questions:**\n1. > SoLS vs. Rejection Sampling (DigiRL): Your motivation for SoLS's conservative negative updates is to avoid the destructive policy changes that can occur with standard actor-critic losses. This is similar to the motivation for rejection sampling methods like DigiRL, which avoids this problem by simply ignoring negative-advantage samples entirely. Your results show SoLS-STR dramatically outperforms DigiRL-STR (51.3% vs 38.8% success rate). Could you elaborate on why \"learning slowly\" from failures is so much more effective than not learning from them at all in this context? What is the key mechanism that accounts for this significant performance gap?\n\nSoLS has a few important differences compared to DigiR, which we will clarify in the next revision:\n\n- **Information preservation from negative samples:** Even negative-advantage samples contain valuable learning signal that complete rejection wastes. SoLS retains and constrains them, while preserving learning potential. This also improves sample efficiency in a data-scarce environment.\n\n- **Advantage-weighted updates:** DigiRL treats all positive samples equally, using standard -log π(a|s) loss regardless of how successful the action was. SoLS weighs positive updates by the advantage magnitude A(s,a), creating stronger gradients for highly successful actions and weaker ones for marginally positive actions. This provides more nuanced learning that better reflects action quality.\n\n- SOLS uses importance sampling to weigh each sample, which appropriately handles off-policy data.\n\nNote that to ensure a consistent and controlled comparison across methods, we re-implemented DigiRL within our training framework. This required some minor adjustments to match our shared setup. For example, we did not include the instruction-value replay buffer (Section 4.3 in DigiRL), instead using STR (lines 230-235).\n\nWhile these changes were made for practical reasons, we do not believe they account for the performance gap. The core training behaviour, retaining only positive-advantage samples and applying uniform log-likelihood updates, remains consistent with the original DigiRL approach. Moreover, in our setup, we observed that the modified step advantage estimator and filtering logic had similar practical effects to SoLS’s advantage-based filtering: positive transitions were retained, and negative ones excluded.\n\nWe believe the observed performance gains are primarily attributable to SoLS’s advantage-weighted updates, off-policy correction, and conservative treatment of negative samples, rather than differences in auxiliary components.\n\n\n2. > The paper argues that the SoLS algorithm is designed to be more stable than standard RL approaches in this context. However, the authors also acknowledge potential limitations where SoLS could face instability. can you present the training curves of multiple seeds?\n\nWe appreciate the reviewer’s interest in the stability of SoLS and agree that multi-seed training curves would provide additional empirical support. Unfortunately, due to the high computational cost of training in the AndroidWorld environment (each run requiring thousands of slow emulator interactions), running multiple full training seeds is not feasible within the rebuttal timeline.\n\nThat said, we have taken several steps to address stability concerns:\n\n- In Table 1, we report the mean and two standard errors over three independent evaluation runs, providing a partial measure of variance across different task initialisations and evaluation conditions.\n- As mentioned in Section 4.2, during development, SoLS exhibited notably more stable training than baselines. A2C-STR required considerable effort to stabilise and frequently diverged, while SoLS trained consistently across our runs and required minimal tuning.\n- In Figure 2a (STR ablation), SoLS outperforms PPO and DigiRL both with and without STR, indicating that its benefits are not overly sensitive to specific replay buffer configurations and suggesting a degree of robustness in training dynamics. Indeed, this suggests robustness to different training runs and variations.\n\nFinally, we acknowledge the concern raised in our limitations section that unrestricted positive updates may lead to convergence toward local optima, such as focusing too narrowly on simpler tasks. However, this is also dependent on other factors such as the quality of the initial SFT model, the representational capacity of the backbone, and the distribution of tasks in the environment. Importantly, we observed that for simpler tasks, solved early in training, the advantage tends to converge toward zero, effectively suppressing further updates to those actions. This self-regulating dynamic helps prevent overfitting to easy tasks and allows SoLS to shift focus to more complex behaviours over time. While we include this point in the limitations for completeness, we did not observe it to pose a major issue in our experiments."}, {"review_id": "DLIuAwINmr", "reviewer": "Reviewer_nx8y", "summary": "The paper proposes a two-stage approach consisting of supervised fine-tuning on human demonstrations, followed by reinforcement learning with SoLS, which applies direct policy updates for positive advantages and PPO-style KL regularization for negative ones. To improve efficiency, training focuses on selectively replaying successful transitions.", "questions": "1. The concept of the behavior policy in Section 3.3 requires further clarification.\n\n2. The authors claim that SoLS is an off-policy method, but the experiments only compare it with on-policy methods. The authors should explain this choice, and if possible, add comparisons between SoLS and state-of-the-art off-policy methods.\n\n3. The explanation of how STR addresses a significant distribution shift from the original SFT phase needs further elaboration.\n\n4. SoLS does not clip updates for positive advantages, unlike PPO, which clips all updates to ensure stability. While this may accelerate learning, it could also lead to instability if the advantages are noisy. Please clarify this design decision and provide supporting evidence.\n\n5. The idea of selectively updating the policy based on chosen samples seems to have been explored in the DigiRL. The main difference appears to lie in the thresholding strategy. The novelty of SoLS needs to be more clearly articulated.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.Successful Transition Replay(STR) improves sample efficiency by selectively replaying successful transitions.\n2.The proposed SoLS applies direct policy updates for positive advantages and conservative, regularized updates for negative ones, reducing policy degradation.\n\nWeaknesses:\n1.There are instances of unclear descriptions. For example, the term \"behavior policy\" in Section 3.3 needs further clarification.\n2.The terms in equations should be explicitly defined and clarified.\n3.The acronym \"RLHF\" appears for the first time in Section 3.3 without being spelled out.", "was_revised": true, "final_justification": "The rebuttal has successfully clarified the novelty of the SoLS method, the effectiveness of STR in mitigating distribution shift, and the details of the experimental setup. These clarifications have effectively addressed many of my initial concerns, particularly with respect to the method’s originality and the rigor of the evaluation. Overall, the paper demonstrates sufficient innovation and empirical strength. However, I recommend that the score update be made contingent upon the authors addressing two remaining issues and providing further clarifications in the final published version.", "rebuttal": "We thank the reviewer for their valuable feedback and comments. We address the main concerns below and clarify key points raised in the review.\n\n**Weaknesses:**\n1. > There are instances of unclear descriptions. For example, the term \"behavior policy\" in Section 3.3 needs further clarification. \n\nThe “behavioural policy” refers to the policy that was used to generate the data contained in the training batch. In off-policy reinforcement learning, the agent learns about a different policy (the target policy) than the one that generated the data. We will clarify where this term is first used, with something along the lines of “that is, the policy that was used to generate the training data, which may differ from the policy currently being optimised.”\n\n2. > The terms in equations should be explicitly defined and clarified. \n\nWe thank the reviewer for this feedback. We agree that several equations, particularly in Section 3, would benefit from more explicit definitions of all terms. In the next version, we will revise the equations to clearly define each variable and function upon first use or within a terminology table, including notations like $d^{\\pi_b}(s)$, $Q^{\\pi_\\theta}(s, a)$, and the advantage function $A(s, a)$. We will also ensure that all assumptions (e.g., use of Monte Carlo returns, importance sampling ratios, clipping thresholds) are clearly stated in context. We will also better define terms such as the behavioural policy, as stated above.\n\n3. > The acronym \"RLHF\" appears for the first time in Section 3.3 without being spelled out.\n\nApologies for this, we moved the related works section towards the end of the paper at the last minute. We will change this so that the first use of RLHF spells out the full acronym.\n\n**Questions:**\n1. > The concept of the behavior policy in Section 3.3 requires further clarification.\n\nWe kindly refer the reviewer to Weakness 1 above.\n\n2. > The authors claim that SoLS is an off-policy method, but the experiments only compare it with on-policy methods. The authors should explain this choice, and if possible, add comparisons between SoLS and state-of-the-art off-policy methods.\n\nSoLS is off-policy in the sense that it uses data from STR (collected under previous policies) alongside on-policy data. It also performs updates using mini-batches of collected data, similar to standard PPO implementations. We maintained consistent implementation details across all algorithms to ensure fair comparison.\n\nHowever, the baseline methods we compare against are off-policy in the same manner. The buffered updates are a necessary component of the training process for foundational models, which all methods follow. Additionally, we systematically explore the use of off-policy STR data across all RL methods in our evaluation. Particularly, we would like to highlight Figure 2a, which illustrates this quite well. In this figure, we compare our method alongside PPO and DigiRL, both with and without off-policy STR data. The results demonstrate that our method outperforms the baselines in both scenarios, while also showing that incorporating STR data yields significant performance improvements across all methods.\n\n3. > The explanation of how STR addresses a significant distribution shift from the original SFT phase needs further elaboration.\n\nWe first recap how the agent faces a significant distribution shift between SFT and RL phases:\n\n- **SFT training:** AndroidControl dataset with wordy task descriptions, different app distribution, and actions like long-press appearing in only 0.2% of examples\n- **RL evaluation:** AndroidWorld with imperative task descriptions, unseen apps, and frequent use of previously rare actions like long-press\n\nThis shift causes the SFT model to initially fail at most AndroidWorld tasks (22.1% success rate), making traditional RL exploration inefficient.\n\nSTR addresses this in the following ways:\n\n- **Bootstrapping from rare successes:** Even with poor initial performance, the agent occasionally succeeds through random exploration or partial transfer from SFT. STR captures these rare successful experiences that would otherwise be lost.\n- **Accumulating domain-specific knowledge:** As training progresses, STR builds a repository of successful interactions specifically for AndroidWorld tasks. This creates a bridge between the SFT distribution and the target distribution.\n- **Preventing catastrophic forgetting:** Without STR, successful exploration in later training might be forgotten when the agent encounters different tasks. STR preserves these difficult successes across the training process.\n\nFigure 3 shows this effect clearly - categories like Files, Maps, and Markor show dramatic improvement from first to last 1000 episodes, indicating that STR helps the agent learn new task-specific behaviours absent from the SFT distribution. Meanwhile, categories like Audio and System (which transfer well from SFT) show smaller improvements.\n\nThe comparison of methods with/without STR (Figure 2a) demonstrates STR's importance: even PPO and DigiRL show substantial improvements when augmented with STR, validating that the technique addresses fundamental challenges in this out-of-distribution setting.\n\nIn essence, STR acts as a mechanism to gradually shift the training distribution from the SFT domain toward the target domain by preserving and amplifying successful domain-specific interactions. We will make this clearer in the next revision.\n\n4. > SoLS does not clip updates for positive advantages, unlike PPO, which clips all updates to ensure stability. While this may accelerate learning, it could also lead to instability if the advantages are noisy. Please clarify this design decision and provide supporting evidence.\n\nOur decision not to clip positive advantage updates is based on the very sparse positive signal of the targeted environment. In mobile app control, positive advantages are extremely rare and valuable. AndroidWorld provides binary terminal rewards, meaning most episodes yield zero reward. When we do observe positive advantages, they represent genuinely successful behaviours that we want to reinforce aggressively without any constraints.\n\nSome supporting evidence:\n\n- **Empirical stability:** Despite applying completely unconstrained updates for positive advantages, we did not observe training instability across our experiments. Our training remained stable throughout.\n- **Comparative performance:** The significant improvement over PPO suggests that aggressive, unconstrained positive updates are beneficial rather than harmful. If instability were a major issue, we would expect more erratic performance.\n\nThe fact that even SoLS without STR outperforms PPO (as mentioned in our ablations Section 4.3 - Figure 2a) suggests that the asymmetric clipping itself, rather than just the experience replay, drives improvement.\n\nWe acknowledge that in noisier reward environments, this design choice might require more careful consideration, but our experimental evidence supports its effectiveness in structured, sparse reward settings like mobile app control. We will further clarify in the next revision of the paper.\n\n5. > The idea of selectively updating the policy based on chosen samples seems to have been explored in the DigiRL. The main difference appears to lie in the thresholding strategy. The novelty of SoLS needs to be more clearly articulated.\n\nThe observation of the reviewer that both DigiRL and SoLS selectively update based on sample quality is correct, and we will more clearly articulate the key differences in the next revision.\n\nWhile both methods are selective, they handle negative samples in fundamentally different ways. DigiRL performs complete rejection sampling, where negative advantage samples are entirely discarded and contribute zero learning signal. On the other hand, SoLS performs conservative regularised updates, where negative samples are retained but updated with PPO-style clipping constraints.\nIn addition, DigiRL does not weigh the updates with the advantage, while SoLS does, and DigiRL does not perform importance sampling when incorporating off-policy data into its updates.\n\nThis matters for a few reasons:\n\n- **Information preservation and sample efficiency:** In sparse reward environments, completely discarding negative samples (as DigiRL does) wastes potentially valuable learning signal. SoLS extracts conservative learning from these samples while preventing harm.\n- **Empirical evidence:** Our empirical results show SoLS-STR (51.3%) significantly outperforms DigiRL-STR (38.8%), a 32% relative improvement. This suggests that conservative learning from negative samples is more effective than complete rejection.\n\nThe novelty lies in finding the optimal middle ground between aggressive learning and complete rejection. SoLS introduces asymmetric treatment, with unconstrained aggressive updates for positive advantages, and constrained conservative updates for negative advantages.\n\nThis is neither pure rejection sampling nor symmetric regularisation; it's a principled asymmetric approach that maximises learning from both positive and negative experiences while maintaining model stability. We acknowledge that we should better emphasise this key algorithmic distinction from DigiRL, and will do so in the next revision."}, {"review_id": "ebh5NeX7PD", "reviewer": "Reviewer_j81U", "summary": "The paper proposes SoLS, an off-policy RL algorithm for efficient fine-tuning of language models in mobile app control. It uses asymmetric policy updates and introduces STR to prioritize learning from successful steps. On the AndroidWorld benchmark, SoLS outperforms GPT-4o-based and prior RL methods.", "questions": "1. Can you clarify how STR handles tasks with very few or no successful transitions early on? Is there a cold-start issue, and if so, how does the method mitigate it?\n2. Given that SoLS suppresses updates for negative-advantage actions, how does the model learn to handle rare or unseen actions (e.g., long-press or clipboard tasks)? Is there a risk of overfitting to early successes?\n3. Do you think SoLS could generalize to other structured control settings beyond mobile apps—such as form-filling on websites or desktop UI automation? (Not a request for additional experiments, just curious about your perspective.)\n4. Can you quantify the total compute cost of SoLS training (e.g., GPU-hours)? This would help assess the practical trade-offs between your method and API-heavy approaches like GPT-4o.\n5. How sensitive is SoLS to the choice of $\\epsilon$ in the clipped importance sampling update? Did you experiment with different values, and if so, how stable was training across them?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The method works well even with a relatively small model. This has clear implications for cost-effective deployment.\n\n2. The writing is clear, and the motivation and methodology are easy to follow. The authors also include a thoughtful discussion of limitations and failure cases, which strengthens the paper.\n3. The results are strong. SoLS beats all the baselines, including GPT-4o-based agents, and does so while being way faster at inference.\n\n\n\n**Weaknesses**\n\n1. The method is quite tailored to mobile app environments, where the action space is structured and discrete. It’s unclear how well SoLS would generalize to other domains with less constrained action formats (e.g., open-ended text generation or robotics)\n2. The success of SoLS seems to depend heavily on the quality of the initial supervised fine-tuning. If the SFT model is poor or underexposed to certain action types, SoLS may not recover well, since negative updates are largely suppressed.\n3. STR only stores successful timesteps, not full trajectories or multi-step episodes. This design improves stability and efficiency but might miss longer-term dependencies or subgoal information, which could be important for some tasks.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and comments. We address the main concerns below and clarify key points raised in the review.\n\n**Weaknesses:**\n\n1. > The method is quite tailored to mobile app environments, where the action space is structured and discrete. It’s unclear how well SoLS would generalize to other domains with less constrained action formats (e.g., open-ended text generation or robotics)\n\nSoLS is specifically designed for environments with structured, discrete action spaces, where actions follow a well-defined format. This makes SoLS well-suited not only to mobile UI environments but also potentially to other domains with similar constraints, such as web UI and robotics, where action spaces are often structured and bounded.\nIn contrast, open-ended domains like free-form text generation present different challenges. In such settings, policy regularisation remains critical to prevent reward hacking and maintain generation quality, as supported by the literature on PPO and GRPO. SoLS’s selective update mechanism may be less appropriate in these domains without significant adaptation.\nIn summary, while SoLS is not intended as a universal RL framework, we see it as a step toward scalable and stable RL with foundation models in high-cost, structured environments. We hope future work can extend its principles to less constrained settings.\n\n\n2. > The success of SoLS seems to depend heavily on the quality of the initial supervised fine-tuning. If the SFT model is poor or underexposed to certain action types, SoLS may not recover well, since negative updates are largely suppressed.\n\nWe agree with the reviewer that SoLS relies on a reasonably competent initial SFT model to be effective. This is a deliberate design trade-off: by applying conservative updates to negative-advantage samples, SoLS avoids harmful drift but limits its ability to recover from a poor starting point.\n\nThat said, this reliance is not unique to SoLS, but rather reflects a fundamental challenge in RL with large action spaces, especially in the LLM setting. Unlike traditional RL settings with small discrete action spaces (e.g., 5 actions), foundation models and mobile phones operate over several actions, making it infeasible to learn effectively through random exploration alone. Without a strong SFT phase to anchor the initial policy, any RL method would struggle to discover successful actions purely via exploration.\nIn our work, the combination of SoLS with STR helps address this by focusing updates on rare but informative successes, improving sample efficiency. Nevertheless, we acknowledge this as a limitation and an important direction for future work.\n\n\n3. > STR only stores successful timesteps, not full trajectories or multi-step episodes. This design improves stability and efficiency but might miss longer-term dependencies or subgoal information, which could be important for some tasks.\n\nThis is simply a design choice of the algorithm, because a timestep is self-sufficient. In practice, the data is kept in steps, as each step contains all necessary information, including current observation, past actions, and return, to compute the loss function of SoLS. It could be easily modified to store trajectories if they were needed, for example, to compute the advantage based on GAE.\n\n\n**Questions:**\n1. > Can you clarify how STR handles tasks with very few or no successful transitions early on? Is there a cold-start issue, and if so, how does the method mitigate it?\n\nIndeed, tasks with few or no early successes may face a cold-start issue. However, SoLS is designed to mitigate this in two key ways:\nOn-policy data is always included in every update batch, alongside sampled STR data (see Equation 2). This ensures that learning continues even for tasks without prior successful transitions, allowing the policy to gradually improve through exploration.\nThe initial SFT model provides a reasonable starting point, enabling early successes on a subset of tasks. These early successes help populate the STR buffer and, over time, generalise to harder or less familiar tasks via shared model weights and transferable behaviours.\n\nIn practice, we observe that STR grows organically throughout training. While some tasks may have slower start times, the combination of SFT bootstrapping and on-policy rollouts ensures that cold-start tasks are not excluded from learning.\n\n\n2. > Given that SoLS suppresses updates for negative-advantage actions, how does the model learn to handle rare or unseen actions (e.g., long-press or clipboard tasks)? Is there a risk of overfitting to early successes?\n\nSoLS’s conservative handling of negative-advantage updates does present challenges for learning rare or unseen actions, especially if they are not well-covered during SFT. However, we address this through several design choices:\nAggressive updates on positive advantages enable the model to rapidly reinforce successful rare actions, even if they occur only once during exploration. This allows the model to quickly adapt once such actions are discovered.\n\nSuccessful Transition Replay (STR) helps maintain diversity and avoids overfitting to early successes by storing and sampling successful steps on a per-task basis. This ensures the policy continues learning from newer and more varied experiences and prevents drowning out the more difficult tasks, mitigating overfitting to easier tasks.\n\nRegarding easy tasks, we observe that the value function quickly converges, and the advantage for those tasks tends toward zero. As a result, updates to the policy are minimal (since they are scaled by the advantage), preventing overfitting of already-solved behaviours. This stability allows SoLS to focus learning capacity on harder tasks where the advantage signal is still meaningful.  \n\n\n3. > Do you think SoLS could generalize to other structured control settings beyond mobile apps—such as form-filling on websites or desktop UI automation? (Not a request for additional experiments, just curious about your perspective.)\n\nYes, we believe SoLS could generalise well to other structured control settings beyond mobile apps, such as web form-filling, desktop UI automation, or even robotic task planning. Some current mobile app tasks, such as creating contacts or recipes, actually involve form-filling already. Generalisation to web-based form-filling or desktop UI should be reasonably seamless, as the domains are very similar. \n\nThese other settings share the same core challenges that SoLS is specifically designed to address: sparse rewards, large but structured action spaces, and possibly costly interactions. The separation of positive and negative updates, combined with STR, can offer similar benefits in those settings without requiring major algorithmic changes.\n\nWe appreciate the reviewer’s perspective and view this as a promising direction for future applications.\n\n\n4. > Can you quantify the total compute cost of SoLS training (e.g., GPU-hours)? This would help assess the practical trade-offs between your method and API-heavy approaches like GPT-4o.\n\nWe used 8 V100 GPUs, where training took approximately 5 days, consistent across RL methods. This could be significantly sped up if the experiments were performed with newer GPUs.\n\n\n5. > How sensitive is SoLS to the choice of epsilon in the clipped importance sampling update? Did you experiment with different values, and if so, how stable was training across them?\n\nWe did not experiment with the epsilon value due to computational constraints and the cost of running training and evaluations. We maintained the standard value of 0.2."}, {"review_id": "t1uBoWflWT", "reviewer": "Reviewer_Css9", "summary": "10373\n\nSummary:\n\nThis paper considers the use of RL to train agents for mobile application interaction. It recognizes two challenges: dealing with sparse rewards, and dealing with limited samples. It addresses these challenges with two innovations. The first is an alteration to PPO-like algorithm that favors the reinforcement of successful actions, while more strongly regularizing (preventing significant policy changes) for unsuccessful actions. [Note that trajectory have binary rewards, and so a successful actions is one that results in the trajectory having a reward of 1.] The authors claim that this helps prevent catastrophic forgetting during exploration. The second innovation is a variant of experience replay, which the authors call Successful Transition Replay (STR), which prioritizes successful state-action pairs from previous episodes.\n\nThe authors evaluate their method on a benchmark called AndroidWorld, obtaining an overall success rate (across three problem difficulties) of 51.3% (17% above the next best), significantly outperforming GPT-4o-based approaches  and other methods based on RL fine-tuning. Their method also achieved this with between a 5x and 60x computational speed-up.", "questions": "What evidence did you observe to support the claim that the bias towards successful actions reduces catastrophic forgetting?", "limitations": "None.", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths. The paper is clear and well-written. It clearly describes the problem it sets out to solve, explains the proposed method, and presents empirical results supporting the claims.\n\nWeakness: Although the paper claims that performance improves due to the bias towards successful actions preventing catastrophic forgetting, that claim is not well-supported by the experiments. It's also not clear to my why this would prevent catastrophic forgetting. However, if the paper removed this claim, it would still be showing improved performance on an existing benchmark.\n\nI note, however, that this paper is outside my area. I am not familiar with any of these algorithms (other than PPO) nor benchmarks. That is a statement about my expertise, not the choices of algorithms / benchmarks in the paper.", "was_revised": true, "final_justification": "I have read the author's response, and it has not changed my evaluation. I continue to recommend acceptance.", "rebuttal": "We thank the reviewer for their valuable feedback and comments. We address the main concerns below and clarify key points raised in the review.\n\n**Weaknesses / Questions:**\n\n> Although the paper claims that performance improves due to the bias towards successful actions preventing catastrophic forgetting, that claim is not well-supported by the experiments. It's also not clear to my why this would prevent catastrophic forgetting. However, if the paper removed this claim, it would still be showing improved performance on an existing benchmark.\nWhat evidence did you observe to support the claim that the bias towards successful actions reduces catastrophic forgetting?\n\nWe acknowledge that our claim regarding “catastrophic forgetting” could have been more clearly explained and better supported. The main evidence we intended to highlight is the comparison with A2C-STR, which does not apply conservative updates to negative-advantage actions. In practice, A2C-STR exhibited training instability and required significant tuning effort, whereas SoLS-STR did not and achieved a 60% relative improvement in final performance (51.3% vs 32.1%).\n\nThat said, we agree that “catastrophic forgetting” may not be the most precise term for this phenomenon. A more accurate framing would be that SoLS helps prevent performance degradation due to negative updates during RL fine-tuning. We will rephrase this claim accordingly in the next revision to better reflect the observed behaviour."}]

Author rebuttal:

Meta-review: This paper introduces SoLS, a novel off-policy reinforcement learning algorithm for mobile app control. The authors combine an asymmetric policy update that favors positive outcomes with a Successful Transition Replay (STR) buffer that stores successful steps. Reviewers were largely positive, noting the method's strong empirical results and practicality, leading to a consensus for acceptance after the authors addressed concerns about novelty, stability, and experimental clarity in their rebuttal.
Decision: Accept (poster)