DigiRL: Training In-The-Wild Device-Control Agents with Autonomous
  Reinforcement Learning
Authors: name: Hao Bai; name: Yifei Zhou; name: Mert Cemri; name: Jiayi Pan; name: Alane Suhr; name: Sergey Levine; name: Aviral Kumar
Published: 2024-06-14 17:49:55

Abstract: Training corpuses for vision language models (VLMs) typically lack sufficient
amounts of decision-centric data. This renders off-the-shelf VLMs sub-optimal
for decision-making tasks such as in-the-wild device control through graphical
user interfaces (GUIs). While training with static demonstrations has shown
some promise, we show that such methods fall short for controlling real GUIs
due to their failure to deal with real-world stochasticity and non-stationarity
not captured in static observational data. This paper introduces a novel
autonomous RL approach, called DigiRL, for training in-the-wild device control
agents through fine-tuning a pre-trained VLM in two stages: offline RL to
initialize the model, followed by offline-to-online RL. To do this, we build a
scalable and parallelizable Android learning environment equipped with a
VLM-based evaluator and develop a simple yet effective RL approach for learning
in this domain. Our approach runs advantage-weighted RL with advantage
estimators enhanced to account for stochasticity along with an automatic
curriculum for deriving maximal learning signal. We demonstrate the
effectiveness of DigiRL using the Android-in-the-Wild (AitW) dataset, where our
1.3B VLM trained with RL achieves a 49.5% absolute improvement -- from 17.7 to
67.2% success rate -- over supervised fine-tuning with static human
demonstration data. These results significantly surpass not only the prior best
agents, including AppAgent with GPT-4V (8.3% success rate) and the 17B CogAgent
trained with AitW data (38.5%), but also the prior best autonomous RL approach
based on filtered behavior cloning (57.8%), thereby establishing a new
state-of-the-art for digital agents for in-the-wild device control.