Lightweight Neural App Control
Authors: name: Filippos Christianos; name: Georgios Papoudakis; name: Thomas Coste; name: Jianye Hao; name: Jun Wang; name: Kun Shao
Published: 2024-10-23 13:57:00

Abstract: This paper introduces a novel mobile phone control architecture, termed ``app
agents", for efficient interactions and controls across various Android apps.
The proposed Lightweight Multi-modal App Control (LiMAC) takes as input a
textual goal and a sequence of past mobile observations, such as screenshots
and corresponding UI trees, to generate precise actions. To address the
computational constraints inherent to smartphones, within LiMAC, we introduce a
small Action Transformer (AcT) integrated with a fine-tuned vision-language
model (VLM) for real-time decision-making and task execution. We evaluate LiMAC
on two open-source mobile control datasets, demonstrating the superior
performance of our small-form-factor approach against fine-tuned versions of
open-source VLMs, such as Florence2 and Qwen2-VL. It also significantly
outperforms prompt engineering baselines utilising closed-source foundation
models like GPT-4o. More specifically, LiMAC increases the overall action
accuracy by up to 19% compared to fine-tuned VLMs, and up to 42% compared to
prompt-engineering baselines.