GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on
  Mobile Devices
Authors: name: Quanfeng Lu; name: Wenqi Shao; name: Zitao Liu; name: Fanqing Meng; name: Boxuan Li; name: Botong Chen; name: Siyuan Huang; name: Kaipeng Zhang; name: Yu Qiao; name: Ping Luo
Published: 2024-06-12 17:44:26

Abstract: Smartphone users often navigate across multiple applications (apps) to
complete tasks such as sharing content between social media platforms.
Autonomous Graphical User Interface (GUI) navigation agents can enhance user
experience in communication, entertainment, and productivity by streamlining
workflows and reducing manual intervention. However, prior GUI agents often
trained with datasets comprising simple tasks that can be completed within a
single app, leading to poor performance in cross-app navigation. To address
this problem, we introduce GUI Odyssey, a comprehensive dataset for training
and evaluating cross-app navigation agents. GUI Odyssey consists of 7,735
episodes from 6 mobile devices, spanning 6 types of cross-app tasks, 201 apps,
and 1.4K app combos. Leveraging GUI Odyssey, we developed OdysseyAgent, a
multimodal cross-app navigation agent by fine-tuning the Qwen-VL model with a
history resampling module. Extensive experiments demonstrate OdysseyAgent's
superior accuracy compared to existing models. For instance, OdysseyAgent
surpasses fine-tuned Qwen-VL and zero-shot GPT-4V by 1.44\% and 55.49\%
in-domain accuracy, and 2.29\% and 48.14\% out-of-domain accuracy on average.
The dataset and code will be released in
https://github.com/OpenGVLab/GUI-Odyssey.