InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning
  and Reflection
Authors: name: Yuhang Liu; name: Pengxiang Li; name: Zishu Wei; name: Congkai Xie; name: Xueyu Hu; name: Xinchen Xu; name: Shengyu Zhang; name: Xiaotian Han; name: Hongxia Yang; name: Fei Wu
Published: 2025-01-08 15:45:21

Abstract: Graphical User Interface (GUI) Agents, powered by multimodal large language
models (MLLMs), have shown great potential for task automation on computing
devices such as computers and mobile phones. However, existing agents face
challenges in multi-step reasoning and reliance on textual annotations,
limiting their effectiveness. We introduce InfiGUIAgent, an MLLM-based
GUI Agent trained with a two-stage supervised fine-tuning pipeline. Stage 1
enhances fundamental skills such as GUI understanding and grounding, while
Stage 2 integrates hierarchical reasoning and expectation-reflection reasoning
skills using synthesized data to enable native reasoning abilities of the
agents. InfiGUIAgent achieves competitive performance on several GUI
benchmarks, highlighting the impact of native reasoning skills in enhancing GUI
interaction for automation tasks. Resources are available at
https://github.com/Reallm-Labs/InfiGUIAgent.