UFO: A UI-Focused Agent for Windows OS Interaction
Authors: name: Chaoyun Zhang; name: Liqun Li; name: Shilin He; name: Xu Zhang; name: Bo Qiao; name: Si Qin; name: Minghua Ma; name: Yu Kang; name: Qingwei Lin; name: Saravan Rajmohan; name: Dongmei Zhang; name: Qi Zhang
Published: 2024-02-08 15:40:35

Abstract: We introduce UFO, an innovative UI-Focused agent to fulfill user requests
tailored to applications on Windows OS, harnessing the capabilities of
GPT-Vision. UFO employs a dual-agent framework to meticulously observe and
analyze the graphical user interface (GUI) and control information of Windows
applications. This enables the agent to seamlessly navigate and operate within
individual applications and across them to fulfill user requests, even when
spanning multiple applications. The framework incorporates a control
interaction module, facilitating action grounding without human intervention
and enabling fully automated execution. Consequently, UFO transforms arduous
and time-consuming processes into simple tasks achievable solely through
natural language commands. We conducted testing of UFO across 9 popular Windows
applications, encompassing a variety of scenarios reflective of users' daily
usage. The results, derived from both quantitative metrics and real-case
studies, underscore the superior effectiveness of UFO in fulfilling user
requests. To the best of our knowledge, UFO stands as the first UI agent
specifically tailored for task completion within the Windows OS environment.
The open-source code for UFO is available on https://github.com/microsoft/UFO.