ColorAgent: Building A Robust, Personalized, and Interactive OS Agent
Authors: name: Ning Li, name: Qiqiang Lin, name: Zheng Wu, name: Xiaoyun Mo, name: Weiming Zhang, name: Yin Zhao, name: Xiangmou Qu, name: Jiamu Zhou, name: Jun Wang, name: Congmin Zheng, name: Yuanyi Song, name: Hongjiang Chen, name: Heyuan Huang, name: Jihong Wang, name: Jiaxin Yin, name: Jingwei Yu, name: Junwei Liao, name: Qiuying Peng, name: Xingyu Lou, name: Jun Wang, name: Weiwen Liu, name: Zhuosheng Zhang, name: Weinan Zhang

With the advancements in hardware, software, and large language model
technologies, the interaction between humans and operating systems has evolved
from the command-line interface to the rapidly emerging AI agent interactions.
Building an operating system (OS) agent capable of executing user instructions
and faithfully following user desires is becoming a reality. In this technical
report, we present ColorAgent, an OS agent designed to engage in long-horizon,
robust interactions with the environment while also enabling personalized and
proactive user interaction. To enable long-horizon interactions with the
environment, we enhance the model's capabilities through step-wise
reinforcement learning and self-evolving training, while also developing a
tailored multi-agent framework that ensures generality, consistency, and
robustness. In terms of user interaction, we explore personalized user intent
recognition and proactive engagement, positioning the OS agent not merely as an
automation tool but as a warm, collaborative partner. We evaluate ColorAgent on
the AndroidWorld and AndroidLab benchmarks, achieving success rates of 77.2%
and 50.7%, respectively, establishing a new state of the art. Nonetheless, we
note that current benchmarks are insufficient for a comprehensive evaluation of
OS agents and propose further exploring directions in future work, particularly
in the areas of evaluation paradigms, agent collaboration, and security. Our
code is available at https://github.com/MadeAgents/mobile-use.