ICML 2025

CRADLE: Empowering Foundation Agents
Towards General Computer Control
Weihao Tan 1 * Wentao Zhang 1 * Xinrun Xu 2 * Haochong Xia 1 † Ziluo Ding 3 † Boyu Li 3 † Bohan Zhou 4 †
Junpeng Yue 4 † Jiechuan Jiang 4 † Yewen Li 1 † Ruyi An 1 † Molei Qin 1 † Chuqiao Zong 1 † Longtao Zheng 1 †
Yujie Wu 5 † Xiaoqiang Chai 5 † Yifei Bi 3 Tianbao Xie 6 Pengjie Gu 1 Xiyun Li 3 Ceyao Zhang 7 Long Tian 5
Chaojie Wang 5 Xinrun Wang 1 ‡ Börje F. Karlsson 3 ‡ † Bo An 1 5 Shuicheng Yan 8 5 Zongqing Lu 4 3
Everyday
Software
Professional
Software
Digital
Games
Memory
Information 
Gathering
Self-Reflection
Task Inference
Skill Curation
Action
Planning
Figure 1: The CRADLE framework empowers nascent foundation models to perform complex computer tasks via the same uniﬁed
interface humans use, i.e., screenshots as input and keyboard & mouse operations as output.
Abstract
Despite the success in speciﬁc scenarios, exist-
ing foundation agents still struggle to generalize
across various virtual scenarios, mainly due to the
dramatically different encapsulations of environ-
ments with manually designed observation and
action spaces. To handle this issue, we propose
the General Computer Control (GCC) setting to
restrict foundation agents to interact with software
through the most uniﬁed and standardized inter-
face, i.e., using screenshots as input and keyboard
and mouse actions as output. Then we introduce
CRADLE, a modular and ﬂexible LMM-powered
framework, as a preliminary attempt towards
GCC. Enhanced by six key modules: Information
Gathering, Self-Reﬂection, Task Inference, Skill
*Equal contribution 1Nanyang Technological University, Sin-
gapore 2Institute of Software, Chinese Academy of Sciences
3Beijing Academy of Artiﬁcial Intelligence 4Peking University
5Skywork AI 6The University of Hong Kong 7The Chinese Univer-
sity of Hong Kong, Shenzhen 8National University of Singapore.
Correspondence to: Weihao Tan <weihao001@ntu.edu.sg>, Bo
An <boan@ntu.edu.sg>, Shuicheng Yan <yansc@nus.edu.sg>,
Zo...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000054. Conference bucket: ICML.