Cradle: Empowering Foundation Agents towards General Computer Control
Authors: Weihao Tan
Wentao Zhang
Xinrun Xu
Haochong Xia
Ziluo Ding
Boyu Li
Bohan Zhou
Junpeng Yue
Jiechuan Jiang
Yewen Li
Ruyi An
Molei Qin
Chuqiao Zong
Longtao Zheng
YuJie Wu
Xiaoqiang Chai
Yifei Bi
Tianbao Xie
Pengjie Gu
Xiyun Li
Ceyao Zhang
Long Tian
Chaojie Wang
Xinrun Wang
Börje F. Karlsson
Bo An
Shuicheng YAN
Zongqing Lu
Venue: ICML 2025 poster (2025)
Keywords: Foundation Agents
Large Multimodal Models
Decision-making
General Computer Control
Primary area: applications
TL;DR: A framework that can play RDR2, Stardew Valley, Cities: Skylines and Dealer's Life2 and various software with unified interface

Abstract: Despite their success in specific scenarios, existing foundation agents still struggle to generalize across various virtual scenarios, mainly due to the dramatically different encapsulations of environments with manually designed observation and action spaces. To handle this issue, we propose the General Computer Control (GCC) setting to restrict foundation agents to interact with software through the most unified and standardized interface, i.e., using screenshots as input and keyboard and mouse actions as output. We introduce Cradle, a modular and flexible LMM-powered framework, as a preliminary attempt towards GCC. Enhanced by six key modules, Information Gathering, Self-Reflection, Task Inference, Skill Curation, Action Planning, and Memory, Cradle is able to understand input screenshots and output executable code for low-level keyboard and mouse control after high-level planning and information retrieval, so that Cradle can interact with any software and complete long-horizon complex tasks without relying on any built-in APIs. Experimental results show that Cradle exhibits remarkable generalizability and impressive performance across four previously unexplored commercial video games (Red Dead Redemption 2, Cities:Skylines, Stardew Valley and Dealer's Life 2), five software applications (Chrome, Outlook, Feishu, Meitu and CapCut), and a comprehensive benchmark, OSWorld. With a unified interface to interact with any software, Cradle greatly extends the reach of foundation agents thus paving the way for generalist agents.

CRADLE: Empowering Foundation Agents Towards General Computer Control
Weihao Tan 1' Wentao Zhang',
Xinrun Xu
Haochong Xia If Ziluo Ding 31
Boyu Li
Bohan Zhou 41
Junpeng Yue Jiechuan Jiang" 4f Yewen Li Ruyi An If Molei Qin 1f Chuqiao Zong Longtao Zheng'
Yujie Wu 5i Xiaoqiang Chai si Yifei Bi Tianbao Xie Pengjie Gu
Xiyun Li
Ceyao Zhang' Long Tian' Shuicheng 85 Zongqing Lu"
Chaojic Wang Xinrun Wang !
Börje F. Karlsson 321
Bo An
Professional
Digital
Self-Reflettion
Information Gathering
O
O
Memory
Skill Planning
Figure 1: The CRADLE framework empowers foundation models to perform complex computer tasks via the unified interface humans use, i.e. screenshots input and keyboard mouse operations as output.
Abstract
Curation, Action Planning, and Memory, CRA- DLE is able understand input screenshots and output executable code for low-level keyboard and mouse control after high-level planning, so that CRADLE can interact with any software and com- plete long long-horizon complex tasks without relying on any built-in APIs. Experimental results show that CRADLE exhibits remarkable generalizabil- ity and impressive performance across four previ- ously unexplored commercial video games (Red Dead Redemption 2, Cities: Skylines, Stardew Valley and Dealer's Life 2), five software applica- tions (Chrome, Outlook, Feishu, Meitu and Cap- Cut), and comprehensive benchmark, OSWorld. With unified interface to interact with any soft- ware, CRADLE greatly extends the reach of foun- dation agents thus paving the way for generalist agents, Video demos and code can be found at tttp://baaiaggente wataiaagtsnssttgbbii.... github.io/Cralle.
Despite the success in specific scenarios, exist- ing foundation agents still struggle to generalize across various virtual scenarios, mainly due to the dramatically different encapsulations of environ- ments with manually designed observation and action spaces. To handle this issue, we propose the General Computer Control (GCC) setting restrict foundation agents interact with software through the most unified and standardized inter- face, i.e. using screenshots as input and keyboard and mouse actions as output. Then we introduce CRADLE, a modular and flexible LMM LMM-powered framework, as a preliminary attempt towards GCC. Enhanced by six key modules: Information Gathering, Self-Reffee ReffeReffectinn Task Inference, Skill
'qqual contribution 'Nayyang Technological University, Sin- gapore "Institute of Software, Chinese Academy of Sciences BBijjing Academy of Artificial Intelligence 'Peking University "Skywork AI "The University of Hong Kong "The Chinese Univer- sity of Hong Kong, Shenzhen NNational University of Singapore. Correspondence to: Weihao Tan eweihac001 ntu intu.uuu.sg. edu.sg>, Bo An <boan @ ntu.edu.sg2. Shuicheng Yan <yanse @ meenuuuuuuus.. Zongging Lu <zongging. Iu @we@ppkueeeeee pku.edu.com edu.cn..
1. Introduction
Artificial General Intelligence (AGI) has long been north- star goal for the Al community (Morris et al., 2023). The recent success of foundation agents, i.e., agents empow- ered by large multimodal models (LMMs) and advanced tools, in various environments, e.g. web browsing (Zhou
Proceedings of the ad International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025, Copyright 2025 by the author(s).

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Path towards AGI
R33 BEAN
ST&RTEW +AMM
General Computer Control
Applications
Games
GUPLES
PEAERS
LIFE
OSWorld
Figure 2: Taxonomy of GCC and the games and software investigated in this work.
et al., 2023; Deng et al., 2023; Gur et al., 2023; Zheng et al., 2024b;a; Hc et al., 2024), operating mobile applica- tions (Yang et al., 2023b; Wang et al., 2024b) and desktop software (Zhang et al., 2024; Wu et al., 2024), crafting and exploration in Minecraft (Wang et al., 2023b; 2024a; 2023a), and some robotics scenarios (Huang et al., 2022; Brohan et al., 2023b; Driess et al., 2023; Brohan et al., 2023a), have shown promise. However, current foundation agents still struggle to generalize across different scenarios, primarily due to the dramatic differences in the encapsulation of en- vironments with human-designed observation and action space. Therefore, developing foundation agents applicable to various environments remains extremely challenging.
the speed of the mouse movement; iii) long-horizontal rea- soning due to the partial observability of complex GCC tasks, which also leads to the demand for long-term mem- ory to maintain past useful experiences; and iv) efficient exploration to discover better strategies and solutions au- tonomously, i.e., self-improving, which can allow agents to generalize across the various tasks in the digital world.
As shown in Figure 1, we introduce CRADLE,, a novel mod- ular LMM-powered framework towards GCC. CRADLE consists of six key modules: 1) information gathering, to extract the relevant information from multimodal observa- tions; 2) self-reflection, to rethink past experiences about whether the actions and tasks are successfully completed and reasons for possible failures; 3) task inference, de- termine whether to continue current tasks or propose new task given the current situation; 4) skill curation, to gener- ate, update, and retrieve useful skills for the current task; 5) action planning, to generate specific executable operations for keyboard and mouse control via skills; and 6) memory, for storage, summary, and retrieval of past experiences.
Computers, as the most important and universal interface that connects humans and the increasing digital world, pro- vide countless rich software, including applications and real- istic video games for agents to interact with, while avoiding the challenges of robots in reality, such as hardware re- quirements, constraints of practicability, and possible catas- trophic failures (Raad et al., 2024). Mastering these virtual environments is promising path for foundation agents to achieve generalizability. Therefore, we propose the General Computer Control (GCC) setting!.
As illustrated in Figure 2, tasks in GCC can be broadly di- vided into two categories: video game playing and software application manipulation. Compared to Software applica- tions, video games provide more comprehensive and chal- lenging testbeds to evaluate and improve agents' various abilities including but not limited to classical UI manip- ulation and embodied control. In this work, we conduct extensive experiments to demonstrate the generalizability of CRADLE in such complex environments, while also mas- tering diverse everyday software applications in distinct domains. We managed to prove that commercial software is out-of- box testbeds under our framework. The four selected representative games are: epic AAA 3D role-playing game, RDR2, 2D pixel-art farming simulation game, Stardew Valley pawn shop simulation game, Dealer's Life and 3D, top-down view, city-building game, Cities: Skylines.
Building foundation agents that can master ANY com- puter task via the universal human-style interface by receiving input from screens and audio and outputting keybward and mouse actions.
There are many challenges to achieving GCC: i) good align- ment across multi-n modalities for better understanding and docision-making ii) precise control of keyboard and mouse to interact with the computer, which has large hybrid ac- tion space, including not only which key to press and where the mouse to move, but also the duration of the press and
Thii setting can be seamlessly extended to other digital de- vices, Le., mobile phones, game controllers, and virtual reality headsets with standard input and output.
2

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Workly Stopp Mera Menaggmmen sermaae
Foloo
Scl
Comteren Controse
Start Video
Outlaw from the West
File
funal
Sictiss
Popus
Past
Twinee
Opan Cloud
Htone
Projeet Dutch
Contict
Mesuage
Page
Go Pruflle
Build
Craate Appentmmnt
Towa
LIGES
Search for
Supplies
Change
Mede
Add
Feishu
Chrome
Send
RED DEAD REDEMFTION
Go
Barn
Framic
New E-itail
Farm Clean Up
SEAREE VALLET
Scarch Barn
Collage
M
Junk Folder
Cullvating
Hoyee
Load
Add Sticker
Meitu
X
Outlook
Pereos
Haply to
Food Zaget E-nual
Catont
Stoppong
Follow Javicr
Saurch
Appls Filis
CapCut
Forwarling
Scop
Bay
Serply
Resum
Kvop
Mabi Preiees
Close
Upen- ended
Enter, Pursued by Memory
Chop
Tomsitum
Adl
Figure 3: Overview of all game tasks (left) in RDR2, Stardew Valley, Cities: Skylines, and Dealer's Life 2 and application tasks (right) in
Chrome, Outlook. CapCut, Meitu, and Feishu.
The target set of diverse software applications for evalu- ation includes: Chrome, Outlook, CapCut, Meitu, and Feishu, as well as one comprehensive software benchmark, OSWorld (Xie et al., 2024). We provide brief introduction these games Appendix
2. Related Work
Agents for Software Applications. While previous LLM- based web agents (Deng et al., 2023; Zhou et al., 2023; Gur et al., 2023; Zheng et al., 2024b) show some promising results in effectively interacting with content on webpages, they usually use raw HTML code and DOM tree as input and interact with the available element IDs, ignoring the rich visual patterns with key information, like icons, images, and spatial relations. Multimodal web agents (Hong et al., 2023; Furuta al., 2023; Yan et al., 2023; He al., 2024; Zheng al., 2024a) and mobile app agents (Yang et al., 2023b; Wang et al., 2024b) have also been explored. Though using screenshots as input, they still need to use built-in APIs to get the available interactive element IDs to execute corresponding actions. Several recent works (Cheng et al., 2024; Wu et al., 2024; Kapoor al., 2024) aim apply web agents more applications by using keyboard and mouse for control. However, they primarily focus the static websites and lack the generalizability to other domains.
As shown in Figure 3, for each game and software applica- tion, representative tasks are designed to measure the various abilities of the agent comprehensively. Experimental results show that CRADLE exhibits remarkable generalization abil- ity and impressive performance across the four previously unexplored commercial video games, the five target soft- ware applications. and the comprehensive contemporaneous OSWorld benchmark. To our best knowledge, CRADLE is the first to enable LMM-based agents to follow the main storyline and complete one-hour-long real missions in complex AAA game, RDR2. CRADLE also manages to cre- ate a city with nearly a thousand people in Cities: Skylines, farm and harvest parsnips Stardew Valley, trade and bar- gain with a maximal weekly total profit of 87% in Dealer's Life 2. Besides, CRADLE can not only operate daily soft- ware, like Chrome and Outlook, but also edit images and videos using Meitu and CapCut, and perform office tasks in Feishu. The success of CRADLE demonstrates the fea- sibility of challenging GCC setting. Being able to interact with various software unified manner, CRADLE greatly extends the reach of agents by making it easy to convert any software, especially complex games, into benchmarks and testbeds to evaluate agents' various abilities and facil- itate further data collection, paving the way for generalist agents. We hope CRADLE can accelerate the development of more powerful foundation agents, thereby advancing the path towards AGI.
Agents for Video Games. Several attempts try to de- velop foundation agents for complex video games, such as Minecraft (Wang et al., 2023b;a; 2024a), Starcraft (Ma et al., 2023) and Civilization-like game (Qi al., 2024) with textual observations obtained from internal APIs and pre-defined semantic actions. Although JARVIS-1 (Wang et al., 2023a) claims to interact with the environment in human-like manner with the screenshots as input and mouse and keyboard for control, its action space is predefined as a hybrid space composed keyboard, mouse, and API. The game-specific observation and action spaces prohibit the generalization of them other novel games. SIMA(Raad
3

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Self-Reflection
Task Inference
Skill Curation
Key screenshots
Reflect what happened
Select next task
Given task, generate skills
skill 1(a, b) Add skill skill_2(c, d)
LaLs action succeedsifails The error is due 2. The task succeeds/falls.
Given reflection results and current observattios, next task is:
LMM
LMM
[skill 1, 1kkkl1 2, skill a,skill_b, b, skill skill_c,...
The reason is... ...
Retrieve
Extract important visual and text information
Episodic Memory
Procedural Memory
Text info
Visual info
Given retrieeed skills) generate next action
LMM
Navigation
Ll elements
Notiftaation
Layout
3. Instruccion
Imagery
Memory
Consens
Anireniions
The action to execute is: skill 1(a0011, b=0.5)
LMM
Executor
Video (Sequences of screenshols)
Information Gathering
Action Planning
Figure 4: An overview of the CRADLE framework, demonstrating the input, output and functionality of cach module. CRADLE takes video from the computer sereen as input and outputs computer keyboard and mouse control determined through inner reasoning.
KEWTTER
STARLEW VALLEY
CC
RURIER
Hluld
IM
View your storad waupors
Ncmms Map
House Wizrres
Tel Wasser
Move Down
Move Right
Quastion
def view_s toreed ())
def access_rappis
def place_residentia Lzone/xt, y2):
Press "tab" to view your stored weapons, omn
one
Press "M" to access map..
Click residential zone icon and place.
jo "Lab")
kry.ess (N')
i Nove mouse to the icon and click.
(788, 958)
def talk onn
walker():
def nove_uptduration
env.nouse_ctlce click,
"left button")
Press "right mouse buttonn talk
Press "W" move up.
Place the res residential zone.
tennessee walker.
jo_env.nouse__eve nove y1)
to hew.kkey__ressste 'W', duration)
in wwl.evv.nuus,,eomm hold! 'left buttonn))
nouse click( rright button')
jo_env.nouse_rewe nxveess y21
Figure 5: Examples for skill generation according to in-game guidance in RDR2 (left), in-game manual in Stardew Valley (middle), self- exploraato in Cities: Skylines (right). Code and comments shown in brevity.
et al., 2024) trained embodied agents to complete 10-second- long basic tasks over ten 3D video games, and the results are promising to be scaled up.
knowledge of computer UI and control, is requirement. In this work, we leverage GPT-4o (OpenAI, 2024b) as the framework's 's backbone model.
Due to the space limitation, we provide a detailed discussion of the related work in Appendix B.
3.1. Environment IO
Observation and Action Space. CRADLE only takes a video clip, recording the execution of the last action, as input and outputs keyboard and mouse operations to in- teract with environments. The observation space is made up of complete screen videos with different lengths. For the action space, it includes all possible keyboard and mouse operations, including key_press, key_hold, key_ pey_release, mouse move, and theel_scroll where keys include both keyhoard keys and mouse buttons. These operations can be combined in various ways to form combos and shortcuts, execute rapid key sequences, or co- ordinate timings. Python code is applied to simulate these
3. The CRADLE Framework
To pursue GCC, we propose CRADLE, illustrated in Fig- ure 4, a modular and flexible LMM-powered framework that can properly handle the challenges GCC presents. The framework should have the ability to understand and in- terpret computer screens and dynamic changes between consecutive frames from arbitrary software and be able to generate reasonable computer control actions for precise execution. This suggests that multimodal model with pow- erful vision and reasoning capabilities, in addition to rich

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Information Gathering
Self-Reflection
Task Inference
Skill Curation
Action Planning
RED DEAD RE REPPTTIO
Connec bwdd conaeee
Waspon
^
Guidance: Hold [TAB] to show
Weapon Wheel.
Toolbar:
Axxxe 2. Hoe (Selected)
Parsnip Seeds (15)
Error Message: Cannot be built on water
Last Action: take_ cover()
Last Action: tool()
Last Action:
Previous Task: Press [Q] to talse cover.
Till the soll using the hoo,
place sater p__poopiing inging wttp_pppnnggtttttiiooooooo statientenen,,, y-)no)
The last action was executeed
Provious Task: use
till soil.
Previous lask: place
pumping station near the river.
successfully, and the completed
The last action was executed successfully.
since new guidance appears.
The hoe is actively being used aiming toward the soil and the soil becomes darker. Therefore, the task completed.
The action not executed successfully. As the water pumping station cannot be built
Hold [TAB] to show the Wespon Whoel
the water shown in Therefore, the task is not completed.
message.
The new task is to plant the parsnip scods the tilled
Generated Skill:
The current task to continue the previous task.
def show, wwwpwweeonnnooooo :
1o_ent eo_env_Kee key_ hold( 'tab')
Retrieved Skills:
Retrieved Skills:
left',
Retrieved Skills:
['ain', 'followw' 'turn', ''nove_fee for
weriight tooll,
Cpplace postertttrr pwwping stationn 'place
ster_dratinpteen "place_ante__pper
ward', ''show weapoow_eeaoo_aaee wheel"
Selects paranip seeds free the toolber.
select_ (key- (key='6') '6')
Execnte
show weapon w)ap))_wheel()
Exeeute
do actioon)
Plants parents sneds tilled coll.
Execute
pplace_water wwwp__aater__upinnggo
(X=180, y-568)
RED DEAD
Contrraction caec 3530
Figure 6: Illustrative examples of CRADLE'S complete workflow in RDR2 (left), Stardew Valley (middle) and Cities: Skylines (right). Prompts shown partially for brevity.
operations and encapsulate them into an io_env class.
which encapsulate lower-level keyboard and mouse control. Similar to how humans improve while playing, these skills can be developed from scratch according to in-game tuto- rials and guidance, game manuals and settings, or through explorattot as the game progresses. These skills can also be pre-defined or composited to solve more complex tasks. An action usually consists of a single or multiple skills instantiated with any necessary parametric aspects, such as duration and position. An Executor will be triggered to map these semantic actions to the OS-Ievel keyboard and mouse commands to interact with the environment.
Information Gathering. Provided with a video clip as in- put, is critical for CRADLE to capture and extract all use- ful visual and textual information to understand the recent situation and perform further reasoning. Visual informa- tion includes layout, imagery, animations, and UI elements which pose high spatial perception and visual understand- ing requirements for LMM models. Moreover, we depend on their OCR capabilities to extract textual information in images, which usually includes content (headings and para- graphs), navigation labels (menus and links), notifications, and instructions convey messages and guide users. For each environment, different tools, such as template match- ing (Brunelli, 2009), Grounding DINO (Liu et al., 2023), and SAM (Kirillov et al., 2023), can be applied to enhance LMMs' abilities to provide additional grounding for object detection and localization.
3.2. Memory
CRADLE stores and maintains all the useful information from the environment or outputted by each module through memory mechanism, consisting of episodic memory and procedural memory.
Skill and Action Generation As shown in Figure 5, to bridge the gap between semantic actions generated by LMMs and OS-level executable actions, CRADLE uses LMMs to generate code functions as semantic-level skills,
Episodic memory is used to maintain current and past experiences, including key screenshots from each video observation, and everything useful outputted by L.MSs and advanced tools, e.g., textual and visual infor-
Episodic Memory.
5

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
mation, actions, tasks, and reasoning from each module. To facilitate retrieval and storage, periodical summariza- tion is conducted to abstract recently added multimodal information into long-term summaries. The incorporation of episodic memory enables CRADLEE to effectively retain crucial information over extended periods.
4. Empirical Studies
In this section, we present empirical results of deploying CRADLE across various challenging environments with GCC setting to demonstrate its comprehensive capabilities.
Experimental Settings. If not specifically mentioned, all experiments are conducted in five runs under maximum step limit, using OpenAl's model, gpt-4o-2024-05-13 (Ope- nAI, 2024b). For each video game, we hired five human players, who never played the corresponding game before, to do the evaluation. Before they start the experiments, they will read the prompts used by CRADLE agents for fair comparison, comparison. Every player played the task once. Similar to SIMA (Raad et al., 2024), we apply human evaluation to all tasks across software and games, except for OSWorld (Xie et al., 2024), which provides automatic evaluation scripts. Due to the dynamism of the RDR2 and Stardew Valley and the LMM inference and communication latency, we need to pause those game environments while waiting for backbone model's 's responses. Other environments execute continuously. All software and games can be run on reg- ular Windows 10 machines, except for RDR2, which is tested on a machine with an NVIDIA RTX-4090 GPU. We provide estimated experimental cost of the time and API usage in Appendix C, implementation details, environment introduction, task description, case study and prompts in Appendices E to K.
Procedural Memory. This memory specific to storing and retrieving skills in code form, which can be learned from seratch as shown in Figure 5, or pre-defined in procedural memory. Skills can be added, updated, or composed to the procedural memory in the skill curation module. Same as Voyager (Wang et al., 2024a), skills are retrieved according to the similarities between their corresponding embedding and task description.
3.3. Reasoning
Based on the extracted information from observations and memory, CRADLE conducts high-level reasoning and then makes the decision. This process is analogous to "reflect on the past, summarize the present, and plan for the future", which is broken down into the following modules.
Self- RefleRefeet The reflection module initially evaluates whether the last executed action was successfully carried out and whether the task was completed. Sequential key screenshots from the last video observation, along with the previous context for action planning and task inference are fed to the LMM for reasoning. Additionally, we also request the LMM to provide an analysis of any failure. This valu- able information enables CRADLE to remedy inappropriate decisions or less-than-ideal actions. Furthermore, reflection can also be leveraged to inform re-planning of the task and bring the agent closer to target task completion, better under- stand the factors that led previous successes, or suggest how to update or improve specific skills.
Task Introduction. As shown in Figure 7 and 8, for RDR2, we mainly focus on evaluating agents on the first two com- plete missions of the main storyline in Chapter I, which can be divided into 13 tasks according to the in-game check- points, including but not limited to navigation, NPC in- teraction, inventory management, house exploration, and combat. It usually takes human player about hour to complete these missions. Few previous studies tackle such long- dur-tinr tasks and rich semantic environments. It is an ideal scenario to emulate a novice player learning to play the game from scratch according to the rich in-game tutorials and hints. For Stardew Valley, we propose three essential tasks at the stage of the game, i.e., Farm Clearup: Clear the obstacles on the farm, such as weeds, stones, and trees, much as possible to prepare for farming: 2) Cultivation: Plant the parsnip seed, water every day and harvest at least one mutual parsnip: 3) Shopping: Go to the general store in the town, which is out of the scope of the current map, to buy more seeds and return home. For Dealer's Life, the agent tasked with managing a pawn shop for a week, appraising item values and haggling with the customers to secure deals. For Cities: Skylines, the task is to build reasonable city ending in as much population as possible, with the initial starting funds of C70,000, and basic road, water and electricity facilities. Moreover, we define five representative domain-specific tasks for each of the five
Task Inference. After reflecting on the outcome of the last executed action, CRADLE needs to analyze the current situ- ation to infer the most suitable task for the current moment. We let LMMs determine the highest priority task to perform and when to stop an ongoing task and start a new one.
Skill Curation. As the task is specified, CRADLE needs to prepare the tactics to accomplish it, by retrieving useful skills from the procedural memory, updating existing skills, or generating new ones. The new skill will be stored in the procedural memory for future utilization.
Action Planning. CRADLE needs to select the appropriate skills from the curated skill set and instantiate these skills into sequence of executable actions by specifying any necessary parametric aspects (e.g., duration, position, and target) according to the current task and history informa- tion. The generated action is then fed to the Executor for interaction with the environment.
6

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
RED DEAD
Oover,
Weason Whot.
Pick up Tamr, Ext homm Opon Cubinct,
10000-1
Cumulative Staps
8000
Skow Horso Down.
His Horso Take Cower
8000
CIRTD Crosch
Arira Shat Waives, House
Put Annay Weason
Edleet Nomm inseet Stored Weappree
Camn Harse, Lead Hors
2000
MABSIONT:QUTLAA FROM THE WEST
Seel
MISSIONZ, ENTER, PURSUED MEMORY
Skills
0-
Follow Dutthh
Co Town
Hitch Horse
Protect Dutch
Search Supplles to Barn Search Dann
Lead Horse
Follow Javees
Smarch John
Keep Wolves aluay
KIt Wolves
Return to Camp
Figure 7: Demonstration of cumulative steps CRADLE takes to complete 13 sequential tasks in the two main storyline missions, starting from the beginning of the game. If task fails, CRADLE can select the 'retry checkpoint" option retry the task. Skills generated during the task completion are also illustrated in the figure. We only provide key skills for brevity. Error bars represent the standard deviation of steps needed to complete each task separately.
Figure 8: First row shows farm clearup, cultivation and shopping in Stardew Valley and haggling and dealing in Dealer's Life 2. Second row shows road construction, water pipe laying, wind turbine building, zoning, and the resulting city in Cities: Skylines.
Software Applications in our diverse target set.
ends up circling aimlessly around the house. Moreover, the room contains numerous interactive items unrelated to the task, resulting in much more steps for the agent to complete the task. Overall, CRADLE requires approximately 8,000 steps to complete both missions, taking around 98 minutes of in- in-game time, compared to the average of 67 minutes for human players. It is the first time for LMM-ppoered AI agents to exhibit comparable performance in AAA games.
4.1. Performance across Environments
Red Dead Red Redemption 2. Figure 7 shows that CRA- DLE can efficiently complete simple navigation tasks with a few steps like following an NPC or going to specific lo- cations on the ground (e.g., Follow Dutch, Go to Town and Go to Barn). Another following task, Follow Javier, and the searching task, Search John, are dangerous for the rugged and winding path up to the snow mountain with cliffs. Note that CRADLE is able to retry the checkpoint automatically according to the game guidance if the task fails. Therefore, CRADLE takes more steps for retrying the task in these dangerous areas. In addition, CRADLE spends about one- fourth of the total steps in the task of Protect Dutch, which is a long- horizooalto task with nighttime combat. Many key skills are generated in this task for weapon management and shooting movement. The visibility is poor due to the snow falling in the dark, preventing GPT-4o from accurately recognizing and locating enemies objects and precisely timing decisions, even equipped with Grounding DINO as an additional detection tool. More times of retry, combined with the need for frequent interactions during combat and the long horizon of the task, lead to this task requiring a large number of steps to complete. The success rate of the combat has significantly improved during the day with much fewer steps for completion, shown by tasks like Keep Wolves away. Additionally, indoor tasks like Search for Supplies are also challenging due to GPT4-o's limited spatial per- ception, which finds it difficult to locate target objects and
Stardew Valley. As shown in Table 1, we surprisingly find that GPT-4o struggles with accurately recognizing and locating objects near the player in this pixel-art game. This leads to difficulties for the agent to interact with objects or people, as it requires the player to stand precisely in front of them in the grid (e.g.. when entering doors, using a pickaxe to break stones). It explains the inefficiency in the farming task though the agent manages to clear up most of the obstacles in front of the house within 100 steps and poor performance in the shopping task. On the other hand, relying on episodic summarization and task inference, CRADLE manages to obtain the parsnip by watering the seed for four days and harvesting. Given GPT-4's limited visual capabilities in this game, there is still room for improvement in narrowing the gap between CRADLE and human players.
Dealer's Life 2. Table I shows that CRADLE demon- strates robust performance and efficient profit-making on the Weekly Shop Management task, successfully finalizing 93.6% of potential transactions, with an average of 2 ne- gotiation rounds per customer, and generally aiming for a profit rate of over 50% at the initial offer. It consistently generates profit across all runs, maintaining total profit
7

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Chrome
Outlook
CapCur
Meita
Feishu
Success Rate (%)
80
60-
40-
20-
0-
Download Paper
Post Twitter
Open Closed Page
Protile
Change Mode Send New E-mai
Empty lank Fulder
Raply Person
Find Target E-mail
Scttpp Fork Forvanding anding
Crost Media Project
Add Tranctionn
Crop by Tinestung Add Sticker
Crop by Centnt Apply Filter
Cutuut Add Sticke Create College
Create Appuintmen Message Contact
Add Freme
Send File
Set Statos Start Video Conference
Figure 9: CRADLE'S performance in software applications. Each task is run for 5 trials.
Table 1: CRADLE'S and human players' performance in Stardew Valley, Dealer's ' Life 2 and Cites: Skylines with each trial at most 100, 500, 1000 steps respectively. 1/5 indicates one successful run out of five runs + y indicate mean and standard deviation.
new residents from moving in. The issue also arises from GPT-4o's limited visual understanding, making it difficult to accurately recognize which areas are already covered by the water pipes. We empirically observed that these mistakes usually could be fixed within three unit operations (building or removing a road/facility/a place of zones is counted as one unit operation). Then cities built by CRADLE can even- tually reach a population of more than one thousand. We provide a detailed case study in Appendix I.5.2. Overall, as shown in Table 1, without the manual fixes, CRADLE still beats human players even though it suffers from local water storage. Human players typically pay insufficient attention to budget management and tend to allocate a disproportion- ate amount of funds to the construction of wind turbines for electricity, resulting in limited road construction and residential areas to attract residents.
Stardew Valley
Task
CRADLE
Human
Farm Clearup Grids Cultivation
14.8 + 5.0 1/5
35.2 + 14.5
5/5
Shopping
1/5
6/5
Dealer's Life 2
Metrics
CRADLE
Human
Avg. Haggling Count
1.95 = 0.43
1.63 + 0.53
Turnover Rate (%)
93.6 1 6,9
68.4 1: 22.2
Item Profit Rate (%)
37.8 + 19.1
21.1 1 13.6
Total Profit Rate (%)
39.6 +: 27.3
17.3 + 15.1
Cities: Skylines
Software Applications. Figure 9 shows CRADLE'S per- formance across tasks on five applications. Multiple tasks remain challenging. Even with a well-known GUI, like Chrome and Outlook, GPT-40 still cannot recognize spe- cific UI items to interact with and also struggles with visual context. For example, forgetting to press the Save button in an open dialog, or not distinguishing between a nearby enabled button vs. distant and disabled one (e.g., when posting on Twitter). The phenomenon is more severe in the UI with non non-standard layouts, like CapCut, Meitu, and Feishu. Lacking prior knowledge by GPT-4o leads to the failure of task inference and selecting the correct skills.
Metrics
CRADLE
Human
Smooth Closed-loop Road
4/5
5/5
Sufficient Water Supply
1/5
8/5
Sufficient Power Supply
5/5
5/5
High Zoning Coverage
4/5
4/5
Population
450
=:224
415 1:46
rate of +39.6% peaking at +87.4% in single run. In this game, CRADLE significantly outperforms human players. The achievements are mainly attributed to its cautious strat- egy, by hargaining within smaller range of price variation but haggling more frequently, resulting in a significantly higher turnover rate. In contrast, human players usually fail the deal due to their aggressive strategy by proposing unreasonable price and sometimes confusing buy and sell.
4.2. Baseline Comparison
Since no existing methods are fully applicable to the GCC setting, we select several representative methods with nec- essary adaptions to make them applicable to GCC, labeling them as "like" in Table 2. Compared to CRADLE, Re- act (Yao et al., 202k)-like method only has gather informa- tion, skill curation, action planning and procedural memory module, while Reflextion (Shinn et al., 2023)-like method adds a self-reflection and episodic memory, compared to React-like. To show the necessity of multimodal input with- out access to APIs, we let GPT-4o describe the image and then feed the textual description to Voyager (Wang et al., 2024 as input. Additionally, experiments with GPT- 4o and Claude Opus (Anthropic, 2024) as backbone are
Cities: Skylines. Table 1 shows that CRADLE is able to complete most of the city design with the averaged maximal population of 450 and the highest single population exceed- ing 860. CRADLE manages to build the roads in a closed loop to ensure smooth traffic flow, place multiple wind tur- bines to provide sufficient electricity supply and cover more than 90% of available area with residential, commercial and industrial zones, but fails to provide sufficient water sup- ply for all the regions reliably. The most common failure arises from the missing of water pipes. CRADLE often fail to connect them with each other to cover all zones, result- ing in localized water shortages in the city, and preventing
8

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Table 2: Baseline comparison for five task in RDR2 and one task in Stardew Valley (Cultivation). Numbers before the brackets are average steps with standard deviation for completion. N/A indicates failure for all trials. Every task is run 5 times. Fach trial is run for at most 500 steps in RDR2 and 100 steps in Stardew Valley.
Follow Dutch
Follow Micah
Hitch Horse
Protect Dutch
Search for Supplies
Method
Cultivation
React-like (GPT-4o)
15 2 2 (%/)
74 O (/a)
N/A
N/A
N/A
N/A
Reflextion-like (GPT-4o)
19 1 (%))
58 + 14 (%)
N/A
N/A
N/A
N/A
Voyager-like (GPT-40)
32 12 (%)
N/A
N/A
N/A
N/A
N/A
CRADLE (Claude Opus)
30 (9/)
52 + 17 (%/5
N/A
N/A
N/A
N/A
CRADLE (GPT-40)
13 + (%/)
33 +
26 5
461
+ O
134 +
24 +
(Ours)
(%/)
(%))
(/6)
(/))
(%)
conducted. Due to the limitation of requests per minute, other prompting methods like self- cofsistnns (Wang et al., 2022) and TOT (Yao et al., 2024) are not considered. Note that methods here refer to the agents initialized by the corre- sponding framework with game-specific implementations.
modules represent a problem-solving mindset; as LMM ca- pabilities improve, some or even all of these modules may be combined into a single request. Exploring other potential GCC frameworks is also promising. v) In this work, we mainly focus on enabling foundation agents interact with various software in unified manner without taking training into consideration. As SIMA (Raad et al., 2024) has already shown promising results in a similar setting with trained agents, we will let CRADLE autonomously explore and im- prove over environments through RL (Tan et al., 2023) or collect expert demonstrations for supervised learning (Raad et al., 2024). vi) Due to the large scope of the experiments conducted in this work, the number of runs for each task and human participants are limited. A more comprehen- sive evaluation can be beneficial. vii) Though CRADLE is broadly applicable to any computer task, only a few selected tasks are investigated in this work. We plan to expand its application to a wider range of targets, delve deeper into complex games, and enhance its adaptability for users.
As shwon in Table 2, all the baseline methods can only complete simple and straightforward tasks without complex targets and time delays. Compared to React-like method, Reflextion-like method has better performance in the task of Follow Micah and still fails to complete more complex tasks, emphasizing the importance of task inference and procedural memory. Voyager-like method that loses vision suffers to accomplish tasks and are the worst of all compar- ison methods. CRADLE with GPT-4o always has the best performance across all tasks. CRADLE with GPT-4o has the best performance, while Claude Opus fails frequently due to unreliable OCR ability of the guidance, leading to incorrect skill generation and failures of complex tasks.
We provide more analysis in Appendix D, including OS- World results (D.1), baseline comparison in Stardew Valley ( (D.2) and a comprehensive ablation study (D.3).
6. Conclusion
We introduce GCC, a general and challenging setting to control diverse video games and software with a unified and standard interface, paving the way towards general foun- dation agents across all digital world tasks. To properly address the challenges GCC presents, we propose novel framework, CRADLE, which exhibits strong performance in reasoning and performing actions to accomplish various missions in a set of complex video games and common software applications. To the best of our knowledge, CRA- DLE is the first framework that enables foundation agents to succeed in such a diverse set of environments without rely- ing any built-in APIs. The success of CRADLE greatly extends the reach of foundation agents and demonstrates the feasibility of converting any software, especially com- plex games, into benchmarks to evaluate agents' general intelligence and facilitate further data collection for self- improvement. Although CRADLE still faces difficulties in certain tasks, serves pioneering work to develop more powerful LMM-based agents towards GCC. We hope CRADIEE could motivate further study on the challenging but general GCC setting and more applications in diverse software applications and commercial video games.
5. Limitations and Future Work
Despite CRADLE'S encouraging performance across games and software, several limitations remain. i) Due to the lim- ited ability of current LMM models, CRADLE struggles in recognizing out-of-distributt
(OOD) icons and complet- ing OOD tasks, such as games with non-realistic styles, i.e., Stardew Valley. As LMMs evolve, they can further improve CRADLE'S performance. ii) Another general bottleneck for LMM--basdd agents is the latency caused by the limited inference speed of LMMs, which can also be alleviated as LMMs evolve (e.g., Realtime API (OpenAI, 2024a)). iii) Audio, an important modality, often plays an important role in games and software; which has not been considered in this work. The future work will be enabling CRADLE to process the audio and graphical input simultancously. iv) As the preliminary attempt towards GCC, most CRADLE'S modules need to call LMM explicitly to process the input for best performance, resulting in frequent interactions with LMM and potentially high costs and long delays. The six

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Impact Statement
supported by the National Research Foundation, Singapore under its Industry Alignment PP) Funding Initiative. Any opinions, findings and conclu- sions or recommendations expressed in this material are those of the author(s) and do not reflect the views of Na- tional Research Foundation, Singapore. This research is also supported by the Joint NTU-WeBank Research Centre on Fintech, Nanyang Technological University, Singapore, This work was also supported in part by NSFC under Grant 62450001. Xinrun Xu is advised by Dr. Zhiming Ding from the Institute of Software, Chinese Academy of Sciences, and is supported by the National Key R&D Program of China (No. FF22YFF0533.0)
Pre-positionnnn (IAF.
CRADLE holds great potential to improve effective general computer task completion and boost research and deploy- ment of foundation agents. However, there are some risks associated with its misuse unintended consequences. Be- low, we outline the potential challenges and safeguards necessary to ensure responsible deployment.
Development of Game Cheats. One of the risks associated with CRADLE is its potential misuse in the development of game cheats or bots that could undermine the integrity of online gaming environments. Such misuse could lead to unfair advantages for certain players, disrupt the gaming experience for others, and potentially harm the reputation of game developers. To mitigate this risk, safeguards such as anti-cheat mechanisms and ethical guidelines for the use of Al in gaming should be implemented.
Team Members and Contributions
Roles
Incorrect Operations and Harmful Failures. The au- tonomous nature of CRADLE means that there is risk of incorrect unintended operations, particularly in critical software applications. For example, malfunctioning agent could inadvertently delete important files, send incorrect emails, or perform other actions with harmful consequences. To address this, robust arro-handling mechanisms, human oversight, and faie- should be integrated into the system to ensure that it operates safely and reliably.
Program Leads: Zongqing Lu, Shuicheng Yan, and Bo An
Team Lead: Weihao Tan
Framework Co-Leads: Börje F. Karlsson and Weihao Tan General Advisors: Xinrun Wang and Börje F. Karlsson Core Contributors: Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, Ruyi An, Molei Qin, Chuqiao Zong, Longtao Zheng, Yujie Wu, Xiaoqiang Chai, Xinrun Wang, and Börje F. Karlsson
Ethical and Privacy Concerns. The deployment of CRA- DLE raises ethical and privacy concern,, particularly in scenarios where the agent interacts with sensitive data or performs tasks on behalf of users. For example, an agent operating in a healthcare setting might have access to con- fidential patient information, raising concerns about data security and privacy. To address these issues, CRADLE should be designed with strong data protection measures, including encryption, access controls, and compliance with relevant regulations.
Detailed Contributions
Framework Design. Weihao Tan, Börje F. Karlsson, Wen- tao Zhang, Ziluo Ding, and Xinrun Wang.
RDR2. Implementation, experiments, and analysis by Weihao Tan, Börje F. Karlsson, Wentao Zhang, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Molei Qin, Longtao Zheng, Xinrun Xu, Yifei Bi, Pengjie Gu, and Yewen Li. Xinrun Wang provided insightful suggestions.
Negative Agent Behavior. There is a risk that CRADLE could exhibit unintended or undesirable behavior, particu- larly in complex or unpredictable environments. For exam- ple, agent might develop strategies that are effective but unethical, such exploiting loopholes in software or engag- ing in deceptive practices. To mitigate this risk, CRADLE should be equipped with ethical guidelines and constraints that prevent harmful behavior. Additionally, ongoing moni- toring and evaluation of the agent's actions should be con- ducted to ensure that operates in alignment with human values.
Stardew Valley. Implementation, experiments, and analysis by Weihao Tan, Wentao Zhang, Chuqiao Zong, Yujie Wu, Xiaoqiang Chai, Haochong Xia, Yewen Li, Ruyi An, and Molei Qin. Xinrun Wang. Long Tian, Chaojic Wang, and Börje F. Karlsson provided insightful suggestions.
Cities: Skylines. Implementation, experiments, and anal- ysis by Weihao Tan, Wentao Zhang, Haochong Xia, and Ceyao Zhang. Xinrun Wang provided insightful sugges- tions.
Dealer's Life 2. Implementation, experiments, and analysis by Yewen Li, Ruyi An. Weihao Tan, Wentao Zhang, and Xinrun Wang provided insightful suggestions.
Acknowledgments
We thank Ye Wang and Jiangxing Wang for their time and effort in helping us test the framework. This research is
Software Applications (including Chrome, Outlook, Cap- Cut, Meitu, and Feishu). Implementation, experiments,
10

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
and analysis by Xinrun Xu, Börje F. Karlsson, and Xiyun Li. Weihao Tan and Xinrun Wang provided insightful sug- gestions.
Do as I can, say: Grounding language in robotic affordances. In Conference Robot Learning, pp. 287- 318. PMLR, 2023b.
OSWorld. Implementation, experiments, and analysis by Haochong Xia, Tianbao Xie, Pengjie Gu. Weihao Tan, Xin- run Xu, Xinrun Wang, and Börje F. Karlsson provided in- sightful suggestions.
Brunelli, R. Template matching techniques in computer vision: theory and practice. John Wiley & Sons, 2009.
Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S.,
Abbeel, P. and Dragan, A. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Paper Writing, Weihao Tan, Xinrun Wang, Börje F. Karls- son, Wentao Zhang, Xinrun Xu, Yewen Li, Ruyi An, and Chuqiao Zong. Haochong Xia, Molei Qin, and Ceyao Zhang further contributed with writers in appendix organization.
Cheng, K., Sun, Q. Chu, Y., Xu, F., Li, Y., Zhang, J., and Wu, Z. SeeClick: Harnessing GUI ground- ing for advanced visual GUI agents. arXiv preprint arXiv:2401.109:: 2024.
Organization. Zongqing Lu, Shuicheng Yan, and Bo An provided directional research advice and organizational sup- port.
Chevalier-Boisver M., Bahdanau, D., Lahlou, S., Willems, L.., Saharia, C., Nguyen, T. H., and Bengio, Y. BabyAI: First steps towards grounded language learning with hu- man in the loop. In International Conference on Learning Representations, 2019. URL https:://///// openrevien. net fort/fooum?aa www.::idddddeeeee.ooommo
Weihao Tan's 's work was conducted during his internships at Skywork AI and BAAL Longtao Zheng was also an intern at Skywork AL. Xinrun Xu, Bohan Zhou, and Junpeng Yue were interns at BAAL
References
Deitke, M., VanderBilt, E., Herrasti, A., Weihs, I.., Ehsani, K., Salvador, J., Han, W., Kolve, E., Kembhavi, A., and Mottaghi, R. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982 5994,, 2022.
Anthropic. The claude 3 model family: Opus, son- net, haiku, 2024. URL https::///// /www. www.anttrrpic.c com/ nwws//wwww//eeeeeeeee cww.ee-aeeeeeeeeeeeeee 3- family.
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, Video pretraining (VPT): Learning act by watching unlabeled online vidcos. Advances in Neural Information Processing Systems, 01224639-2465- 2022.
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2Web: Towards a generalist agent for the web. arXiv preprint arXiv:2306.00 06000, 2023.
Bakhtin, A., Brown, N.. Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624); 1067-1074, 2022.
Driess, D., Xia, F. Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv: 03378, 2023.
Ellis, B., Cook, J., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, N., and Whiteson, S. SMACv2: An improved benchmark for cooperative multi-agent rein- forcement learning. In Thirty-seventh Conference Neu- ral Information Processing Systems Datasets and Bench- marks Track, 2023. URL https: /tttpp:///ppeeeeee /pppnevvin.....e net / ftrumorrmm?/mmmm ? id- 50 LGiJ33u.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The Areade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., et al. Dota 2 with large scale deep reinforcement learning, arXiv preprint /9Xiv:1912.00 191200660, 2019.
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo:
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D.. Dubey, A., Finn, C., et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 15818, 2023a.
Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Informa- tion Processing Systems, 35:18343-18362, 2022.
Furuta, H., Nachum, O., Lee, K.-H., Matsuo, Y., Gu, S. S., and Gur, I. Multimodal web navigation with instruction-finetueed foundation models. arXiv preprint arXiv:2305. /854, 2023.
Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D.,, Ibarz, J., Irpan, A., Jang, E., Julian, R., al.

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Gao, D., Ji, L.,, Bai, Z., Ouyang, M., Li, P., Mao, D., Wu, Q- Zhang, W., Wang, P., Guo, X., et al. ASSISTGUI: Task-oriented desktop graphical user interface automa- tion. arXiv preprint arXiv:2312.13100 2023.
W.-Y,, al. Segment anything. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 4015-4026, 2023.
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M. C., Huang, P.-Y.. Neubig, G., Zhou, S., Salakhutdinov, R., and Fried, D. Visual WebAren: Evaluating multimodal agents on re- alistic visual web tasks. arXiv preprint arXiv:2401. 2024.
Gao, X., Gong, R., Shu, T., Xie, X., Wang, S., and Zhu, S.- C. Vrkitchen: interactive 3d virtual environment for task-oriented learning. arXiv preprint 2019.
Gur, I., Furuta, H., Huang. A., Safdari, M., Matsuo, Y., Eck, D., and Faust, A. A real-world webagent with planning. long context understanding, and program synthesis. arXiv preprint ar arXiv:2307. /2856, 2023.
Kurach, K., Raichuk, A., Stañczyk, P., Zajac, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michalski, M., Bousquet, O., al. Google research football: A novel reinforeement learning enviromment. In Proceedings of the AAAI conference on artificial intelligence, pp. 4501- 4510, 2020.
Guss, W. H., Houghton, B., Topin, N., Wang, P. Codel, C., Veloso, M., and Salakhutdinov, R. Minerl: A large- scale dataset of Minecraft demonstrations. arXiv preprint arXiv: /907. 13440, 2019.
Leibo, J. Z., DueñezzGuzmaan E. A., Vezhnevets, A., Aga- piou, J. P., Sunehag. P., Koster, R., Matyas, J., Beattie, C., Mordatch, I., and Graepel, T. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International conference on machine learning, pp. 6187- 6199. PMLR, 2021.
He, H., Yao, W., Ma, K., Yu, W., Dai, Y., Zhang, H., Lan, Z., and Yu, D. Vobyee:::r: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401. 13919, 2024.
Hong, W., Wang, W., Lv, Q. Xu, J., Yu, W., Ji, Wang, Y., Wang. Z., Dong. Y., Ding. M., et al. CogAgent: A visual language model for GUI agents. arXiv preprint arXiv:2312.08914 2023.
Li, C., Xia, F., Martín MartinnMarttn, R., Lingelbach, M., Srivas- tava, S., Shen, B., Vainio, K., Gokmen, C., Dharan, G., Jain, T., et al. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv:2108.03272 2021.
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:220.005608 2022.
Liu, F. Z., Guu, K., Pasupat, P., Shi, T., and Liang, P. Rein- forcement learning on web interfaces using workflow- guided exploration. In International Conference on Learning Representations (ICLR), 2018. URL https: //arxiv. org abs / 110.//8020000 08802.
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castaneda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. Human-level performance in 3D multiplayer games with population- based reinforcement learning. Science, 364(6443):859- 865, 2019.
Liu, S., Zeng. Z., Ren, T., Li, F., Zhang, H., Yang. J., Li, C., Yang, J., Su, H., Zhu, J., et al. Grounding Dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprins arXiv:2303.05499, 2023.
Ma, W., Mi, Q., Yan, X., Wu, Y., Lin, R., Zhang, H., and Wang. J. Large language models play StarCraft II: Bench- marks and a chain of summarization approach. arXiv 2023.
Johnson, M., Hofmann, K., Hutton, T., and Bignell, D. The Malmo platform for artificial intelligence experimenta- tion. In Ifcai, pp. 4247,4247.25 2016.
preprint arXiv:2312.
Kapoor, R., Butala, Y. P., Russak, M., Koh, J. Y., Kamble, K., Alshikh, W., and Salakhutdinov, R. OMniACT: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web, 2024.
Manolis Savva Abhishek Kadian Oleksandr Maksymets* Zhao, Y., Wijmans, E., Jain, B., Straub, J., Liu, J., Koltun, V., Malik, J., Parikh, D., and Batra, D. Habitat: A Platform for Embodied Al Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 2019.
Kauten, C. Super Mario Bros for OpenAl Gym. GitHub, 2018. URL httph:ttt//// htttt:://githhlle com Koom/Kautejja/ gym- sym-suppr-marro-bree
Mialon, G., Fourrier, C., Swift, C., Wolf, T., LeCun, Y., and Scialom, T. GAIA: a benchmark for general Al assistants. 2023.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo,
arXiv preprint
12

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Morris, M. R., Sohl-dickstein, J., Fiedel, N., Warkentin, T.,
Foerster, J., and Whiteson, S. The Starcraft multi-agent challenge. arXiv preprint
Dafoe, A., Faust, A., Farabet, C.,, and Legg, S. Levels of AGI: Operationalizing progress on the path to AGI. arXiv 2023.
2019.
preprint arXiv:23
Shen, B., Xia, F., Li, C., Martin-Martin, R., Fan, L., Wang. G., PérezérrD Arpino, C., Buch, S., Srivastava, S., Tchapmi, L., et al. igibson 1.0: a simulation environment for interactive tasks in large realistic scenes. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7520-7527. IEEE, 2021.
Niu, R., Li, J., Wang. S., Fu, Y., Hu, X., Leng, X., Kong, H., Chang. Y., and Wang, Q. ScreenAgent: A vision language model-driven computer control agent. arXiv preprint arXivv2402.07945. 2024.
OpenAI. Universe, 2016, URL htwpt:t/p///e/ /tpptaa:////p....... com/ index/universe!
Shi, T., Karpathy, A., Fan, L., Hernandez, J., and Liang, P. World of bits: An open-domain platform for web- based agents. In International Conference on Machine Learning, pp. 3135-3144 PMLR, 2017.
OpenAI. New and improved embedding model, 2022. URL https:// careiiiooiiiiiin com/index/ aaa.aadd--mmooveee-emddddddddddd
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K. R., and Yao, S. Reflexion: language agents with verbal rein- forcement learning. In Thirty-seventh Conference on Neu- ral Information Processing Systems, 2023. URL https: / //openeeveww.. wwwe/efruunemeee www.moiideiiiooeeoe wianniiidvvvveeee..
OpenAl. Introducing the realtime api, 2024a. URL https://opena.: index/
introducing-the--iee wtweealttmemapp.//
OpenAL. Hello gpt-4o, 2024b. URL httpt::t/tpeeaaaee hom/index.oo httbee//helllopp-000000
Szot, A., Clegg, A., Undersander, E., Wijmans, E., Zhao, Y., Turner, J., Maestre, N., Mukadam, M., Chaplot, D., Maksymets, O., Gokaslan, A., Vondrus, V., Dharur, S., Meier, F., Galuba, W., Chang, A., Kira, Z., Koltun, V., Malik, J., Savva, M., and Batra, D. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Park, J. S., O'Brien, J., Cai, C. Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposiun on User Interface Software and Technology, pp. 1-22, 2023.
Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., and Torralba, A. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 8494 8502, 2018
Tan, W., Zhang, W., Liu, S., Zheng, L., Wang, X., and
An, B. True knowledge comes from practice: Aligning large language models with embodied environments via reinforcement learning. In ICLR, 2023.
Vinyals, O., Babuschkin, I., Chung, J., Mathieu, M., Jader- berg, M., Czarnecki, W. M., Dudzik, A., Huang, A., Georgiev, P., Powell, R., et al. AlphaStar: Mastering the real-time strategy game Starcraft II. DeepMind blog, 2:20, 2019.
Puig. X., Undersander, E., Szot, A., Cote, M. D., Yang. T.- Y., Partsey, R., Desai, R., Clegg, A. W., Hlavac, M., Min, S. Y., et al. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint 1rXiv:2310.13724, 2023.
Qi, S., Chen, S., Li, Y., Kong, X., Wang, J., Yang. B., Wong. P., Zhong, Y., Zhang, X., Zhang, Z. et al. CivRealm: A learning and reasoning odyssey in Civilization for decision-making agents. In ICIR, 2024.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open-ended embodied agent with large language models. Transac- tions on Machine Learning Research, 2024a. ISSN 2835- 8856. URL https: /ttpreeveeeeeeeeeee/eeeeeeeeeeee net forum? id=ehfRiFOR3a.
Raad, M. A., Ahuja, A., Barros, C., Besse, F., Bolt, A., Bolton, A., Brownfield, B., Buttimore, G., Cant, M., Chakera, S., et al. Scaling instructable agents across many simulated worlds. arXiv preprint arXiv:2404.1017,, 2024.
/
Wang, J., Xu, H., Ye, J., Yan, M., Shen, W., Zhang, J., Huang, F., and Sang, J. Mobile-Agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16155. 2024b.
Rawles, C., Li, A., Rodriguez, D., Riva, O., and Lillicrap, T. Android in the wild: A large-scale dataset for Android device control. arXiv preprint urXiv:2307. 10300.00088 10088, 2023.
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 2022.
Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, G., Hung, C.-M., Torr, P. H.,
13

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z.,, Zheng, Z.,, Yang, Y., and Liang, Y. Jarvis-l: Open-world multi-task agents with memory- augmented multimodal language models. arXiv preprint arXiv:231 (rxiv:23/105597, 2023a.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. ReAct: Synergizing reasoning and act- ing in language models. In International Conference on Learning Representations (ICLR), 2023.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Wang, Z., Cai, S., Liu, A., Ma, X., and Liang, Y. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. In ICML, 2023b.
Zhang, C.,, Li, L., He, S., Zhang. X., Qiao, B., Qin, S.,
Wu, S. A., Wang, R. E., Evans, J. A., Tenenbaum, J. B., Parkes, D. C., and Kleiman- Kleiman-Weiner, M. Too many cooks: Coordinating multi-agent collaboration through inverse planning. Topics in Cognitive Sci- ence, n/a(n/a), 2021. doi: https://oi..rg//00 10wl/////.11/////00010.1111 1/tops
Ma, M., Kang, Y., Lin, Q., Rajmohan, S., al. UFO: A Ul-focused agent for Windows OS interaction. arXiv 2024.
preprint arXiv:2402.0783
Zheng, B., Gou, B., Kil, J., Sun, H., and Su, Y. GPT- 4V(ision) generalist web agent, if grounded. arXiv preprint 2rX1v:2401.0161 2024a.
12525. URL https: http:::://olinelbbbrrrr wiley. com/doi/ /tw.//a/////ae/ /lbs/10... http///b//....1/opp........ www./topso.......o 12525.
Wu, Z., Han, C., Ding, Z., Weng, Z., Liu, Z., Yao, S., Yu, T., and Kong, L. OS-copilot: Towards generalist computer agents with self- cel--improvement. arXiv preprint arXiv:2402.07456 2024.
Zheng, L., Wang, R., Wang, X., and An, B. Synapse: Trajectory Trajettoryas--xemmpla as-exmpplar prompting with memory for com- puter control. In ICLR, 2024b.
Zhou, S., Xu, F. F., Zhu, H., Zhou, X., Lo, R., Sridhar, A., Cheng, X., Bisk, Y., Fried, D., Alon, U., et al. WebArena: A realistic web environment for building autonomous agents. arXiv preprint vrXiv:2307.1385. 2023.
Wurman, P. R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T. J., Capobianco, R., Devlic, A., Eckert, F,, Fuchs, F., et al. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature, 602(7896)223--225 2022.
Xiao, Y. Tan, W., and Amato, C. Asynchronous actor- critic for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35:4385 4400, 2022.
Xie, T., Zhang, D., Chen, J., Li, X., Zhao, S., Cao, R., Hua, T. J., Cheng, Z., Shin, D., Lei, F., et al. Os- world: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972. 2024.
Yan, A., Yang. Z., Zhu, W., Lin, K., Li, L., Wang, J., Yang, J. Zhong, Y., McAuley, J., Gao, J., Liu, Z., and Wang, L. GPT-4V in wonderland: Large multimodal models for zero-shot smartphone GUI navigation. arXiv preprint arXiv:2311.07562, 2023.
Yang, J., Zhang, H., Li, F., Zou, X., Li, C., and Gao, Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023a.
Yang, 7., Liu, J., Han, Y., Chen, X., Huang, 7.,, Fu, B., and Yu, G. AppAgent: Multimodal agents as smartphone users. arXiv preprint 4xXin:2312.13771, 2023b.
Yao, S., Chen, H., Yang, J., and Narasimhan, K. Web- shop: Towards scalable real-world web interaction with grounded language agents. Advances Neural Informa- tion Processing Systems, 35:20744-20755 2022.
14

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
A. Game & Task Introduction
The four selected representative games are:
Red Dead Redemption 2 (RDR2), an epic AAA 3D role-playing game (RPG) with rich storylines, realistic scenes, and immersive open-ended world; where players can complete missions by following the instructions, freely explore the world, interact with non-player characters (NPCs) and engage in variety of activities such as hunting and fishing, in first- or third-person perspective. This game offers great challenges in 3D embodied navigation and interaction.
Stardew Valley, a 2D pixel-art farming simulation game where players can restore and expand a farm through carefully planned activities such as planting crops, mining, fishing, and crafting. Players can build relationships with the villagers, participate in seasonal events, and uncover the mysteries of the valley. The game encourages strategic planning and time management, as each day brings new opportunities and challenges. Players have to balance their energy and resources to maximize their farm's 's productivity and profitability.
Dealer's Life 2, simulation game where players manage pawn shop. They must assess the value of items, haggle with customers, and make strategic decisions to grow their business. The game offers a dynamic market influenced by trends, customer preferences, and random events, requiring players to adapt and refine their negotiation tactics.
Cities: Skylines, a 3D, top-down view, city-building game where players take on the role of a city mayor, tasked with the development and management of thriving metropolis, engaging in urban planning by controlling zoning, road placement, taxation, public services, and public transportation in an area. They must balance the needs and desires of the population with the city's s budget, addressing issues such as traffic congestion, pollution, and citizen satisfaction. The game provides a sandbox environment where creativity and strategic thinking are key to building efficient and aesthetically pleasing urban landscapes. It also requires highly precise mouse control.
B. Extended Related Work
B.1. Environments and Benchmarks for Computer Control
Environments and Benchmarks on Software Applications. Simulated environments on computers have been popular benchmarks and testbeds for the research community. Earlier computer control environments primarily focused on web navigation tasks (Shi al., 2017; Liu et al., 2018; Yao et al., 2022; Deng et al., 2023; Zhou et al., 2023; Koh et al., 2024). Recent benchmarks start to include various common software (Kapoor et al., 2024; Xie et al., 2024), aiming to develop generalist agent in the digital world. However, none of them takes video games into consideration, missing a key component of computer control.
Environments and Benchmarks on Video Games, On the other side, many research environments are built on top of video games, significantly advancing the study of mecision-making, especially, reinforcement learning (RL). Examples include but are not limited to Atari games (Bellemare et al., 2013), Super Mario Bros (Kauten, 2018), Google Research Foothall (Kurach et al., 2020), Minecraft (Johnson al., 2016; Guss et al., 2019; Fan et al., 2022), Dota II (Berner et al., 2019), StarCraft II (Vinyals et al., 2019; Samvelyan et al., 2019; Ellis et al., 2023), Quake III (Jaderberg et al., 2019), Gran Turismo (Wurman et al., 2022), Diplomacy (Bakhtin et al., 2022) and Civilization (Qi et al., 2024). Additionally, many custom-built environments, especially grid world and embodied scenarios, are created from scratch in a gime-like manner to facilitate agent development, such as BabyAl Overcooked (Carroll et al., 2019; Wu et al., 2021; Xiao et al., 2022), VRKitchen (Gao et al., 2019), VirtualHome (Puig et al., 2018), iGibson (Shen et al., 2021; Li et al., 2021). ProcTHOR (Deitke al., 2022), Habitat (Manolis Savva et al., 2019; Szot et al., 2021; Puig et al., 2023), and Generative agents (Park et al., 2023).
Beeevaller-Boisverr et al., 2019), Melting Pot (Leibo et al., 2021),
Each of these environments highly relies on the accessibility of the open-source code or provided built-in APIs Significant human efforts are required for implementation and encapsulation, enabling agent interaction. Therefore, despite the abundance of software and games available for human use, only a limited number are accessible to agents, especially for commercial closed-souree games and software applications. Additionally, the lack of consensus on environment standards further complicates the interaction, as each environment has specific observation and action spaces, tailored to its unique requirements. This variation exacerbates the challenge of enabling agents interact with diverse environments and collect data with a consistent level of fine-grained semantics to improve the agent's capabilities. Few agents can complete tasks across multiple environments far.
Similar to OpenAl Universe (OpenAI, 2016) and SIMA (Raad et al., 2024), our goal is to explore unified way that allows agents to interact for measuring and training agents' abilities across a wide range of games, websites, and other applications
15

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
without heavy human efforts needed. This approach aims to prove that diverse software applications and games can serve as out-of-the-hox environments for Al development.
B.2. LMM-based Agents for Computer Tasks
Agents for Software Manipulation. Agents for software applications are developed to complete tasks such as web navigation (Zhou al., 2023; Deng et al., 2023; Mialon et al., 2023) and software application control (Rawles et al., 2023; Yang et al., 2023b; Kapoor et al., 2024). While previous LLM-based web agents (Deng et al., 2023; Zhou et al., 2023; Gur et al., 2023; Zheng et al., 2024b) show some promising results in effectively interacting with content on webpages, they usually use raw HTML code and DOM tree as input and interact with the available element IDs, ignoring the rich visual patterns with key information, like icons, images, and spatial relations. Recently, multimodal web agents (Yan et al., 2023; Gao et al., 2023; He et al., 2024; Zheng et al., 2024a; Niu et al., 2024; Zhang et al., 2024; Wu et al., 2024) and mobile app agents (Yang et al., 2023b; Wang et al., 2024b) have been explored. Though using screenshots as imput, they still rely on built-in APIs and advanced tools to get internal information, like available interactive element IDs, to execute corresponding actions, which greatly limits their applicability. Other train-based agents (Hong et al., 2023; Furuta et al., 2023; Cheng et al., 2024) also suffer from generalizing to unseen software and tasks. Moreover, all of these works primarily focus on static websites and software, which greatly reduces the need for timeliness and simplifies the setting by ignoring the dynamics between adjacent screenshots, L.e., animations, and incomplete action space without considering the duration of the key press and different mouse mode. It results in the failure of deployment to the tasks with rapid graphics changes, egg., game playing.
Agents for Game Playing. Several attempts try develop foundation agents for complex video games, such as Minecraft (Wang et al., 2023b;a; 2024a), Starcraft II (Ma et al., 2023) and Civilization-like game (Qi et al., 2024) with textual observations obtained from internal APIs and pre-defined semantic actions. Although JARVIS-1 (Wang et al., 2023a) claims to interact with the environment in a human-like manner with the screenshots as input and mouse and keyboard for control, its action space is predefined as a hybrid space composed of keyboard, mouse, and API. The game-specific observation and action spaces prohibit the generalization of them to other novel games. Pre-trained with videos with action labels, VPT (Baker et al., 2022) manages output mouse and keyboard control with raw screenshots as input without any additional information. However, collecting videos with action labels is time-consuming and costly, which is difficult to generalize to multiple environments. Another concurrent work, SIMA (Raad et al., 2024) trained embodied agents to complete 10-second- long tasks over ten 3D video games. Though their results are promising to scale up, they focus on behavior cloning with gameplay data from human experts, resulting in high expense.
In both targeting complex video games and diverse software applications, CRADLE attempts to explore a new way to efficiently interact with different complex environments in a unified manner and facilitate further data collection. In a nutshell, to our best knowledge, there are currently no agents under the GCC setting, reported to show superior performance and generalization in complex video games and across computer tasks. In this work, we make preliminary attempt to explore and benchmark diverse environments in this setting, applying our framework to diverse challenging environments under GCC and proposing an approach where any software can be used to benchmark agentic capabilities in it.
C. Experimental Cost
Table 3: Financial and time-related costs of running all the tasks once each environment or domain.
Cities: Skylines
Stardew Valley
Dealer's Life 2
Software Apps
RDR2
OSWorld
Total
Tasks Num.
14
3
25
369
Input Tokens
600M
150M
60M
25M
45M
Output Toekns
20M
7.5M
4M
IM
2.5M
Cost (USD)
$3300
$862.5
$345
$140
$262.5
$500
$5410
Time
240 hrs
60 hrs
30 hrs
20 hrs
50 hrs
240 hrs
640 hrs
Table shows the approximate cost of experiments in Section 4.1 with gpt-40-2024-05-13. Baselines comparison and ablation studies are not included. Since all the tasks were run 5 times except for OSWorld once, the total cost of getting all the results shown in Section 4.1 is approximately 5400 USD. claude-3-opus-2024022 will roughly use 3X more money and
16

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
2X more time compared gpt-4o-2024-05-13 due its higher price and longer lateney. We also want that with the latest model, gpt-4o-2024-08-06 the cost will be halved. We estimate that costs will decrease by orders of magnitude in the coming few years, Then the cost will be affordable every researcher and developer.
D. Additional Results
D.I. Results in OSWorld
Table shows that CRADLE achieves the overall highest success rate OSWorld, compared to the baselines without relying on any internal APIs to provide extra grounding labels, e.g., Set-of-Mart (SoM) (Yang al., 2023a). The information gathering module improves grounding for more precise action execution, increasing the performance. The self-reflection module enables CRADLE to predict infeasible tasks and subsequently fix mistakes, shown in the Professional domain results, where it achieves a 20.41% success rate, significantly surpassing the baselines.
Table 4: Success rates (%) of different methods OSWorld.
Office (117)
OS
Daily
Workff-- ow(101)
Professi- onal (49)
All
Method
(24)
(78)
(369)
GPT-4o
3.58
8.33
6.07
5.58
4.08
5.03
GPT-40+SoM
3.58
20.83
3.99
3.60
2.04
4.59
CRADLE
3.58
16.67
6.55
5.48
20.41
7.81
D.2. Baseline Comparison in Stardew Valley
Figure 10 provides the detailed performance of each baseline method in the Cultivation task in Stardew Valley. Without task inference and episodic memory for summarization, even React-like and Reflexion-like methods sometimes managed to get the parsnip to sprout from the ground, they failed to harvest because GPT-4o failed to recognize the mature parsnip. Episodic memory can help CRADLE record the days of watering and know when the crop can harvested. Voyager-like like method struggles with getting out of the house and returning home due to the lack of visual input. Claude Opus also has difficulties in localizing the position the character and the crop. Moreover, it prefers moving characters much more frequently than GPT-4, resulting in the failure position the character in front of the crop.
React-like
Reflexion-like
Voyager-like
Cradle (Claude Opus)
CRADLE (Ours)
0
10
20
30
50
Steps
Figure 10: Performance of cach method in task Cultivation. The Y-axis shows the stage of parsnip. Only if the mutual parsnip (shown on the top of the y-axis) is obtained will this trial be counted as success.
D.3. Ablation Study
Besides comparing with other bascline methods, we provide a complete ablation study by systematically removing each module of CRADLE to show the effectiveness in Table 5. We mainly show the results of 6 consecutive subtasks at the beginning of the main storyline, separated from the tasks of Follow Micah, Hitch Horse and Protect Dutch in RDR2. Note that the combination skill curation, action planning and procedural memory the minimal unit of our framework. Without
17

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
any of them, the agent cannot generate and execute valid actions successfully. So these modules are not ablated.
The most significant decline in agent capabiliities arises from the absence of the information gathering module. Without this module, the agent is unable to extract key information in the observation, which is critical for all other modules to function effectively. The second largest impact comes from the lack of the self-reflection module, which is instrumental in correcting mistakes and recognizing when the agent is stuck, such as in the subtask of Go to Shed. Third, the task inference module is vital for tasks that require strict adherence to guidance, like Switch Weapon. these cases, the in-game instructions appear only at the beginning of the task, as seen in Follow Micah and Go to Shed. Lastly, episodic memory becomes increasingly important as tasks grow more complex, requiring more steps to complete, such as in Go to Shed and Combat, which involve far more steps than other subtasks. Overall, each module plays crucial and distinct role in the CRADLE framework. Removing isolating any of them significantly reduces the agent's 's effectiveness, underscoring the importance of their integrated function.
Table 5: Success rates of each variant by systematically removing CRADIEE'S module six consecutive subtasks in RDR2. Every subtask is run 5 times. Each of subtasks are run for at most 500 steps.
w/o Information
w/o Self-
w/o Task
w/o Episodic
Subtask
CRADLE
Gathering
Reflection
Inference
Memory
Follow Micah
0%
0%
40%
80%
100%
Hitch Horse
0%
100%
100% %
100%
100%
Go to Shed
0%
20%
40%
20%
80 %
Peek out of Cover 60%
100%
80%
100%
100%
Switch Weapon
0%
80%
60%
80%
100%
Combat
0%
0%
0%
0%
20%
E. General Implementation
Here we introduce the general implementation details of CRADLE. For specialized implementations addressing issues unique to their own environment, please refer to the corresponding section.
Hardware. All software and games can be run on regular Windows 10 machines, except for RDR2, which is tested on machines with RTX-4090 GPU separately.
Backbone Model. We employ GPT-4o (OpenAI, 2024b), currently one of the most capable LMM models, as the framework's 's backbone model. If not mentioned explicitly, all the experiments are done with 200--o-2022-05-13 Temperature set to to lower the variance of the text generation. Same as Voyager (Wang et al., 2024a), we use OpenAl's gext-ebbedding-ate -atiiin0aaa-02222 model (OpenAI, 2022) to generate embeddings for each skill, stored in the procedural memory and retrieved according to the similarities.
Evaluation Methods. Unlike conventional research benchmarks, which usually provide grounding signals for evaluation, it is difficult to have a unified and general method to determine whether a task is completed automatically in diverse software, especially in video games. Similarly to SIMA (Raad et al., 2024), we apply human evaluation to all tasks across application software and games. Moreover, to provide more quantitative results and a comparison baseline, we provide results for the OSWorld (Xie et al.. 2024) benchmark, contemporaneous benchmark that provides evaluation scripts for at least one solution per task.
Observation Space. CRADLE only takes a video clip, which records the progress of execution of the last action, input. To lower the frequency of interaction with backbone models and reduce the strain on the computer, video is recorded at 2 fps (a screenshot every 0.5 seconds), which proves to be sufficient in most cases for information gathering without missing any important information. It is important to note that, due to the dynamism of the RDR2 and Stardew Valley and the LMM inference and communication latency, we must pause those game environments while waiting for backbone model responses. Other environments execute continuously.
Action Space. For the action space, it includes all possible keyboard and mouse operations, including key_press, key_hold, key_release, mouse_move wheel_scrooll which can be combined in different ways to form combos and shortcuts, use keys in fast sequence, or coordinate timings. We choose to use Python code to simulate these operations and encapsulate them into an io_env class. Skill code needs to be generated by the agent in order to utilize such functions and affordances so executed actions take effect. Table illustrates CRADLE'S action space.
mouse_click, house_hold, mouse_ mouse_released and
18

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Table Action space the CRADLE framework, including action attributes. Coordinate system is either absolute or relative. Actions with durations can be either synchronous or asynchronous.
Type
Action
Attributes
Key name (string),
Key Press
Key press duration (seconds:float
Key Hold
Key name (string)
Key Release
Key name (string)
Key names (strings),
Key Combo
Key combo duration (seconds:float)
Wait behaviour (sync/async)
Keyboard
Key names (strings),
Hotkey
Hotkey sequence duration (seconds:float,
Wait behaviour (sync/async)
String to type (string),
Text Type
Typing duration (seconds:float)
Mouse button (left/midddl/rrihht))
Button Click
Button click duration (seconds:float)
Button Hold
Mouse button (left/middleeright)
Button Release
Mouse button (left/middlerright)
Mouse position (width:int, height:int),
Mouse speed (seconds:float),
Move
Coordinate system (relative/absolute),
Tween mode (enum)
Orientation (vertical),
Mouse
Scroll
Distance (pixels:int),
Duration (seconds:float)
Wait
Noop
important to note that, while some works (e.g., AssistantGUI (Gao et al., 2023), OmniACT (Kapoor et al., 2024) and OSWorld (Xie al., 2024)) use PyAutoGU1 for keyboard and mouse control, this approach does not work in all applications, particularly in modern video games using DirectX 4. Moreover, such work chooses to expose subset of the library functionality in its action space, ignoring dimensions like press duration and movement speed, which are critical in many scenarios (e.g., RDR2, for opening the weapon wheel and changing view).
To ensure wide game and software compatibility and accommodate different operating systems, in our current implementation we use the similar PyDirecilnpul library and PyAutoGUl for keyboard control, utilize A/IK and write our own abstraction (using the ctypes library ?) to send low-level mouse commands to the operating system for mouse control. For increased portability and ease of maintenance, all keyboard and mouse control is encapsulated in class, called 10_env.
Notably, our low-level control wrapper is adapted for both MacOS and Windows systems, making the OS transparent to us. At the software window level, we implemented automatic switching between the target software window and the window running the agent (using Python ctypes for Windows and AppleScript for MacOS 8).
Python library that provides cross passf-oltooom GUI automation module- https: / http////wwwe/ggrt///aatttoooo "iicrosoft DirectX graphics provides a set of APIs for perffornree multimedia apps https://eearn
en 2-- us windows// win32/ / /ddrectx
'Python library encapsulating Microsoft's DirectInput calls for convenience manipulating keyboard httss com wenncodebyggminnnn wwwaaangggnirreeeeeeeeeeeepeeeee pydi rect input
A fully typed Python wrapper around AutoHotkey to keyboard and mouse control httppps:/ggtaab..ccc///oooooetet
ahk
Python library that provides C compatible data types, and allows calling functions DL.L/s binaries /3/ 3/ bibarr//aa tbrary/ctypes. ctypes. html ctml
org/
'AppleScriptis is scripting language created by Apple, which allows users to directly control scriptable applications, as well as parts of MacOS- https:/ / /eveel oper.appll com.c.omm111. /ttpaa/ccceeeecccaaaaaaa////////
19

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Procedure Memory. This memory stores pre-defined basic skills and the generated skills captured from the Skill Curation. However, as continuously obtain new skills during game playing. the number of skills in procedural memory keeps increasing, and it is hard for GPT-4o to precisely select the most suitable skill from the large memory. Thus, similar to Voyager (Wang et al., 2024a), we use OpenAl's eext-embedding-i -aaag-00a020222 model (OpenAI, 2022) to generate embeddings for each skill and store pre-defined basic skills and any generated skills captured from Skill Curation, along with their embeddings in a procedural memory. We retrieve a subset of skills, that are relevant to the given task, and then let GPT-4o select the most suitable one from the subset. In the skill retrieval, we pre-compute the embeddings of the documentations (code, comments and descriptions) of skill functions. which describe the skill functionality, and compute the embedding of the given task. Then we compute the cosine similarities between the skill documentation embeddings and the task embedding. The higher similarity means that the skill's functionality is more relevant to the given task. We select the top K skills with the highest similarities as the subset. Using similarity matching to select small candidate set simplifies the process of choosing skills.
This memory stores all the useful information provided by the environment and LMM, which consists of short-term memory and long-term summary.
Episodic Memory.
The short-term memory stores the screenshots within the recent k interactions in game playing and the corresponding information from other modules, e.g., screenshot descriptions, task guidance. actions, and reasoning. We set k to five, and it can be regarded as the memory length. Information stored over k interactions ago will be forgotten from direct short-term memory. Empirically, we found that recent information is crucial for pecision-making, while a too-long memory length would cause hallucinations. In addition, other modules continuously retrieve recent information from short-term memory and update the short-term memory by storing the newest information.
For some long-horizon tasks, short-term memory is not enough. This is because the completion of a long-horizon task might require historical information from long steps ago. For example, the agent might do a series of short-horizon tasks during a long-horizon task, which makes the original long-horizon task forgotten in short-term memory. To maintain the long-term valuable information while avoiding the long-token burden of GPT-4o, we propose a recurrent information summary as long-term memory, which is the text summarization of experiences in game playing, including the ongoing task, the past entities that the player met, and the past behaviors of the player and NPCs.
In more detail, we provide GPT-4o with the summarization before the current screenshot and the recent screenshots with corresponding descriptions, and GPT-4o will make new summarization by organizing the tasks, entities, and behaviors in the time order with sentence number restriction. Then we update the summarization to be the newly generated one, which includes the information in the current screenshot. The recurrent summarization update, inspired by RNN, achieves linear- time inference by preserving a hidden state that encapsulates historical input. This method ensures the compactness of summarization token lengths and recent input data. Furthermore, the incorporation of long-term memory enables the agent to effectively retain crucial information over extended periods, thereby enhancing decision maaciign-mmking capabilities.
Information Gathering. Given the video clip as input, we mainly depend on GPT-4o's 's OCR capabilities to extract textual information in the keyframes, which usually contain critical guidance and notifications for the current situation. We also rely on GPT-4o's 's visual understanding analyze the visual information in the frames. Besides, augment LMMs' visual understanding via some tools, like template matching (Brunelli, 2009), Grounding DINO (Liu et al., 2023), and SAM (Kirillov et al., 2023), to provide additional grounding for object detection and segmentation. Some visual prompting tricks, like drawing axes and colorful directional bands, are also applied to enhance the GPT-4oo's 's visual ability.
Task Inference. After reflecting on the outcome of the last executed action, We let GPT-4o analyze the current situation to infer the most suitable task for the current moment and estimate the highest priority task to perform and when to stop an ongoing task and start new one.
GPT-4o is required to strictly follow the provided interfaces and examples to generate the corresponding code for new skills. Moreover, GPT-4o is required to include documentation/commeen functionality of each skill. Procedural Memory where skills are stored will then check whether the code is valid, whether the format of documentation is right, and whether any skill with the same name already exists. If all conditions are passed, the newly generated skill is persisted for future utilization.
Skill Curation.
within the generated code, delineating the
Action Planning. GPT-4o needs to select the appropriate skills from the curated skill set and instantiate these skills into a
peppeescrptllagee dee htt///ttooductt on han/AStRRiinte. innrootoroo.o htmll
20

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
sequence of executable actions by specifying any necessary parametric aspects (e.g. duration, position, and target) according to the current task and history information. The generated action is then fed to the Executor for interaction with the environment.
Prompts. Prompts used by each module are initialized by the corresponding templates in Markdown-style format. These prompt templates provide a minimal workflow with basic rules for the module to run and use placeholders of each key for input and output. CRADLE automatically retrieves the corresponding value for each key in the input from the episodic memory and forms valid requests to query LMMs with the values and templates. After receiving responses from LMMs, CRADLE automatically extracts the keys in the output and stores them in the episodic memory. Users can freely customize their own prompts without writing any code.
Apply to new environments. Theoretically, CRADIEE can be directly deployed to new video games or other software applications with the default prompt templates and empty procedural memory. Due to the limited ability of current LMMs and the complexity of challenging environments and tasks, prompt engineering may need to be applied to every module to enhance LMMs' reasoning ability and introduce domain knowledge. Additional tools can also be applied to provide extra grounding and domain knowledge as part of the prompt input. Procedural memory can be initialized with hand-craft skills to mitigate the incomplete tutorials provided by the software and the complexity of tasks. Users may need analyze the task- sask-specifie issue and choose suitable solution. We provide all the implementation details for each video game and software in Appendices E to K.
F. Red Dead Redemption II
F.1 Introduction to RDR2
Red Dead Redemption (RDR2) is an epic AAA Western-themed action-adventure game by Rockstar Games. As one of the most famous and highest-selling games in the world, it widely acknowledged for its movie-like realistic scenes, rich storylines, and immersive open-enden world. The game applies a typical role-playing game (RPG) control system, played from first- or third-person perspective, which uses WASD for movement, mouse control for view changing, first- or third-person shooting for combat, and inventory and manipulation.
For most of the game, players need to control the main character, Arthur Morgan, upon choosing to complete mission scenarios following the main storyline. Otherwise, they can freely explore the interactive world, such as going hunting, fishing, chatting with non-player characters (NPCs), training horses, witnessing or partaking random events, and participating in side quests. As the main storyline progresses, different skills are gradually unlocked. As close-source commercial game, no APIs are available for obtaining additional -nteeinternal information nor pre-defined automation actions. Following its characteristics, this game serves as fitting and challenging environment for the GCC setting and a comprehensive benchmark for embodiment.
E2. Objectives
In Chapter 1 of RDR2, the first two missions of the main storyline are Outlaws from the West and Enter, Pursued by Memory. These missions serve as the tutorial content for RDR2, guiding players step-by-step into the role of Arthur. They immerse the player in the story's 's development while teaching the game's controls and mechanics.
We divided Mission I and Mission into 8 and 5 tasks respectively based on the checkpoints within each mission. Each checkpoint may present failure scenarios. For example, in Mission 1, there are six failure scenarios: i) Assaults, kills, or abandons Dutch or Micah; ii) Allows Dutch Micah to be killed; iii) Abandons the homestead; iv) Assaults, kills, or abandons their horse; v) Assaults, kills, or abandons the horse in the barn; vi) Dies. We categorized each sub-task as either "Easy" or "Hard" based on the likelihood of failure at each checkpoint and the need to retry the checkpoint.
To evaluate CRADLE'S capabilities in an open-world environment, Mission 3 is designed as hard open-ended task. Unlike the first two tutorial missions, it does not include any checkpoints. Consequently, the entire Mission 3 is treated as a single, comprehensive task. Although we do not subdivide Mission 3 into finer tasks, we aim to identify key points to facilitate a clearer understanding of Mission for the reader.
Tables 7 and 8 provide a brief introduction of each task in the first two missions of the main storyline and an open-ended mission, along with approximate estimates of their difficulty. Due to GPT-4o's 's poor performance in spatial understanding and finemmanpulation skills, it can be challenging for our agent to perform certain actions, like entering or leaving a building,
21

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Spider Gorge
LENTINE
ELD
Mission 1: Outlaws from the West
Mission 2: Enter, Pursued by Memory
Open-cnded Mission: Buy Supply
Success Raie (%)
100-
80-
60
40
20
Follow Dutch
Go to Town
Hitch Horse
Protect Dutch
Search Supplies
Go Barn
Search Barn
Lead Horse
Follow Javier
Search John
Keep Wolves away
Kill Wolves
Return to Camp Buy Supply
Figure 11: Trajectory and success rates of 13 main storyline tasks and open-world task in RDR2. Each task is 5 times and cach trial run for at most 500 steps. Long horizontal and challenging tasks like Protect Dutch and Search for Supplies usually need several times of retry to complete, resulting in the demand for more steps. It explains the low success rate of these tasks within 500 steps.
or going to precise indoor locations to retrieve specific items. Additionally, the high latency of GPT-4o's responses also makes it harder for an agent to deal with time-sensitive events, e.g., during combat.
E3. Implementation Details
Our experiments are based on the latest version of RDR2, 'Build 1491.50', As shown in Figure 15, strictly following the GCC setting, our agent takes the video of the screen as input and outputs keyboard and mouse operations to interact with the computer and the game. An observation thread is responsible for the collection of video frames from the screen and each video clip records the whole in-game process since executing the last action.
Information Gathering. To extract keyframes from the video observation, we utilize the VideoSubFinder tool 9, professional subtitle discovery and extraction tool. These keyframes usually contain rich meaningful textual information in the game, which are highly relevant to the completion of tasks and missions (such character status, location, dialogues, in-game prompts and tips, etc.) We use GPT-4o to extract and categorize all the meaningful contexts in these keyframes and perform OCR, and call this processing "gathering text information". Then, to save interactions with GPT-4o, we only let GPT-40 provide a detailed description of the last frame of the video.
While GPT-4o exhibits impressive visual understanding abilities across various CV tasks, we find that it struggles with spatial reasoning and recognizing some game--ppeiff icons. To address these limitations, we add visual augmentation sub-module within our Information Gathering module. This augmentation step serves main purposes: i) utilize Grounding DINO (Liu al., 2023), an open-set object detector, to output precise bounding boxes of possible targets in an image and serve as spatial clues for GPT-4o; and ii) perform template matching (Brunelli, 2009) to provide icon recognition grounding truth for GPT-4o when interpreting instructions or menus shown on screen. As LMM capabilities mature, it should be possible to disable such augmentation.
Seff-Reflection The reflection module mainly serves to evaluate whether the previously executed action was successfully carried and whether the current executing task finished. To achieve this, we uniformly sample most sequential frames from the video observation since the execution of the last action and use GPT-4o estimate the success of its execution. Additionally, we expect GPT-4o can also provide analysis for any failure of the last action (e.g., the move-forward action failed and the cause could be the agent was blocked by an obstacle). With such valuable information as input for Action Planning, including the failure/success of the last action and the corresponding analysis, the agent is capable of attempting to remedy an inappropriate decision or action execution.
Moreover, some actions require prolonged durations, such as holding down specific keys, which can coexist or interfere with other actions decided by subsequent decisions. Consequently, the reflection module must also decide whether an ongoing
VVidenSubFinder standalone tool wttphhtt////eett http:///////oooooogeee/ooo htwpa//wwwoo....eoopp//attttttta
22

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Table Tasks in the first two missions of RDR2. the tutorial guide, the prompt text Start Dialogue signifies end the previous checkpoint and the beginning of the current checkpoint. Difficulty refers to how hard to accomplish the corresponding tasks. Figures 12 and 13 showcase snapshots of each task (specific figures marked in parenthesis in the table). The maximal number of steps (agent takes one action) for each task is 500.
Mission I: Outlaws from the Description
Start Dialogue
Difficulty
West
Follow Dutch (Fig. 12a)
Arthur follows Dutch horseback into the snow to find their scouting gang members.
Use [W] to Follow Dutch
Easy
Go Town (Fig. 12h)
Arthur rides his horse, following Micah to the vicinity of a little homestead Micah discovered.
Hold [W] match speed
Fasy
with Dutch and Micah
Hitch Horse (Fig. 12c)
Arthur hitches the horse the hitching post, then goes the old shed and takes cover.
Hold [F] to hitch your horse
Fasy
Protect Dutch (Fig. 12d)
Arthur uses his gun to shoot all of the O'Driscolls inhabit- ing the housc and protect Dutch.
Use [W] peak out of cover
Hard
Scarch for Supplies (Fig. 12c)
Arthur follows Dutch to the house scarch supplies.
Hold [R] near items to
Hard
pick the up while scarching house.
Go to Barn (Fig. 12f)
Arthur follows Dutch's 's directions and goes to the barn to see if there's 's anything inside. Arthur searches the barn and defeats the O' D'riscolll hiding inside.
Dutch: Micah, Arthur, keep looking for stuff [F] Attack the O' Driscoll
Easy
Search Barn (Fig. 12g)
Hard
Lead Horse (Fig. 12h)
Arthur calms the horse and takes it out of the barn.
Hold [Right Mouse Button]
Easy
to focus on the horse
Mission 2: Enter, Pursued by Memory
Description
Start Dialogue
Difficulty
Follow Javier (Fig. 13a)
Arthur rides his horse following Javier up the mountain
Follow Javier
Hard
through the blizzard searching for John's 's trail.
Search John (Fig, 13b)
After dismounting, Arthur followed Javier over slopes and ledges to find John and carry him away.
Javier: Down this way
Hard
Keep Wolves away (Fig.
Arthur manages to shoot all of the wolves before they can attack Javier and John.
Keep the wolves away from Javier and John
Hard
13c)
Kill Wolves (Fig. 13d)
Three people ride horses down the mountain. Arthur elim- inate the wolves, protecting Javier and John ahead. Arthur followed Javier horseback back camp.
Javier: Come on, let's get
Hard
back to the others
Return Camp (Fig. 13e)
Yea... c' c'mon.
Let's push
Easy
hard and get back
action should continue to be executed. Furthermore, self-reflectio can be leveraged to dissect why the last action failed to bring the agent close to the target task completion, better understand the factors that led to the successful completion of the preceding task, and so on.
Besides, we observe that instead of providing GPT-4o with sequential high nigh-resolution images for self-reflection, low- resolution images make it easier for GPT-4o understand the relation among the sequential screenshots and capture dynamic changes, resulting in significantly higher success rate of detecting whether the action is executed successfully and take any effect. We hypothesize that since high-resolution image can cost as many as 2000 tokens, too many high- righuresolution images make GPT-4o fail to capture the overall changes across screenshots and be caught up in the local details.
Task Inference. During gameplay, we let GPT-4o propose the current task to perform whenever it believes it is time to start a new task. GPT-4o also outputs whether the task is long- or short-horizon task when proposing a new task. Long-horizon location, typically require multiple iterations, whereas short-horizon tasks, like picking up an
tasks, such as traveling item conversing with someone, involve fewer iterations. The agent will follow the newly generated task for the next 3 interactions. After interactions, the agent returns to the last long-horizon task in the stack. Deciding on binary task horizon is much easier and more robust for GPT-4o, than re-planning at every iteration. Since a long-horizon task frequently includes multiple short-horizon sub-tasks, this implementation also helps avoid forgetting the long-horizon tasks under execution.
Skill Curation, As shown in Figure 17, during gameplay, instructions often appear on the screen, such as "press [Q] to take
23

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Table Key points in the open-ended mission, Buy Supply RDR2. Figure showcases snapshots key points (specific sub sub-figures marked in parenthesis in the table).
Mission 3: Buy Supply
Description
Find Horse (Fig. 14a)
Find and mount the horse in the camp.
Prepare to Navigate to Saloon (Fig. 14b)
Open map, find the saloon and create waypoint.
Go to Saloon (Fig. 14c)
Ride horse to the saloon.
Prepare to Navigate to Shop (Fig. 14d)
Open map, find the general store and create waypoint. Ride horse to the shop.
Go to Shop (Fig. 14c)
Enter Shop (Fig. 14f)
Dismount the horse and enter the shop.
Talk to Shopkeeper(Fig. 14g)
Approach the shopkeeper and talk. Open the menu, find and buy the target product.
Buy Target Product (Fig. 14h)
over" and "hold [TAB] to view your stored weapons", which serve as essential directives for completing current and future tasks proficiently. To save interactions with GPT-4o, we implement a simple version of this module inside Information Gathering to reduce interactions with GPT-4o. When GPT-4o detects and classifies some instructional text in the recent observation, which usually contains key and button hints, it will directly generate the corresponding code and description.
Action Planning. Upon execution of this module, first retrieve the top k relevant skills for the task from procedural memory, alongside the newly generated skills. We then provide GPT-4o with the current task, the set of retrieved skills, and other information collected in Information Gathering that may be helpful for decision-makes making (e.g., recent screenshots with corresponding descriptions, previous decisions, and examples) and let it suggest which skills should be executed. We also request that GPT-4o provide the reasons for choosing these skills, which increases the accuracy, stability, and explainability of skill selection and thus greatly improves framework performance. While GPT-4o sometimes may generate a sequence of actions, we currently only execute the first one, and perform Self- Reff-Reflection, since we observe a tendency for the second action usually suffer from severe hallucinations.
Action Execution. Unlike the conventional mouse operation in standard software, where the cursor is restricted to 2D grid and remains visible on the screen to navigate and interact with elements, the utilization of the mouse in 3D games like RDR2 introduces a varied control scheme. In menu screens, the mouse behaves traditionally, offering familiar point-and-chee functionality. However, during gameplay, the mouse cursor disappears, requiring players to move the mouse according to specific action semantics. For example, to alter the character's viewpoint, the player needs to map the actual mouse movement to in-game direction angle changes, which differ in magnitude in the X and axes. Another special transition applies shooting mode, where the front sight is fixed the center of the screen, and players must maneuver the mouse align the sight with target enemies. This nuanced approach to mouse control in different contexts adds extra layer of challenge to general computer handling, showcasing the adaptability required in game environments, compared to regular software applications.
Procedural Memory. In our target setting, We intend to let the agent learn all skills from scratch, to the extent possible for the main storyline missions. The procedural memory initialized with only preliminary skills for basic movement, which are not clearly provided by the in- ta-game tutorial and guidance.
turn(degree), move move_ooreeeeeeeeee forweffowrrddddu duration): Since the game does not precisely introduce how to move in the world through in-game instructions, we provide these two basic actions in advance, so GPT-4o can perform basic mobility, while greatly reducing the number of calls to the model.
shoot(x, y): RDR2 also does not provide detailed instructions on how to aim and shoot. Moreover, due to limitations with GPT-4o spatial reasoning and the need to sometimes augment images with object bounding boxes, we provide such basic skill for the agent to complete relevant tasks.
sedeet_jteem_aee _a(xx y): Similarly to shoot(), due to the lack of instructions, we provide such skill for the agent to move the mouse to a certain place to select given item.
Beyond these basic atomic low-level actions, we introduce few composite skills to facilitate the game playing progress.
24

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
(a) Follow Dutch
(b) Go to Town
(c) Hitch Horse
(d) Protect Dutch
(c) Search for Supplies
Go to Barn
(g) Scarch Barn
(h) Lead Horse
Figure 12: Image examples of tasks in the first mission Outlaws from the West, (The picture has been brightened for casier reading.)
The agent should be able complete tasks using only the basic skills above and the skills it learns, but these composite skills streamline the process by greatly reducing calls to the backend model.
turn_and iii_and_mov lat__mov__ooor forvardddrrddeeeer.e duration): This skill is just simple composition of turn() and move_forvard() to save frequent calls to GPT--4o in a common sequence.
follow duraatin) and navigate (red lines) in the minimap to certain locations. This can reliably accomplished via the basic movement skills, but requires numerous interactions with GPT-4o. To control both cost and time budgets involving GPT-4o's responses, we leverage the information shown in the minimap to implement composite skill to follow target NPCs or red lines for short set of game iterations. The default duration is 20 iterations. Increasing the duration the performance in task Follow Dutch, Follow Javier and Killing Wolves but significantly decrease the success rate of Search John since this task requires frequent exchange of the skills between climbing and following.
path(duration): navigate_pathhduratieeeee
In RDR2, tasks often guide players to follow NPCs or generated paths
dramatically improve
fight(): As output of an interaction with GPT-4o, the agent will only take one action per step. However, though the action is generated correctly. specifically in fight scenarios, the action frequency may not be high enough to defeat an opponent. In order to allow sub-second punches, we provide pre-defined action that wraps this multi-action punching, which can be selected by GPT-4o to effectively win fights.
For the open-ended mission, since the agent skips all the tutorials in Chapter we provide all the necessary skills in the procedural memory at the beginning of the mission.
Episodic Memory. This module stores all the useful information, e.g., input and output of GPT-4o. In each iteration, after the self-reflection, we will request GPT-4o to summary the event that happened in the last action and the past experiences.
Game Pause. To prevent in-game time from passing in real-time games like RDR2, we have to pause the game while waiting for LMMs' response. The time interval between two consecutive actions can be long as one minute. RDR2,
25

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
(a) Follow Javier
(h) Search John
(c) Keep Wolves Away
(d) Kill Wolves
(c) Return to Camp
Figure 13: Image examples of tasks in the second mission of Enter, Pursued by Memory.
INEE
INE
(a) Find Horse
(b) Prepare Navigate Saloon
(c) Go to Saloon
INDEX
TIINE
(d) Prepare to Navigate to Shop
(c) Go to Shop
(f) Enter Shop
DANGER
(g) Talk to Shopkeeper
(h) Buy Target Product
Figure 14: Image examples of key points in the open-ended task of Buy Supply.
26

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Self-Reflection
Task Inference
Skill Curation
1. The previous action check shire( took effect since the sccondary menu of further actions appeured. 2. The task of leading the horse is not completed yet.
def show (nfo(duuttion))
Show inforeation.
Add/Update
press press_key( key duration)
"Lead the horse"
def lead honse
Leed the horse.
Gemerated
press_key("E), duration)
def turn (degreee:
+ Turn the character certain angle.
move_mousee (ove_mouse(degree)
LEAD
Episodic Memory
Procedural Memory
def move move_ (ow Move ahead for duration time.
Retrieved
press_key( (W", duration)
Memory
RED DEAD REDEMP TION
II
Observe
Execute
Lead the horse
_ead_ aaadhhossedduatiooee
Information Gathering
Environment
Action Planning
Figure 15: The detailed illustration of how CRADLE is instantiated game agent play RDR2.
after the agent finishes executing outputted actions, will be automatically pressed pause the game and when the agent determines the next action, esc will be automatically pressed again unpause the game. Note that there will be animation lasting up to 0.5 seconds for both pausing and unpausing. During this animation, we not control the character, but the dynamics of the game world keep changing, e.g., the wolves are still moving. It introduces additional challenges for the tasks that require precise timing, like combat.
F.4. Case Studies
Here we present a few game-specific case studies for more in-depth discussion of the framework capabilities and the challenges of the GCC setting.
1.4.1. SELF-REFLECE REFLECTION
Self-reflection an essential component CRADLE as it allows our framework reasoning to correct previous mistakes or address ineffective actions taken in-game. Figure 16 provides example of the self-refiection module. The task requires the agent select weapon equip, in the context of the "Protect Dutch" task. Initially, the agent selects knife as its weapon by chance, but since the game requires gun to be chosen, this is incorrect and the game still prompts the player to re-open the weapon wheel. The self-reflection module is able to determine that the previous action was incorrect and on a subsequent iteration the agent successfully opts for the gun, correctly fulfilling the task requirement and advancing to the next stage in the story.
F.4.2. SKILL CURATION
For skill curation, we first provide GPT-4o with examples of general mouse and keyboard control APIs, e.g., io jo_eev.key__..o..ovooo _press and io enw.mouse click. Figure 17 shows that GPT-4o capture and understand the prompts appearing screenshots, i.e, icons and text, and strictly follow the provided skill examples using our IO interface to generate correct skill code. Moreover, GPT-4o also generates comments in the code to demonstrate the functionality of this skill, which are essential for computing similarity and relevance with a given task during skill retrieval. The quality the generated comment directly determines the results of skill retrieval, and further impacts reasoning to action planning. Curation can also re- re-generate code for given skill, which is useful GPT-4o wrongly recognized key or mouse button in a previous iteration.
27

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Task Inference
The frame does not change, therefore, the task is not finished yet. 2. The selected weapon is wrong, therefore, need change the weapon.
Holld slow the Wasper Wherl
Skill Curation
Hold Tab show the Weapon Wheel.
Task Inference
Execute
Action Planning
Hold Tab to show the Weapon Wheel.
show weapon wheel()
Skill Curation Retrieved Skills: ('aim', 'follow', 'tum", "move forward", "show weapon wheel", etc.)
Execute
Action Planning
show weapon wheel()
2
Task Inference
Use [Mouse] to select weapon and release [TAB] to equip.
Skill Curation
Retrieved Skills: ('aim', 'sclect weapon at', "move forward', "show weapon wheel', etc.]
Task Inference
Use [Mouse] to select a wespoon and zelease [TAB] equip.
Action Planning
select wwwpwn_atteeeee (8.55, 0.27)
Skill Curation
Decision Reasoning: The weapon in
def select ceccct_weaoo___t__eee axxx, y):
Execute
green box slightly more relevant to the target the last self-reflection.
Move the wouse select the weapon luveennoousee www.smuuuuuueeeee. y)
Retrieved Skills: ['aim', "follow", 'tum", "move forward', "show_weappne weapon wheel', etc.
6
Action Planning
select wwwpon_atteeteeeeeeeeeeeeeeeeeae (8.6, 68)
5
Execute
Decision Reasoning: The red box selected it is near the currently selected weapon.
Self-Reflection
1. The last executed action was to select a weapon,
and executed successfully.
2. In this frume, the ammo count is 6 in the upper
Huild
Wheel
right corner, so the character is ready to fire.
Figure 16: Case study of self-reflection on re-trying failed task. Task instruction and context require the agent to equip the gun. A wrong weapon (knife) is first selected, but the agent equips the after self-reflectin Only relevant modules are shown for better readability, though all modules (Figure 4) executed per iteration.
28

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
press
horse
vlaw your storad wargona
WEAPONS
HORSE f
WARKER
def take_ cover (duratiio
def focus_on, ppccu__on_horseeee on_hoose()::
def view_stored, Stee__Stored_vessposss)
press "Q" to take cover
hold "right nouse buttonn to focus on horse
press "tab" to view your stored weapons
je_env.kyy_resss"" '4', duration)
so_env.nouse_ www.sooouseuoeee""ee holdd""ighhh button')
to_ to_env.key_halde (aaa(dtaad"""
Figure 17: Skill code generation based on in game instructions. As the storyline progresses, the game will continually provide prompts on how to use new skill via keystrokes utilizing the mouse.
F.4.3, ACTION EXECUTION AND FEEDBACK
Proper reasoning about environment feedback critical due the generality of the GCC setting and the level of abstraction to interact with the complex game world. The semantic gaps between the execution of action, its effects in the game world, and observing the relevant outcomes for further reasoning lead to several potential issues that CRADLE needs to deal with. Such issues can be categorized into four major cases:
Lack of grounding feedback. In many situations, due to the lack of precise information from the environment, it can be difficult for the system to deduce the applicability or outcome of given action. For example, when picking an item from the floor, the action may fail due to the distance to the object not yet being close enough. Or, if within pick up range, the chosen action may not exactly apply due to other factors (e.g., character's package is full).
Even if the right action is selected and executed successfully, the agent still needs figure out its results from the partial visual observation of the game world. If the agent needs to pick or manipulate an object that is occluded from view, the action may execute correctly, but no outcome can be seen.
representative example in RDR2 happens when the agent tries to pick up its gun from the floor after a fight. Getting to the right distance, without completely occluding the object, can lead to multiple re-trials. Figure 18a showcases a situation where, though the character is already standing near the gun (as seen in the minimap), it's still not possible to pick it up.
Previous efforts (Wang et al., 2023b; 2024a) that utilize in-game state APIs unreasonably bypass such issues by leveraging internal structured information from the game and the full semanties of responses (data) or failures (error messages).
Imprecise timing in 10-level calls. This issue is caused by the ambiguity in the game instructions or differences in specific in-game action behaviors, where even the execution of a correct action may fail due to minor timing mismatches. For example, when executing an action like 'open cabinet', which requires pressing the [R] key on the keyboard, if the press too fast, no effect happens in the game world. However, as there no visual change in the game nor other forms of feedback, it can be difficult for GPT-4o figure out if an inappropriate action was chosen this game state if the minor timing factor is on screen), but this is easily missed and any key release before the circle completes also results in no effect. Figure 18b illustrates the situation.
the problem. Pressing the key for longer triggers animation around the button (only if the helper menu
The same problem also manifests in other situations in the game, where pressing the same key for longer triggers a completely different action (e.g., lightly pressing the [Left Alt] key vs. holding for longer).
Change in the semantics of key and button. somewhat similar situation occurs when the same keyboard key or mouse button gets attributed different semantics in different situations (or even in multi-step action). GPT-4o may decide to execute given skill, but the original semantics no longer hold. The lack of in-game effect parallels the previous situations. Worse yet, an undesired effect will confuse the system regarding the correct action being selected or not.
For example, when approaching a farm in the beginning of the game, the agent needs to hitch the horse to a pole to continue. The operation to perform the action consists of pressing the [E] key near a hitching post (as shown in Figure 18c). However, the same [E] key press is the only constituting step other actions with different semantics, like dismount the horse or open
29

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
OPEN CARMEET
NIBEINE
OPEX CARINCT
MEDICNEE CHEST
tach your here
(a) PPick gun' unavailable
(b) 'Open cabinet' press timing
(c) 'Hitch horse" re-use of (E] key
Figure 18: Examples of action execution uncertainty. Lack of environmental feedhack to actions and semantic gaps hetween action intent
and game command lead to challenging situations for agent reasoning.
the door, Wrongly triggering horse dismount the situation shown in the figure can lead to undesired side effects, i.e., may mislead the system about the actual effects of the action or affect the planning of which next actions perform.
Interference issues. I. Lastly, completion of some actions requires the correct execution of multiple steps sequentially, which could be interrupted in many ways not related to the agent's own actions. Without the use of APIs that expose internal states or other forms feedback, it is much harder for the agent to decide when to repeat sub- actionsion or try different strategies. For example, if the agents gets shot and loses aim while in combat, or an unrelated in in-game animation is triggered mid-action, canceling it.
Since there is no direct environment feedback, the agent needs carefully analyze the situation and try to infer if any action step needs re-execution.
E55 Limitations of GPT-40 and GPT-4V
Deploying CRADLE in complex game like RDR2 requires the backbone LMM model to handle multimodal input, which revealed several limitations of both GPT-4V and GPT-4o, necessitating external tools enhance overall framework performance. Initial tests and exploration were performed using GPT-4V, as GPT-4o was not yet available. These tests highlighted significant weaknesses in spatial perception, icon understanding, history processing, and world understanding. Upon the release of GPT-4o, further testing demonstrated some notable improvements in spatial perception. However, enhancements in other areas remained marginal, while some regressions were also observed, all indicating the need for additional tools aid dpcision-making
Spatial Perception. As shown in Figure 19a and 20a, GPT-4V's spatial-visual recognition capability is insufficient for precise fine-grained control, particularly in detecting whether the character is being or going to be blocked and in estimating the accurate relative positions of target objects. In contrast, GPT-4o exhibits significant enhancement in spatial perception, capable of recognizing obstacles ahead and estimating the approximate relative positions between objects. However, both models require supplementary information, such as bounding boxes of potential target objects, to make fine-grained decisions. These led to the need to augment certain images to provide auxiliary visual clues for decision-making, i.e.,
bounding boxes of possible target objects.
Icon Understanding. Both GPT-40 and GPT-4V struggle with domain-specifer concepts, such as unique icons within the game, which may represent specific targets refer to certain mouse and key actions. As shown in Figure 19b and 20b, GPT-4V and GPT-4o fail to recognize the left shift, right mouse button, and space icons. Attempts to incorporate few-shot learning to improve image understanding cannot be generalized. Therefore, we match prepared pattern templates, egg., icon images, against each screenshot to continuously detect and highlight any appearing icons.
Figure 19c and Figure 20c also demonstrate that although GPT-4o performs better than GPT-4V in understanding the mini- map, still fails consistently interpret this crucial information regarding the position and direction of the character. This failure in localization leads the agent to sometimes get lost in the town and miss the task target. While the aforementioned issues can be slightly alleviated by providing additional few-shot examples, significant improvement is only achieved by cropping the image and providing GPT-4o with the exact region containing the icon to be recognized. This dependency on precise input makes the issue challenging and unreliable for decision- decision-making, Although the above issues can be slightly alleviated by providing additional few-shot examples, can only have an obvious effect if we crop the image and provide the GPT-4o with the region exactly containing the icon to be recognized, which makes the issue intractable.
30

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
BHEAD
810
ChatGPT
ChatGPT
There are immediate obstacles directly in front of the character in the image. The character has a clear path ahead on a muddy street what appears to be Westem-style town setting.
The shopkeeper in the image is the right side, behind the counter.
(a) GPT-4V fails to recognize obstacles in the environment and the position of the shopkeeper.
ChatGPT
ChatGPT
Based the provided game screenshot, I've
From the provided image screenshot from the game, can identify the following keyboard and mouse
identified the following control prompt:
"Hold L to ride faster"
control prompts:
This control suggests that holding certain key (which seems to be an upward arrow key, often associated with the "W" key for forward movement
1. "Toggle View" The key for this action is "V".
2. "Zoom" The key for this action is "O"
3. "Flip" The key for this action "L".
many games) will make the character ride faster.
"Back" The key for this action is "ESC".
(b) GPT-VV struggles to recognize the icons for keys keyboard and mouse buttons.
ChatGPT
ChatGPT
The central white arrow is facing towards the left.
The central white arrow with the white arrowhead is facing towards the left.
(c) GPT-4V cannot understand the correct direction of arrow points (character heading) in the mini-map.
Figure 19: Example situations of GPT-4V's limitations in understanding visual information from the game.
31

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
BREAD
$ 810
ChatGPT
ChatGPT
There an obstacle front the character. It uppears to be small wooden railing or barrier along the edge of the platform. The character would need to navigate around it find way over it to proceed forward.
The shopkeeper in the image is the left side
(a) GPT-4o recognize obstacles in the environment and the position of the shopkeeper.
ChatGPT
ChatGPT
Based on the provided imuge, the control prompt shown is:
From the provided image screenshol from the game,
I can identify the following keyboard and mouse
"Hold to ride fäster'.
control prompts:
The 'L' refers to the left analog stick on controller, indicating that the player should hold the left analog stick in forward position to increase the spoed of their character's movement, typically while riding a horse.
1. Toggle View:
2. Zoom: Mouse Scroll Wheel
3. Flip: R
4. Back: Esc
5. Photograph: (No specific key ussigned, just
label)
(b) GPT-4o struggles to recognize icons keys on keyboard and mouse buttons.
ChatGPT
ChatGPT
The central white aroow in the mini-map is facing to the right.
The central white arrow on the mini-map is facing
(c) GPT-4o occasionally accurately determines the direction of arrow points (character heading) in the map.
Figure 20: Example situations of GPT-4o's capabilities understanding visual information from game.
32

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
History Processing. Moreover, both GPT-4o and GPT-4V can easily get distracted by irrelevant information in longer contexts, resulting in hallucinations. For example, when action planning utilizes too many historical screenshots, they may confuse past and present frames. Additionally, performance fluctuates and both model versions frequently generate output not adhering to the rules in the provided prompts. To mitigate the issue of hallucinations, we more strictly control input information by further summarizing long-term memory.
World Understanding, Lastly, the absence an RDR2 world model limits GPT-4V and GPT-4o's understanding of the consequences of its actions in the game. This often results in inappropriate action selection, such as overestimating the necessary adjustments for aligning targets or misjudging the duration required for certain actions. To alleviate this problem, we introduced extra prompt rules regarding action parameters and more flexibility into the self-reflectio module.
G. Stardew Valley
G.1. Introduction to Stardew Valley
Stardew Valley is an open-ended country-life RPG game developed by ConcernedApe, which has a 98% positive rating on Steam and is rated as Overwhelmingly Positive, Players take on the role of a character disillusioned with city life who inherits a dilapidated farm from their late grandfather. Initially, the farmland is overrun with boulders, trees, stumps, and weeds, which players must clear to make way for crops, buildings, and placeable items. The main goal is to restore and expand the farm through activities such planting crops, raising animals, mining, fishing, and crafting. Additionally, players can interact with NPCs in town, forming relationships that can lead to marriage and children. Players complete quests for money or to restore the town's Community Center by completing "bundles," which reward items like seeds and tools and unlock new areas and game mechanics. All activities are balanced against the character's 's health, energy, and the game's s clock. Food provides buffs, health, and energy. The game features a simplified calendar with four 28-day months representing each season, affecting crop growth and activities. Compared to RDR2, this game is more lightweight and easy control. This game features wealth of production and social activities, presenting a comprehensive test of an agent's 's abilities, which is an ideal platform to observe and evaluate agents' comprehensive behaviors and abilities, like in the Generative Agents (Park et al., 2023). We use the latest version (1.6.) of the game to conduct all the experiments.
G.2. Objectives
We find that GPT-4o surprisingly struggles with accurately recognizing and locating objects near the player in this 2D game. This leads to difficulties for the agent to interact with objects or people, as it requires the player to stand precisely in front of them in the grid (e.g., when entering doors, using pickaxe to break stones). Even some basic tasks are already challenging enough for current agents in this game. Therefore, as shown in Figure 21, we evaluate three essential tasks in the early stages of the game:
Farm Clearup. Clear the obstacles on the farm, such as weeds, stones, and trees, much as possible to prepare for farming. This task requires agents to move precisely to be in front of the obstacles, identiffy the type of obstacles correctly and select corresponding tools to deal with them.
Cultivation. Use the hoe to till the soil, use a parsnip seed packet on the tilled soil to sow a crop, water the crop every day and harvest at least one parsnip. This task requires long-horizontal memory and reasoning.
Shopping. Go to the general store in the town, which on the other map. to buy more seeds and return home. This task is used to evaluate agents' long-distance navigation ability.
For each task, the maximal steps is 100.
G.3. Implementation Details
Visual Prompting. As cartoon-style pixel game, the game screen of Stardew is quite different from the real world. Although GPT-40 can observe coarse-grained information from screenshots, more fine-grained information is required to complete tasks. Therefore, as shown in Figure 22, we divide each screenshot into 3 x grids and require GPT-4o to describe the screenshot in a wrid-by-grid format. We empirically find that it can result in a more precise and accurate description. And GPT-4o can also make better control based on the grids. In addition, we also augment the image with two blue and yellow
33

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
(a) Before Farm Clear-up
(b) After Farm Clear-up
Parasip Seeds
Bosn Starter
60
Gaaliflouor Souds
80
Welcome
Parrell bf Don't
Potate Saeds
forget dady
wallpaper and
500
(c) Cultivation
(d) Shopping
Figure 21: Three tasks in Stardew Valley.
bands on the left and right sides, respectfully, with the prompt, "The blue band represents the left side and the yellow band represents the right side". Our empirical results show that this method significantly improves GPT-4o's 40's ability to accurately distinguish left from right.
Information Gathering, As mentioned in the introduction of visual prompting, we let GPT-4o describe the image grid by grid, which is helpful in locating the position of the character, surrounding objects and buildings and facilitates the understanding of the relative positions among them for GPT-4o. Besides, while compared to GPT-4V, GPT-4o is able to recognize most of the icons and their quality in the toolbar shown at the bottom of the screenshot, GPT-4o cannot output the items in the inventory sequentially one by one as it always skips a few in between. We have to clip the box for each item out of the toolbar and feed them to GPT-4o independently, augmented with template matching, for recognition, which turns out to be more accurate. The success of recognition of the tools in the toolbar is critical to tasks like Farm Clearup and Cultivation.
The duration of actions in Stardew is usually much shorter than in RDR2, so we only use the first and last frame from the video observation to reduce the number of tokens used per request. Additionally, we provide some helpful prior information for GPT-4o. For example, a screenshot of the inside of the store is provided to check whether the store was successfully entered. This is useful because there are many other buildings near the store, and sometimes GPT-4o controls the character to enter the wrong one. However, this is not realized if the screenshot is not provided.
Self-Reflection.
Skill Curation. For skill curation, as mentioned in Figure 5, we mainly rely on the in-game manual generate atomic skills, like move_up(), do ___atiitoon and use_ use_tool(). In addition, handle the challenges of locating objects, especially doors, we have special set of composite skills specifically for Stardew. e.g., go through_ _dor, buy ___tem,, get_out of house and enter_ dooor_and, _leep. With the restrictions of GPT-40 in fine-grained control, we designed go_ pop_thhough__ee goggg_hdorrrr composite skills for the agent to control the game character to accurately reach various doors and successfully enter, such as the house and the store door, and in order to buy certain items such as parsnip seeds, we designed the composite skills buy jitmmtmm to control the game character to interact with the salesman and buy parsnip seeds. similarly, we designed the get_ get_outt_rr of house and enter door door_and sleep composite skills to accurately exit the house from the bed and enter the house and walk to the bed.
34

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Thu
1:10 pm
(4.1)
C 300
(1.2)
(2.2)
3.2
(5.2)
(1.3)
(2:3)
(4.3)
373
Figure 22: Augmented screenshot via visual prompting. The full sereenshot is divided into x grids and each grid has a unique white coordinate. Additionally, we augment all input images with color bands, with the prompt, "The blue band represents the left side and the yellow band represents the right side", which significantly improves GPT-440's ability to accurately distinguish left from right.
Action Planning. In this game, we let GPT-4o output at most two skills in a single action every time, which turns out to be efficient. The agent usually needs to select the correct tool first and then use the tool or do action.
Procedure Memory. Procedure Memory is used to store and retrieve skills code form. order for agents to quickly get started and complete some special tasks in Stardew, we have predefined skills in Procedure Memory. These skills are divided into atomic and composite skills. atomic skill consists of basic operations such moving, selecting tools, etc. The description of all the atomic skills is listed as follows:
do_action(): The function to perform a context-specific action on objects or characters.
use use__oooe [ee_]oo)(): The function to execute an in- in-game action commonly assigned to using the character's current selected tool.
move_ movv_upp(duiatin):: The function to move the character upward (south) by pressing the "w" key for the specified duration.
move_down( move_doww(duraaton) The function to move the character downward (north) by pressing the "w" key for the specified duration.
move denuuruinn: The function to move the character left (west) by pressing the 'w" key for the specified duration.
noveerighttduuration):: The function to move the character right (east) by pressing the 'w" key for the specified duration.
select_tool(key) The function to select specific tool from the in-game toolbar based on the given tool number.
and the composite skills are designed for the agent to complete variety of special tasks. The description of the composite skills is listed as follows:
jie_ite(): The function to interact with the salesman and buy the item.
enter iiter_dorr_ae _aniannddaa and_sleep((): The function to enter the house and walk to the bed.
get_out_og of house(): The function accurately exit the house from the bed
35

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
ti_thhough_door// The function reach and enter all kinds of doors.
The game will pause automatically when the game window is not focused. So when the character finishes executing actions, we will activate another window, e.g., code window, pause the game and stop the passage of the time in the game.
Game Pause.
G.4. Case Studies
Here we present few game-specific case studies further discuss CRADLE'S self-reflection and task-inference processes the GCC setting.
Task Inference
The charncters' positions in those frames are identical. 2. probable that the character blocked by seen or unseen obstncles.
Skill Curation
Retrieved Skills:
('move left',
"move rrghht" "move _p', 'move down', etc.]
Task Inference
Action Planning
Move exit town.
move up up(duration=0.5
Skill Curation
Decision Reasoning: The most suitable action now is to try moving slightly to fine
Retrieved Skills: Timov_llft'
Execute
tune the character's position and then
move right', 'move up", 'move down', etc.]
attempt move move_left() again required..
Execute
Action Planning
(duradurt)o
Self-Reflection
1. The last executed action was to move and it was executed successfully.
2. In this frame, the character successfully surmounted the obstacle.
Figure 23: Case study of self-reflection on re-trying failed task. Task instruction and context require the agent to exit town. A wrong direction is first selected, but the agent moves after self-reflection. Only relevant madules are shown for better readability, though all modules (Figure 4) are executed per iteration.
G.4.1. SELF-REFLECTION
The Self-reflection module plays an important role in the completion of game missions in Stardew, giving our framework the ability to determine the actions performed are complete and effective and to correct the errors of invalid actions. In the "Purchasing Seeds" task, the Agent is asked to return home from the store after purchasing items. At the "Home on
36

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
the left side of the store" prompt, the Agent controls the character to go left, but there are obstacles to keep going left, and the character must go up to circumnavigate the obstacles. As shown in the Fingure 23, the role will initially be stuck the obstacle and cannot continue to the left. Through Self-Reflection, the Agent can judge that it is currently in state of obstruction, and moving to the left cannot be implemented smoothly. Therefore, the agent can adjust the direction upward to bypass the obstacle and enable the role to continue to the left until it returns home.
G.4.2. INFF-NNEEENNCE
Task Inference is a very effective module for completing game quests in Stardew. Its function is to decompose a vague and grand task into specific sub-task, which effectively guides the Agent to complete the overall task. For example, in the Farming task, as shown in Figure 24, the task that the character needs to complete is "cultivate and harvest a parsnip." This is a complete but vague task. Through the Task Inference module, the Agent breaks down the task into (I) till the soil with the hoe, (2) plant the parsnip seeds, (3) water the planted seeds once daily for four days, (4) harvest the fully grown parsnip. This enables the Agent to know more clearly the steps needed to complete and finish the task successfully.
Informationn Gathering
Task Inference
Action
Planning
Toolbar: Scythe (Selected)
The next subtask till the hoe
select tool(key="""
use tool()
with the
(I) Till the soil with the hoc.
The next step is plant the parsnip seeds.
Toolbar: 2. Hoc (Selected)
select tool(key "6") do actionl)
Cultivate and harvest a parsnip.
(2) Plant the parsnip seeds.
Task Inference
Toolbur: 6. Parsnip Seeds (II) (Selected)
The next step water the planted soods.
select lool(key""3") use (_ool))l)))
(3) Water the planted sceds once daily for four days.
3. Watering the next step is harvest the parsnip.
Toolbar:
de action()
Can (Selected)
(4) Harvest the fully grown parsnip.
Figure 24: Case study of task inference on decomposing task into specific sub-tasks. The complete task is to cultivate and harvest parsnip. CRADLE decomposes the task into four sub-tasks by task inference. Only relevant modules are shown for better readability, though all modules (Figure 4) are executed per iteration.
G.S. Limitations of GPT-40
Fine-grained Control. Stardew Valley requires that players are positioned precisely interact with objects, such as doors and NPCs. However, it is difficult for GPT-4o take a pixel-level precise action. For example, GPT-4o not take precise movement even though the speed at which the figure moves is known. To alleviate this problem, we make some composite skills that use semplate-matching to complete some complex interaction tasks, such as purchasing items.
Perception in a 2D virtual world. In Stardew Valley, it's common for a character to be blocked by rocks or trees, and GPT-4o fails to tell if a character is blocked by looking at the image once, and can't 'tn't predict if the next move will be blocked, which very easy for a human to do by looking at the image. This indicates that GPT-4o is relatively weak in
37

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
perceiving the virtual world in this game. In order to solve this problem, we compare the successive frames before and after in Self-Reflection enable GPT-4o to judge the corresponding changes.
H. Dealer's Life 2
H.1. Introduction to Dealer's Life 2
Dealer's Life 2 is a captivating indie simulation game developed by Abyte Entertainment. Renowned for its intricate negotiation mechanics and humorous portrayal of a pawn shop environment, the game is celebrated for its engaging gameplay that combines strategy with quirky, cartoonish art style. As a simulation game with role-playing elements, Dealer's Life 2 is played from first-person perspective, utilizing mouse for point-and-click interactions and keyboard for price imputs. This interface facilitates item appraisals, customer interactions, and comprehensive shop management.
In the game, players assume the role of pawn shop manager, tasked with acquiring and selling various items to make a profit while managing their store's ' reputation and inventory. Players engage with a wide range of unique non-player characters (NPCs), each with their own distinct behaviors and negotiation styles. Whether bartering over the price of a rare collectible or managing unforeseen shop events, players must hone their haggling and strategic recision-making skills to succeed. Dealer's Life operates in a closed-source format with no APIs available for accessing in game data or automating gameplay functions. This setup ensures hands-on experience where players immersed in the day-to-day challenges of running a pawn shop. This game environment provides unique and entertaining setting for testifying the GCC's haggling and strategic decision-making abilities. We run our experiments using the latest version, V. 1.013 W96 of the game.
H.2. Objectives
We concentrate on evaluating the sustained management skills required to maximize profits through buying and selling diverse range of items from customers. Therefore, the task in this game is defined as Weekly shop management, i.e., managing shop for a week automatically. This game could effectively demonstrate the negotiation ability of the LMM in a trade and bargain. For example, giving an unacceptable price to the customers, i.e., a pretty low price for seller customer or a very high price for a buyer customer, could cause the deal to fail directly, which brings no profit in this situation. The key is to carefully analyze the description of the item, e.g., the rarity and condition of the item, and more importantly, the response of the customer, i.e., the customer's mood changes.
Contrary to many games that feature detailed tutorials highlighting specific operations and objectives through each crucial step, Dealer's 's Life 2 does not provide such guidance. This absence transforms the game into a zero-shot, hard open- world task, where the LMM must directly apply its prior knowledge of haggling and strategic decision-making to new and unfamiliar environment. To provide readers with a clear and straightforward understanding the task, we illustrate the typical flow of a day's s shop management through several key steps, presented in Table
Table Key points the open-ended mission, Weekly shop management in Dealer's Life Figure 25 showcases snapshots of key points (specific sub-figures marked in parenthesis in the table).
Task: Weekly shop management
Description
Open shop (Fig. 25a)
Start new day shop management.
Dialog (Fig. 25b)
Choose an option in a dialog.
Item Description (Fig. 25c)
View the item information
Haggle (Fig. 25d)
Give price for the item.
Deal Result (Fig. 25c)
View the deal results.
Stats (Fig. 25t)
View shop stats.
H.3. Implementation Details
The implementation of Dealers' Life 2 also strictly follows the GCC framework, which includes Information Gathering, Self-Reflection, Task Inference, Skill Curation, Action Planning, and Action Execution. The details are described in
38

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Daily News
my nume Briin bring wondertaly
oolle
Aurty
OPEN
Toiler brush
(a) Open Shop
(b) Dialog
(c) Item Description
Weekly Stats
Karlyy
Raance 5243
Gateperyi Home Eatimatic 77?
Deaf You suconsifily cutio B5m
Sally Stats
Eariinss (temi $2.507
My to
Now: $60
Eamings 90
Eamiingh
Monblly Stets
Stavni $1250
Expenues Charri $20
Game Stuts
Expenses Homeit $0
(d) Haggle
(e) Deal Result
(f) Stats
Figure 25: Image examples of key points in the open-ended task of Dealers' Life 2.
Appendix E. Therefore, we emphasize the specific implementations for Dealers' Life 2.
Procedural Memory. Due to the absence of new-user guide, the LMM cannot directly and accurately know the operation method effect of an action in the game, e.g.. giving the price can only use the keyboard to input integer in an abstract box in the bottom right of the haggle screen as shown in Figure 25d, by directly observing the screen. Unless the player executes an action and observes what is happening, the player cannot know what its effect is. However, this could easily cause severe errors in an open-world environment. For example, if the player gives a price at $100,000 for an item without knowing what the box is, it could cause the player to lose all the money. Besides, this game is very simplified with finite types of screen content and fixed buttons positions for processing the deal, where we could categorize the screen types and design general atomic skills for them. Thus, with a focus on evaluating the LMM's zero-shot haggling and strategic decision-making ability in managing shop, we believe it is reasonable to skip the skill curation by directly setting several atomic skills the initialization of the procedural memory, such "process_ dialog(" for clicking on the option of a dialog screen to keep the deal going on as shown in Figure 25b. The description of all the atomic skills is listed as follows:
open_shople The function open the dealer's shop to start dealing for today.
give give_price(price): The function to give a price for the item the deal. The price must be an integer number.
process_dialog(} The function to click on to choose the first option of the dialog to make the game go on.
close_ Descriptiio_pageff The function to close a description page showing information about the item details, daily stats, or the traits of the buyer or seller.
accept_dealfy The function to click on the check mark to accept the deal on the confirmation dialog.
reject_deal(): The function to click on the cross mark to reject the deal on the confirmation dialog.
finish_buyl The function to click on the ok button to finish the deal on the confirmation dialog.
finish_selli The function to click on the ok button to finish the selling on the confirmation dialog.
Additionally, Dealers' Life 2 has heavy need for long-ten reflection, so we only the first and last frame of the video as input reduce the number of tokens used per request. Finally, this self-reflection module could help to keep the game going, instead of sticking to the same point in the game.
Action Planning. In this game, we restrict GPT-4 to output only one skill per action because it is a round-basee game that does not require frequent execution of actions, and the state of the next time-step after an action is executed is highly uncertain, e.g., the unpredictable mood changes in customer's response.
39

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
140000
Drac
Rambby common Conditione bad Categary: Lelsure Estimater $280
Payer handhock prokuee
p
Conditon
Dom handb
Estiatte $290
Now: $220
Draconians yye
$140
plaver
(a) Agent: Given that the customer a seller and the item on offer is in bad condition with an estimated price of $ 280, it would be wise offer around $150 to leave for negotiation.
(b) Agent: Given the item's bad condition and $280 estimate, A reasonable starting counteroffer would be 60-60% of its estimated value, and $160 reasonable counteroffer.
Don't offiend Sue say $250.
Mermn $190 und whot say 92057 Acceps
Condition:
Conditiorc Category:
Category:
$280
$280
Now: $210
Now: $205
RINCE
(c) Agent: A reasonable next offer could be $180 $190, aiming to reach middle ground. would still risk with $160 to see the offer cut further.
(d) Agent: The customer is proposing her final offer $205. It will be necessary accept the offer this still profitable margin.
Figure 26: The reasoning of CRADLE in successful deal with haggling. The price provided by CRADLE is keyed in by keyboard and mouse operations in the digital display box in the bottom-right comer.
H.4. Case Studies
Here we present a few game-specific case studies to further discuss CRADLE'S reasoning and decision-making process in the GCC setting.
1.4.1. SUCCESSFUL, NEGOTIATION
Figure 26 illustrates a successful negotiation by CRADLE with an NPC seller over an item valued at $280. CRADLE determines a strategic starting offer by considering both the item's quality and the customer's initial proposal. Throughout subsequent negotiation rounds, CRADLE leverages its memory maintain offer close the initially assessed $160, applying pressure the customer to reduce their expectations. However, CRADLE also demonstrates flexibility, adapting its strategy when faced with the customer's 's final offer- -ffer-signaled by their incline to leave. This allows CRADLE to secure final agreement that still yields a profitable deal.
H.4.2. UNSUCCESSFUL NEGOTIATION
Figure 27 illustrates a scenario where CRADLE engages in an unsuccessful negotiation. The seller consistently demands a price above the estimated value of the item, while CRADLE, aiming secure profit, steadfastly offers a price below the estimated value. A common price cannot be arrived at after rounds of negotiation. Consequently, the negotiation fails to reach agrcement, resulting in the departure of the high-expectation customer.
H.4.3. ACQUIRING AND SELLING OF COUNTERFEIT ITEM
Figure 28 depicts a complete buying and selling cycle involving an item later identified as counterfeit by buyer. Notably, CRADLE initially acquires the item without known price estimate, basing its purchase solely on the seller's initial offer. In subsequent transactions, although CRADLE recognizes the item's potential forgery, it continues to trade the item pursuit
40

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
IS 1O7
Basie
$255
Rariby uncommmnn Conditione temille Categary: Leisure Estimates $164
391
Conditon
Citagary
Now: $255
etball
$140
Basketball
(a) Agent: will offer an initial price of $100 to reflect the poor condition while allowing room for negotiation since the condition is poor.
(b) Agent: The buyer suggests higher than the estimated price.1 should offer price lower than the item's 's esti- mated value. will offer $110.
107
Are you ensty" Listen, thinking
ddo''t chink we'l ever agnopmmnnt That's
Rariby Conditions lerviie Category: Leisure Estimate: $164
Rariby Conditions temble Categary: Leisure Estimate: $164
Corgrne
Now: $250
Now: $135
$1R5
$185
(c) Agent:
The seller has countered several times al- ready, suggesting that further negotiations could result in mutually acceptable price, possibly just little bit below S164.
(d) Agent: The negotiation failed. My next step is to acknowledge the failure and continue the game-play to interact with other customers profit.
Figure 27: The reasoning of CRADLE in an unsuccessful deal with haggling. The price provided by CRADLE is keyed in by keyboard and mouse operations the digital display box in the bottom-righ niight corner.
of maximizing profit. This strategy yields substantial profit margin of 145%. However, it also poses risk to the in-game character's reputation, highlighting a strategic decision made under the directive to prioritize profit.
H.5. Quantitative Evaluation
The effectiveness of CRADLE in terms of game performance is evaluated through experiments conducted over 7-day gaming scenario using nine quantitative financial metrics. A comprehensive description of these metrics is provided in Section H.6.
H.6. Evaluation Metries
Assuming the buying price for item is denoted by B,, the selling price by S;, the market valuation by V., and the number of successfully traded items is n. To evaluate CRADLE'S profitability and performance in Dealer's Life 2, we use the following evaluation metrics:
Turnover Rate (TR) can be calculated as the ratio of the number of successfully traded items to the total number of items considered (both successfully and unsuccessfully traded). reflects the Agent's ability to successfully complete transactions and can indicate operational efficiency, market competitiveness, and negotiation effectiveness. The calculation formula is TR
mm
Gross Profit Margin (GPM) is the ratio of gross profit to sales revenue, reflecting the dealer's ' direct profit capability
after selling items. The calculation formula is GPM Σn S.BB
21
Return on Investment (ROI) is the ratio of profit to investment, used to measure the dealer's return on investment for items. The caleulation formula is ROT
> 8 B
Σ B,

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Are de soo
Yo sar de BL00
Cotegarys Meuurbble
Cotegarys Munncble
Collegay:: Weurche
$5,600
$5,200
$4,900
$5,000
$5,00
$5,000
(a) Agent: Given that do know (b) Agent: Since the seller is willing (c) Agent: The counteroffer of $4,900 the estimated price, proceed lower price, current counteroffer seems reasonable, can go further for 80% of the asked price, around $4,200 to may still be negotiated. $4,480.
the best possible price.
Meem Be 500unl
Seet This Bike worf stance Time
Barlyy incommon
BLR
Conatten
Conelion
Grtegary: Metanbble
BLAbO
Now: $4,650
New: $4,850
Now: $5,100
(d) Agent: The response will be final.(c) Agent: As am selling the item, I'll (f) Agent: note that the customer re- jected my counter-offer, calling the item fake.
Given this, accepting the deal is sen-try for a higher price for higher profits. sible the item.
Helo thare
Blikkig alout
Prepure
reluee competent
you revilation an Expert works
Feis Bludd
$11,400
Jewe
$1L400
necklace
(g) Agent: The seller is in urgent need (h) Agent: From the warning window, I (i) Agent: Though the jeweled necklace buy the jeweled necklace.
am told that this necklace fake.
fake, accepting this offer provides sub- stantial profit with the buyer's offer at $11,400.
Sold -46, 750 (+1450 sutressdully BIAADO revisation sseen
(j) Agent: Despite the successful sale at good price, the item is revealed a fake. The added profit is good (+ (+145%)
Figure 28: Case in acquiring and selling an item for multiple attempts with reasoning, and dealing with unexpected information on the authenticity. The price provided by CRADLE is keyed in by keyboard and mouse operations in the digital display box in the bottom-right corner.
42

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Valuation Deviation (VD) reflects the difference between the selling price and the market valuation, used evaluate the reasonableness of the pricing strategy. It is denoted as VD Σ S,-Vi
Buying Price to Valuation Ratio (BPVR) can help determine whether the buying price lower than the market valuation, reflecting the success of the procurement. The calculation formula is BPVR
- R; -i-1 Vi
Selling Price to Valuation Ratio (SPVR) reflects the selling price relative to the market valuation, helping to assess the success of the sales. The calculation formula is SPVR
21 8
Average Profit Rate (APR) reflects the overall profitability of the dealer on items. Assuming the return rate for item i is S.B. the calculation formula of average return rate is denoted as APR = 1.2 - S.BB
B,
Maximum Return Rate (MRR) is the highest return rate among all items. The calculation formula is MRR max(
S
-B
Su-Bi
B2
Sn-B..)
B1
B2
"B,, B,
Minimum Return Rate (mRR) is the lowest return rate among all items. The calculation formula is mRR min(2
81
B1
S1
B2
Ba
Table 10: Performance of CRADLE with GPT-4o in Dealer's Life 2 gameplay. "# attempts" represents the total number of all negotiation attempts items, including both successful and unsuccessful transactions.
Exp
attempts
TRt
GPM
ROIt
VDf
BPVRL
SPVRT
APR
MRR
mRR
01
13
92.86
20.38
25.60
13.17
90.10
113.17
42.97
105.56
0.00
02
12
91.67
18.89
23.30
23.30
100.00
123.30
17.98
97.76
0.00
03
12
83.33
26.81
36.63
34.39
98.36
134.39
38.68
127.27
-8.06
04
9
100.00
49.35
87.45
80.69
93.53
165.74
66.45
145.16
0.00
05
12
100.00
20.61
25.25
25.25
100.00
125.25
23.08
44.33
0.00
Avg.
11.6
93.57
27.21
39.65
35.36
96.40
132.37
37.83
104.02
-1.61
I. Cities: Skylines
L1. Introduction Cities: Skylines
Cities: Skylines is a single-player open-ended city-building simulation game developed by Colossal Order. In the game, players assume the role of city planner, tasked with building and managing various aspects of city ensure its growth and prosperity. Players engage with wide range of urban challenges, from managing traffic flow balancing the budget, and from providing essential services to fostering vibrant economy. Each decision impacts the city's 's development, requiring players to hone their planning and strategic meciiiinnmmaiiggggg skills to succeed. Effective city management leads to thriving neighborhoods, a growing economy, and high citizen satisfaction, while mismanagement can result in traffic congestion, service shortages, and decline in population and reputation. Proper planning and responsive governance are crucial for a city that flourishes and remains appealing to its residents and visitors.
As the city's infrastructure and various supporting resources are well-developed, it can attract more people. And larger population brings more tax revenue and also brings greater expenses to the city's operations. If operated properly, the increasing population can continuously unlock richer urban facilities; if operated improperly, such as road congestion, insufficient services, housing shortage, water and electricity shortage, noise pollution, water pollution, excessive garbage, disease, fire Situation, etc., will all lead to population decline.
This game could be used to evaluate agents' strategies managing urban development and resource allocation. By simulating different scenarios, agents can experiment with various policies and infrastructural changes to see their impacts on the city's growth and sustainability. Effective strategies may involve optimizing public transportation systems to reduce road congestion, investing in renewable energy sources to prevent power shortages, and implementing comprehensive waste management programs to handle excessive garbage. It offers a risk-free environment to test innovative ideas and learn from the consequences of their actions, ultimately promoting deeper understanding of sustainable urban development.
Though this game is ranked very positive on Steam, is notorious for its extremely high difficulty for beginners, as lacks a detailed tutorial in the beginning, which introduces more challenges for CRADLE to deal with. On the other side,
43

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Although the successor, Cities: Skylines 2, simplified the controls and provided a detailed tutorial for beginners, it became notorious for poor optimization and frequent crashes that caused computer blue screens. As result, had to back using Cities: Skylines instead of 2. And we do not apply any modes to the game, We use the latest version of the game (version 1.17.1-f4).
1.2. Objectives
Our mission is to build cities that they can support many people as possible. Maps in this game are usually very large, which usually costs human players dozens of hours to cover all areas. Besides, the technology tree unlocks as the population grows, which requires multiple turns of planning and building. In this work, we simplified the problem by starting the game near the water and fixing the viewpoint (as shown in Figure 29a), so that CRADLE can leverage the pixel position in the screenshot to locate the position of placed buildings and facilities. Agents start with a plot of land, which is equipped with an entry and an exit from a major highway, providing crucial access for future traffic flow, and proximity the water source, which is essential for the city's water supply needs. And we foeus on the first turn of planning. i.e., pause the game and stop the passage of the ga-game time, use the initial starting funds of C70,000 and the most basic road, water, and electricity facilities provided at the beginning of the game, which is enough to achieve the first milestone, Little Hamlet with the population of 440 in the game. Then what kind of city can CRADLE create? Can this city ensure water and electricity supply to keep functioning normally while reasonably dividing residential, commercial, and industrial zones? A run is terminated when it reaches the maximal steps, 1000, or the budget used up (less than 1000).
384
768
1152
1536
1920
(1.1)
(2.1)
(3.1)
(4.1)
((5.1)
(2.2)
(3,2)
(4.2)
(5.2)
(2.3)
(3.3)
(4.3)
(5.3)
(a) Demonstration for the initialization location of our mission in City: Skylines. which is near the river and contains the entry and exit of the highways.
(b) Visual prompting methods used in Cities: Skylines. The full screenshot is divided into 3 x 5 grids and each grid is assigned unique white coordinate.
I.3. Evaluation Metric
To measure the completeness of the city built by the agent, we design the following preliminary metrics:
Roads in closed loop: Whether the road is a closed loop, which is crucial for ensuring smooth traffic flow and is beneficial for the city's 's future development.
Sufficient water supply: To ensure a sufficient water supply, the player needs to construct water pumping station at the shoreline and then water pipes cover every district along the roads. To manage the effluent effectively, the other end of the water pipe network must be equipped with the water drain pipe which is also required to be placed near the shoreline.
Sufficient electricity supply: Both zones and water facilities need electricity to power. To provide sufficient electricity supply. the player can build a coal power plant or wind turbine. Considering coal power plants cost too much and will create heavy pollution, wind turbines combined with the power lines are better choice at the beginning. The electricity area extends automatically based the presence of buildings and infrastructure that consume electricity.
Zones Coverage > 90%: The built two-lane road will provide empty space for the development of zones, i.e., residential zone, commercial zone and industrial zone. Residential zones provide houses for people to live in, which is the most essential zone to increase the population. Commercial zones provide places for small businesses, shops,
44

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
and services produced in the industrial zones imported. Industrial zones provide jobs for the residents and products for commercial buildings, which is also important attract more people to move the city. This metric is used to evaluate whether 90% of the available areas are covered by the zones. The agent needs to reasonably allocate the areas and proportions of various zones to achieve better city development and attract a larger population.
Maximal population: After CRADLE finishes building, we will unpause the game and start the simulation. Then houses start to be built and residents start to move in. We will record the maximal population during the simulation as the value for this metric.
Maximal population with assistance: We find that cities built by CRADLE manage to meet most of the requirements but suffer a significant population loss due few easy-to-fix fix mistakes. So after CRADLE finishes the design of the city, we apply human assistance that attempts to address these small mistakes within unit operations (building or removing a road/facility/a place of zones is counted as one unit operation). We will also record the maximum population during the simulation in the city with human assistance.
1.4. Implementation Details
The implementation of Cities: Skylines also strictly follows the GCC framework, which includes Information Gathering, Celf-Reflection, Task Inference, Skill Curation, Action Planning and Action Execution. The details are described in Appendix E. Therefore, we emphasize the specific design for Cities: Skylines.
Pause. Since the game is stopped before starting the simulation, there is no need to unpause and pause the game while executing actions.
Visual Prompting. As shown in Figure 29b, similar to Stardew Valley, we divide each screenshot into 3 x grids with an axis based on the resolution of the game screen. Then CRADLE can utilize the pixel-level position in the screenshot to locate the building and facility. We empirically find that this visual prompting method can result in a more precise control of GPT-4o.
Information Gathering. In Cities: Skylines, the game's 's perspective is typically adjustable, allowing players to zoom in and out, rotate, and pan across their cityscape to get detailed view of their urban development. To ensure consistency and ease of navigation for GPT-4o, we have locked the camera angle and applied a visual prompting method to enhance GPT-4o's visual understanding. Besides, we use GPT-4o to extract key information, such as budget, population, construction information and error messages, in the game.
It worth noting that in this module, we feed the original screenshot to GPT-4o, rather than the augmented screenshot with axis and coordinates. We find that the numbers and lines may cover some key information and result in wrong OCR recognition. For example, the construction information, "Estimated Production: 120,000m"/weckk may be mistakenly interpreted as "Estimated Production: 000,000m //wek"" by GPT-4o, due to interference from the lines and numbers. This construction information is a key signal for the suitable place of the water pumping station. For the other modules, we feed GPT-4o with the augmented screenshots.
Self-Reflection. Since actions in this game are very short, and each of them has a significant effect shown in the last screenshot. We only use the first screenshot and the last screenshot of the video clip input to this module, which is proved to be enough for not missing any important information.
Task Inference. Due to the lack of a detailed tutorial, we have to provide draft blueprint for the GPT-4o as the plan at the beginning to help GPT-4o to determine the next step to do. This plan provides guidance to the orders of building each facility and how to build a closed road, how to ensure water and electricity supply and zone placement. Even so, we find that GPT-4o failed frequently to follow the plan, resulting in the lack of building some important facilities, like water pumping stations.
Skill Curation. Due to the lack of detailed tutorials in the game, we generate the skills through self- self-exploration in this game. The skill generation basically involves manipulating the toolbar to understand the items on it. The pseudo-code for skill generation is described in Algorithm 1. This process leverages SAM for objective grounding and GPT-4o to gather information about the objects provided by the game, subsequently generating skills based on a predefined template. An example of the process is shown in Fig 30, 31, 32a, 32b, 33a and 33b.
Action Planning, this game, we only let GPT-4o output one skill for each action since we observe that GPT-4o tends to
45

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Figure 30: The toolbar in Cities: Skylines
16
18
10
15
13
21
Figure 31: The grounding result of the toolbar Cities: Skylines
output try_place and confirm placement together if allow it output and execute multiple skills in one action, which is against the intention of our design for the try_place action.
Procedure Memory. Skills generated through self-exploration are listed below:
open_roads mendm(]):():: The function to open the roads options in the lower menu bar for further determination of which types of roads to build.
open open_electricity pncetricitymenalle The function to open the electricity options in the lower menu bar for further determination of which types of power facility to build.
open_water water _swwter_swwgeeeeeee menu(): The function open the water and sewage options in the lower menu bar for further determination of which types of water and sewage to build.
open_zone gaan_aonnggmennlee The function to open the zoning options in the lower menu bar for further determination of which types zonings build.
try_place two _an__ane__a sedee_oad(111, 31, 22, y2): Previews the placement of road between two specified points, (æ1, V1) and (I2, U2), with T1, 81 being the coordinate of start point of the road, and (I2,, y2) being the coordinate of end point of the road. This function does not actually construct the road, but rather displays visual representation where the road would be placed if confirmed.
try try_place_ actually construct the wind turbine, but rather displays visual representation of where the wind turbine would be placed if confirmed.
latplpalaaneeaaeeeeeeelelaeeeeee wind (trnnddtrrbieeer,rr. y): Previews the placement of wind turbine on point, (z, y)- This function does not
y): Previews the placement of water pumping station point, (I,, y). This function does not actually construct the water pumping station, but rather displays visual representation of where the water pumping station would be placed if confirmed.
try_place_ _aatrerwteerreeeeeeeeeeereeeeeer hatpllupmmppppppiliaelill
try_place patplaccewwttr___e (atterr_ipe((1,))) y1, #2, 32): Previews the placement of water pipe between two specified points, (x1 y1) and (Iz, 32). This function does not actually construct the water pipe, but rather displays a visual representation of where the water pipe would be placed if confirmed.
try try_place secee_attee_ddde lii_ddran__ppp pittir_ppppxx____ y): Previews the placement of a water drain pipe on point, (I, y). This function does not actually construct the water drain pipe, but rather displays visual representation where water drain pipe would be placed if confirmed.
try try_place_comee commercial _onelloooo___a_ 21, 22, U2): Previews the placement of commercial zone within rectangular region with diagonal corners at (I1,8)) and (2). This function does not actually construct the commercial zone, but rather displays a visual representation of where the commercial zone would placed if confirmed.
try_place industrial industrial_zone(x1,, V1, 22, y2): Previews the placement of industrial zone within rectangular region with diagonal corners at (I1, V1) and (Iz, U2). This function does not actually construct the industrial zone, but rather displays a visual representation of where the industrial zone would be placed if confirmed.
try cone-zon(ee (11, y/, F2, y2): The function remove the zone in the game. The zone must cover the road.
46

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
(a) When hovering the mouse over a toolbar item, the pop-up de- scription is "Water & Sewage". The skill generated is then called "open_wwee water_sewagee_eeecceeee menu_.
(b) When hovering the mouse over a toolbar item, the pop-up de- scription is "Education Reach population of 440", As this is not selectable for now, GPT-4o does not generate new skill for it.
(a) The Water & Sewage menu is opened by executing the new skill "open_water_ www____wwwaaeeeeeeeeeee sewor____w____"_..... The Agent then hovers the mouse over a second-level toolbar item, the pop-up description is "Water Pipe", and the generated skill is called "try "try_place_____ waaer_pipe""
(b) The Roads menu is opened by executing the new skill "open_roads. _oadd__menud. The Agent then hovers the mouse over second-level toolbar item, the pop-up description is "Two-Lane Road", and the generated skill called try__paac___v_ aoo_lane_rraae
confirm_placement placement(): The function confirm the placement and build the object after the try whlppllaacjjeeeeeeee function.
cancel placement) The function to cancel the placement of the object after the try tra_place [object] function.
[objeccc_[ooject)
Episodic Memory. Besides the common information to store in the episodic memory. We initialize the memory with the coordinates of the entry and exit of the highway. Then CRADLE is able to extend the roads according to these two points at the beginning. When road or a facility such as wind turbine, water pumping station, water drain pipe and water pipe placed on the map. the corresponding coordinates will also be stored in the memory for future development of the city.
I.5. Case Studies
1.5.1. FAILURE FOR ROAD BUILDING.
As shown in Figure 34, sometimes GPT-4o will build long road, which ends on the top of water. The recorded endpoint of the road is actually the projection of the road on the sea level, resulting in the offset from the projection point and the real endpoint of the road. It leads to the failure of extending the road to the other places.
Figure 34b, 34c, 34d and 34e tells story that GPT-4o sometimes forgets to confirm the placement (from 34c to 34d) and directly moves to the next step of building the next road (from 34d to 34e), resulting in the disconnection of the roads.
1.5.2. FAILURE FOR SUFFICIENT WATER SUPPLY.
Figure 35 displays three cases where CRADLE fails to ensure the water supply due to the disconnection of water pipes and the missing water pumping station. All of them can be fixed within three unit operations. As shown in Figure 35b and 35f,
47

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Algorithm 1 Skill Generation
Input: Toolbar with objects, Skill template
Output: Procedure memory with generated skills
Initialize procedure memory
for each object in the toolbar do
Hover the mouse on the object to get the description Generate skill using GPT-4o based on the object description and the skill template Store generated skill in procedure memory Execute the generated skill to enter the second-level toolbar
for each object in the second-level toolbar do
Hover the mouse on the object to get the description Generate skill using GPT-4o based on the object description and skill template Store generated skill in procedure memory
return procedure memory
aa)
(c)
Figure 34: Failure cases of building roads in a closed loop. Figure 34a shows that the road is built over the water and is difficult to continue. Figure 34b, 34c, 34d and 34e tells a story that GPT-4o sometimes forgets to confirm the placement (from 34c to 34d) and directly moves the next step of building (from 34d to 34c), resulting in the disconnection of the roads.
we observe significant increase in the population if these mistakes are fixed, which proves that CRADLE already has the ability to build a rcasonable city but some minor adjustments are needed.
48

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
(a) CRADLE'S craftwork I. The upper left corner of the city is experiencing severe local water shortage since the water pipes there not connected. Population: 800+.
(b) CRADLE's 's craftwork with assistant within three unit operations to develop the idle area in the upper right comer of the city into a residential zone and put two water pipes to ensure all the water pipes connected and cover the whole city. Population: 1150+.
(c) CRADLE'S craftwork II. The left side of the city a localized area on the right suffers from water shortage because of the water pipes connected issues. Population: 640+.
pight
(d) CRADLE'S craftwork II with assistant within three unit operations by selling the redundant water pumping station and the independent water pipe the right get some budget and using the budget get the water pipes connected. Population: 730+.
(e) CRADLE'S craftwork III. The entire city is experiencing severe water shortage due the lack of the water pumping station. Population: 200+.
(f) CRADLE'S craftwork III with assistant within three unit operations place the water pumping station, lay water pipe on the right side and develop the bottom area with industrial zones. Population: 780+.
Figure 35: Demonstrations of three cities built by CRADLE in zoning view (left), water view (middle) and electricity view (right).
Figures 35b, 35d, 35f show the cities with human assistance to address construction issues (shown in red arrow). Populations shown in the figures close to but not exactly the maximal population since they changed dynamically.
49

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
J. Software Applications
J.1. Selected Software Applications
Besides targeting complex digital games, CRADLE also includes an initial benchmark task set across diverse software applications. The selected applications include Chrome, Outlook, Feishu, CapCut, and Meitu. These applications cover popular applications for daily tasks in different usage categories, such web browsing, communication, work, and media manipulation. Table 11 shows the exact application versions benchmarked in this paper. Five distinct tasks were designed for each application to represent their target domains and explore the difficulties posed to LMM-based agents and analyze their limitations. Figure 3 shows an overview of all tasks across applications and Tables 12 and 13 detail each task.
Chrome and Outlook were selected as common representatives for web browsing and e-mail, with well-known functionality and UI design. CapCut and Meitu are two popular media editing applications for video/image editing with their own interaction styles. Lastly, Feishu (also known Lark) an office collaboration and productivity application, which includes messaging, calendar//mettngss, and approval workflows. It represents a complex business application that doesn't 't strictly follow OS-specific UI guidelines. To the best of our knowledge, this is the first agent targeting applications like CapCut, Meitu, and Feishu.
J.1.1. BRIEF DESCRIPTIONS
Chrome is a web browser developed by Google. It allows users to access and utilize online resources through activities such as browsing websites, streaming videos, and using web applications. Additionally, users can customize their browsing experience with various extensions, manage bookmarks and passwords, and synchronize their data across multiple devices for seamless access.
Table 11: Exact software versions utilized in the de- scribed experiments. Similar versions should behave similarly.
Software
Version
Chrome
125.0.6422.142
Outlook is an application by that allows users to manage emails, calendars, contacts, and tasks. It includes tools for communication and scheduling through features such as sending and receiving emails, setting up meetings, and keeping track of appointments. Additionally, users can customize their experience and integrate Outlook with other Microsoft Office applications.
Outlook
1.2024.529.200
CapCut
4.0.0
Meitu
7.5.6.
Feishu
7.19.5
CapCut is popular video editing application developed by ByteDance. It provides casy-to-use editing tools and and enables users to create quality videos with a range of advanced features. CapCut offers a set of editing tools, including trimming, cutting, merging, and splitting video clips; the application of various effects, filters, and transitions; as well as adjusting speed, and adding music or text overlays.
Meitu a photo editing application. It is designed to cater broad audience and enables users enhance and transform their photos with minimal effort. Meitu offers editing tools, including basic adjustments like cropping, rotating, and resizing, as well as advanced features such as beauty retouching, filters, and special effects. Additionally, Meitu offers a wide range of stickers, frames, and text options to further personalize photos.
Feishu, also known as Lark, is a business communication and collaboration platform by ByteDance. It integrates various tools for office workflows and project management. Feishu offers a wide array of functionalities, including instant messaging, video conferencing, file sharing, and collaboration within the app. It also includes an integrated calendar, which helps users schedule and manage meetings and events, and task management tools that allow users assign and track tasks.
J.2. Software Tasks
For each of the five applications, we selected a set of representative tasks for their respective domains. For example, search, navigation, and settings tasks on Chrome; sending, searching, and deleting emails, plus changing settings on Outlook; basic video and image editing operations on CapCut and Meitu (e.g., adding special effects and creating collage); and communication and organization operations on Feishu. Tables 12 and 13 describe in detail the 25 tasks CRADLIE performs and analyzes on the five selected applications; also illustrated in Figures 36, 37, 38, 39, 40, and 3.
is worth noting that we add a special task on CapCut to demonstrate the agent's 's ability for tool use. In this task, a
50

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Table 12: Task Descriptions for Chrome, Outlook, and CapCut. Difficulty refers to how hard it is for our agent to accomplish the corresponding tasks. Figures 36, 37, and 38 illustrate each task (specific sub-figures marked in parenthesis in the left-most column along with task name).
Software
Description
Difficulty
Chrome
Download Paper (Fig. 36a)
Search for article with title like (paper_tille and download its PDF file.
Hard
Post in Twitter (Fig. 36b)
Post "It's good day." on my Twitter.
Hard
Open Closed Page (Fig. 36c)
Open the last closed page.
Easy
Go to Profile (Fig. 36d)
Find and navigate to (person_nanee satiennamessssseee homepage on GitHub.
Medium
Change Mode (Fig. 36c)
Customize Chrome to dark mode.
Medium
Outlook
Send New E-mail (Fig. 37a)
Create a new e-mail to (email_aulress) with subject "Hello friend" and send it.
Medium
Empty Junk Folder (Fig. 37b)
Open the junk folder and delete all messages in it, any.
Medium
Reply to Person (Fig. 37c)
Open e-mail from pherson_name in the inbox, reply to it with "Got it. Thanks.", ", and click send.
Medium
Find Target E-mail (Fig. 37d)
Find the e-mail whose subject is "Urgent meeting" and open it. Easy
Setup Forwarding (Fig, 37e)
Set up email forwarding for every email received to go to Medium
(email address).
CapCut
Create Media Project (Fig. 38a)
Create a new project, then import [video_ sittee__fle_mmmee to the media, click Hard the "Audio" button add music the timeline, and finally export the video.
Add Transition (Fig. 38b)
Open the first existing project. Switch to Transitions panel. Drag transition effect between the two videos, and then export the video. Delete the video frames after five seconds and then before one second in Medium this video, and then export the video.
Medium
Crop by Timestamp (Fig. 38c)
Add Sticker (Fig. 38d)
Open the first existing project. Switch to Stickers panel. Drag a sticker Hard of person's 's face to the video, and then export the video.
Crop by Content (Fig. 38e)
Crop the video when the ball enters the goal, and then export the video.
Very hard
Table 13: Task Descriptions for: Meitu, and Feishu. Difficulty refers to how hard it is for agent accomplish the correspouding tasks. Figures 39, and 40 illustrate each task (specific sub-figures marked in parenthesis in the left-most column along with task name).
Software
Description
Difficulty
Meitu
Apply Filter (Fig. 39a)
Apply filter from Meitu (picture_file _ille_mm_mm and save the project.
Easy
Cutout (Fig. 39b)
Cutout person from ipicture_file cele_name] and save the project. Add flower sticker to /picture_filennnne [atee_emmeeaa and save the picture.
Easy
Add Sticker (Fig. 39c)
Middle
Create Collage (Fig. 39d)
Make a collage using 3 pictures and save the project. Add a circle-shaped frame to /picture_ wall-fianaaaaa and save the picture.
Hard
Add Frame (Fig. 39e)
Hard
Feishu
Create Appointment (Fig. 40a)
Create a new appointment in my calendar anytime later today with title "Focus time".
Hard
Message Contact (Fig. 40b)
Please send "Hi" chat message to (contact_namek,
Easy
Send File (Fig. 40c)
Send the AWS bill file at (pdf_path) in chat with (contact_namel
Hard
Set User Status (Fig. 40d)
Open the user profile menu and set my status to "In meeting". Create a new meeting and meet now.
Medium
Start Video Conference (Fig, 40e)
Easy
51

---

It is worth noting that in Chrome's 's task 3 ("Open the last closed page"), CRADLE knows how to use the shortcut key directly, skill directly with the correct keyboard shortcut: 'Ctrl + Shift + T", whereas humans typically do not CRADLE: Empowering Foundation Agents Towards General Computer Control
(a) Download Paper
(b) Post in Twitter
(c) Open Closed Pago
(d) Go
Change Mode
Figure Screenshots of Chrome tasks.
pre-defined skill uses GPT-40 as a tool for video understanding capabilities. The skill can be selected to answer content-basee questions about a video (e.g,, "when the ball enters the goal") and the response be used during task completion. This task illustrated in detail in Figure 47.
J.3. Quantitative Evaluation
We calculate CRADLE'S performance over the 25 tasks the applications set. Each task is executed five times and performance measured in three metries: success rate, average number of steps taken by the agent (and variance over the five runs), and efficiency. Efficiency is defined the ratio between the expected number of steps in given task and the total number of steps taken by the agent. The expected number of steps per task is calculated by having humans perform each task.
Table 14 and Figure 41 show the details of the evaluation. CRADLE presents overall good performance over the diverse tasks and applications (compared to Expected Steps, CRADLE achieves an overall efficiency 50%). However, performance for certain tasks can vary considerably due different factors. The main reason for the higher number of task step during agent execution is the frequent incorrect positioning decisions for the mouse, i.e., the backbone model chooses position of bounding box tag that does not correspond the UI item described in the model reasoning. We discuss examples of task-specific issues in Sections J.5 and J.6 below.
calling the know this.
To further evaluate the performance of CRADLE in diverse software applications scenarios, we provide quantitative results over OSWorld, new contemporaneous benchmark with similar characteristics to settings. More details in Appendix K and overview of the results in Table 15.
J.4. Implementation Details
The implementation of CRADLE targeting all five software applications follows the GCC setting and framework modules (which include Information Gathering, Self-Reflection, Task Inference, Skill Curation, Action Planning, and Action Execution). Implementation details of the overall framework are described in Appendix E. Therefore, here we emphasize any application-specifcc differences or customization.
To apply CRADLE to the target application set described this appendix, we start with base common prompts, and customize those prompts for specific modules, if necessary, to handle inleecatioo-spccffi characteristics. For example, for
52

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Towards Opparal Copogle Cortret
(a) Send New E-mail
(b) Empty Junk Folder
(c) Reply to Person
(d) Find Target E-mail
(e) Setup Forwarding
Figure 37: Screenshots of Outlook tasks.
(a) Create Media Project
(b) Add Transiiion
(c) Crop by Timestapp
(d) Add Sticker
Crop by Content
Figure 38: Screenshots of CapCut tasks.
53

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
A
(a) Apply Filter
(b) Cutout
(c) Add Sticker
(d) Create Collage
(c) Add Frame
Figure 39: Scrcenshots of Mcitu tasks.
(a) Create Appointment
(b) Message Contact
(c) Send File
(d) Set User Status
(e) Start Video Conference
Figure 40: Screenshots of Feishu tasks.
54

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Chrome
Outlook
CapCut
Meitu
Feishu
100
Suceess Rate (%)
80-
60
40-
20
Download Paper Post in Twittrr
Open Cleaed Page
Go Profile
Change Mode
Send New E-mail
Empty Juni Foldor Roply to Person Find Target E-mail
Setup Forwarding
Creste Modia Project
Add Transiiion
Crop by Timestamp
Add Stickar
Crop by Content
Apply Filee
Cutoaa
Add Sticker
Create Collage Add Frame
Create Appointmnnt
Mosuage Contact
Sendl File Set User Status
Start V Video Conference
Figure 41: Success rates for tasks in software applications
CapCut we add few-shot examples for Self-Reflection, to let it properly perform success detection, as the application U1 by itself is non-standard and sometimes provides little post-action feedback to users, making it harder for the backend model to determine action success.
Information Gathering. Noticeably, GPT-4o presents the same limitations in both spatial reasoning (e.g., confusing up/down, leftrrigt) and image understanding identifying specific UI items or the state of the forefront GUI, across all applications.
To help mitigate such issues, we perform augmentation on the captured screenshots similarly to the Set-of-Mark (SoM) approach (Yang et al., 2023a), by only utilizing SAM (Kirillov et al., 2023) to generate potential UI items bounding boxes and assign them numerical tags. Our SoM-like augmentation differs from recent agent-related work (e.g., (Zhang et al., 2024: Xie et al., 2024)), which use OS-specific APIs to draw ground-trum UI structure info, like types and element tree) to the results, while CRADLE relies only on image input and the segmentation output as augmentation. To make this distinction explicit, we call our augmentation approach SAM2SOM 10 Figure 45 illustrates the difference. While our approach produces many more potential bounding boxes, it is more general by relying only on a screenshot (or video frame).
bounding boxes for interactable elements (plus
To ensure all bounding box labels are consistently positioned, CRADLE'S SAM2SOM implements two rendering styles, as shown in Figure 43 first and second rows. In the standard style, we pad the SAM2SOM-enhaneed image when showing the label IDs in the upper left corner of the bounding boxes (to prevent labels from hiding the contents of small areas), so no numerical label ID is drawn outside the image area). In the uniform style, all bounding boxes utilize single-color borders with labels in black text over white background, placed within the bounding box area (top left corner).
Moreover, in specific situations we may still need to re- fine SAM2SOM's 'ss output further. For example, in the Feishu case, observe that watermarks generated by the software affect the segmentation negatively, complicating GPT-4o's selection of the correct bounding boxes to in- teract with. Therefore, we implement a simple filtering method for such watermarks. This filter is enabled only in the Feishu benchmark and, as shown in Figure 44, can greatly reduce the number of unnecessary bounding boxes (from 216 to 166, in this example).
Discard message
mesige
In addition to using the SAM2SOM method for image augmentation, we also redraw the mouse pointer not present in captured screenshots in a more prominent ma- genta color based on its screen position, to emphasize both its presence and position for image understanding (e.g., Figure 42). The augmentation process in Information Gathering can then result in four versions of a screenshot: a) base image, b) SAM2SOM image, c) base image with mouse pointer, and d) SAM2SOM image with mouse pointer.
Figure 42: Sample augmented image w/ drawn mouse pointer Zoom overlay shows the image difference.
We do not claim the method itself as core contribution. SAM2SOM is used to illustrate a possible extra capability of the backend model, as mitigation for current spatial reasoning issues.
55

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
(a) CapCut: Base image
(b) CapCut: Base image w/ SAM2SOM
(c) Meitu: Base image
(d) Meitu: Base image w/ SAM2SOM
Figure 43: Image examples of the SAM2SOM augmentation styles. As CapCut's UI (top row) has very dark background, utilize single-color borders with IDs in black text over white background, placed within the bounding box area. Other application software and WSWdrl use the "standard" SAM2SOM multi-color style, as shown for Meitu (bottom row).
SAM2SOM image w/ watermarks
(b) SAM2SOM image w/o watermarks
Figure 44: Examples of filtering watermark in Feishu. The number of labels is greatly reduced from 216 to 166.
56

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
-3
Google
Google
Google
aa) Chrome window
(b) CRADLE SAM2SOM output
(c) OSWorld SOM image output
(d) GIMP window
(e) CRADLE SAM2SOM output
(I) OSWorld SOM image output
Figure 45: Comparison of CRADLE'S visual-only SAM2SOM and OSWorld's API-based SOM image results. Chrome: 78 53
bounding boxes; GIMP: 227 vs. 98 bounding boxes.
Self- Ceff-Refecctin. As the applications in the software set are much less dynamic than complex games, there is no need to send multiple video frames to Self-Reflection. For the software applications, pre- and post-action screenshot usually suffice, i.e, one image before and one image after an action executed. Digital games often have continuous and dynamic environments that require multiple frames properly capture the full context and thus help the backbone LMMs understand what happened. In contrast, software operations are typically more discrete and static, where the state before and after an action provides sufficient information for most analysis.
Nonetheless, we find that irrespective of images used, GPT-4o sometimes can have difficulty determining the success of certain tasks. For example, when downloading a file on Chrome, after either pressing 'Ctrl + S', or using 'Save' menu, the agent must also press 'Enter' or click the 'Save' button to complete the task. However, GPT-4o often assumes the task is complete when the dialog opens and before this final step. Similar cases incorrect conclusion happen when an action correctly closes a new panel or dialog. To address this category of issues, we add mandatory reasoning rules in the prompt for the Reff-Refeecton module help mitigate such mistakes. If specific applications this still remains an issue, we can fho- image examples reinforce how the backend model should correctly judge success.
57

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Table 14: Application Software results. Success Rate determines the ratio of successful completions over five runs. Average Steps refers to the number of actions the agent takes to fulfil lask, if successful. Expecied Steps represents the number of steps estimated by humans performing the task. Efficiency represents the ratio between the expected number of steps and the total number of steps taken by the agent.
Software
Success Rate
Average Steps
Expected Steps
Efficiency
Chrome
88%
8.23 + 6.75
4.20
48.05%
Download Paper
80%
16.00 5.52
6
37.50%
Post in Twitter
80%
11.75 = 5.26
7
61.14%
Open Closed Page
100%
1.00 +
3
300.00%
Go to Profile
100%
4.00 -.0
1
25.00%
Change Mode
80%
11.25 = 4.71
4
35.56%
Outlook
60%
7.13 + 5.61
4
48.48%
Send New E-mail
40%
11.00 4
5
45.45%
Empty Junk Folder
40%
8.50 3.50
3
35.29%
Reply to Person
60%
8.33 + 4.71
4
48.02%
Find Target E-mail
100%
1.40 + 0.80
1
71.43%
Setup forwarding
60%
12.00 4.90
7
58.33%
CapCut
56%
10.87 5.56
4.80
44.16%
Create Media Project
60%
13.67 = 5.25
7
51.20%
Add transition
60%
10.67 = 4.03
4
37.49%
Crop by Timestamp
60%
11.00 5.66
5
45.45%
Add Sticker
40%
12.00 8.00
4
33.33%
Crop by Content
60%
7.00 + 1.41
4
57.14%
Meitu
44%
12.36 = 3.34
8.00
64%
Apply Filter
60%
14.67 2.36
7
47.72%
Cutout
60%
9.33 + 1.89
5
53.59%
Add Sticker
40%
9.50 + 0.50
8
84.21 %
Create Collage
40%
16.00 = 2.00
12
75.00%
Add Frame
20%
13.00 = 0.00
7
53.85%
Feishu
56%
7.50 + 4.50
4.00
46.07%
Create Appointment
40%
8.00 + 1.00
4
50.00%
Message Contact
40%
6.00 + 1.00
3
50.00%
Send file
20%
11.00 0.00
7
63.64%
Set User Status
100%
14.60 7.50
3
20.55%
Start Video Conference
80%
4.50 + 2.60
3
46.15%
58

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Skill Curation. In software tasks, direct skill genera- tion was not necessary, UI operations generally map closely to specific mouse keyboard actions, making them more straightforward. In contrast, digital game en- vironments involve continuous interactions and decision- making, raising new previously undiscovered information, and requiring the development of new skills to handle novel scenarios and adapt changing contexts.
Shortist
Spitt
Select mode
Spitt mode
Select lefward
paint
Solect righwardd
Select
However, we do add some additional pre-defined skills, per-application basis, for specific knowledge like less- widely known keyboard shortcuts which could be learnt from the application. For example, CapCut's shortcuts screen, shown Figure 46, or toolbar/icon processing output similarly to the process described for Cities: Sky- lines. Moreover, we also introduce pre-defined complex skills to demonstrate CRADLE's capability to leverage tools into novel functionality, such as using GPT-4o tool to extract information from video to complete task 5 in CapCut.
Man track magnott
Unsolett area
10
Auto snapping
Group
Litoge swicch
Uniroup
Previw switch
Show pamd
Reant to defsuit
Cancal
Figure 46: Shortcuts screen in CapCut.
When dealing with shortcuts, e.g., as alternatives to mouse operations, may be the case that specific shorteuts require "calibration". For example, using the keyboard to navigate the timeline in CapCut (as seen in the bottom area of Figure 43b) requires mapping the keyboard shortcut ('Alt + arrow keys') to pixels time, which perform priori and use the mapping in the pre-defined skill go_to. to go_tt_ttteetttteeeeeee timestamp(seconnds)
Task Inference. During the execution of an application task, we let GPT-4o decompose the execution strategy for the next step based on the overall task description and the subtask description. If the previous task decomposition is found to be unrcasonable, a new decomposition plan should be proposed and this is evaluated at each iteration round.
Action Planning. To enable usage of SAM2SOM, for Action Planning, we insert new mouse skills, which mirror existing coordinates-basse mouse skills (i.e., that use x.y coordinates), but take bounding box numerical label as an argument.
Furthermore, unlike in game playing, which focuses performing
action per turn, when manipulating software CRADLE can be configured to perform two actions in sequence and thus lower interaction frequency requirements to the backend model. We find that GPT4-o can usually correctly output two-step compound actions. For example, when performing a search in the browser, it can typically output two consecutive action steps, egg. type_tteeeee type_text(text="(A "uusee(uuuurrrrre "), followed by the required press_key press_key(ke=='enter"
Action Execution. While atomic and composite skills can involve complex operations, Action Execution happens over the regular CRADLE action space, shown in Table 6. For example, during Action Execution, post-processie step converts the bounding box calls into regular mouse actions, using the centroid of given bounding box its coordinates for regular mouse operations.
Tool usage, like calling GPT-4o separately to analyze the contents of a media file, is not considered as an action, as tools do not operate on the environment, only as code steps inside a composite skill.
J.S. Case Studies
J. TASK HARDNESS
is well known that the difficulty of task completion can vary widely between humans and agents. The results in Table 14 help illustrate some such cases. While many application operation issues may be attributed to UI variety or non-conformity, that is not necessarily the main source of task hardness (i.e., how unexpectedly complex performing an operation is).
Here we use Outlook, a well-known e-mail client, as a study discuss how different factors affect CRADLE task completion in real-world application situations (the exact version used is listed in Table 11). Taking task 1 ("Create new e-mail to (email_address) address) ) with the subject 'Hello friend' and send it." ") as example, success rate of 40% and efficiency of 45,45% may seem lower than expected.
59

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Such task could be reasonably broken down into steps like: a) Create new e-mail, b) Add recipient, c) Write title, and d) Send e-mail. And the Task Inference module performs such decomposition consistently. However, Action Planning needs to define specific actionable operations with mouse and keyboard to execute each step.
Firstly, CRADLE needs to decide based on the knowledge and visual understanding capabilities available to it to either use a known keyboard shortcut (e.g., 'Ctrl+ + N') or to click at the "New mail" button. In our experiments, CRADLEE tends to chose clicking on the button, which is then affected by the previously discussed issues that led to the integration of SAM2SOM into the framework. Issues in spatial reasoning issues or icon/image understanding may cause a few incorrect click attempts.
Adding the recipient the e-mail requires typing address the appropriate location, i.e,, the typical "To" field. This can be accomplished in multiple ways, mainly by typing the address on the UI next to the "To" item choosing pre-existing contact.
Clicking on the "To" button triggers the UI to search and select a pre-existing contact e-mail address (with no option of adding a new contact entry, which requires first accessing the "Contacts" " menu, outside of "Mail"). Moreover, the UI interaction sequence to select an existing contact can be unintuitive even to experienced users, requiring a minimum of four steps, at each step offering multiple UI options that go away from contact selection. Attempting this flow usually leads CRADLE to exceed the maximum number of allowed step it gets confused by the UI design.
Nonetheless, choosing the simpler alternative of typing the e-mail address (assuming the correct text field is selected) triggers assistive UI pop-ups (as shown in Figure 48), which lead GPT-4o to falsely conclude the e-mail address is either already typed at the correct location or that it is duplicated and needs to be edited/removed. Furthermore, the pop-ups partially hide the subject area, making it harder for CRADLE to choose the next UI item to interact with for the next task step.
Similar issues with positioning and correctly identifying the typed subject text can also occur, but at much smaller frequency.
Lastly, completing the task and sending the e-mail requires step similar to creating a new message. But determining send success requires additional attention/reflection as not all cases of the "Send mail" interface disappearing indicate a successful send (e.g,, clicking on an unrelated e-mail on the Inbox or closing the current window pop-up).
The Self-Reflection module plays a key role in moving task completion forward by detecting failed attempts at executing each sub-task and providing rationale for failures, even if Information Gathering and Action Planning make repeated mistakes. Such feedback from Self-Reflection and allows Action Planning to tune its process and move ahead.
J.5.2. TOoL USE IN CAPCUT
Some general computer control tasks may require additional capabilities during execution preparation that can benefit from external tools to enhance agent abilities.
When performing video editing, like in CapCut, a user may need to determine the precise frames to operate on based on video content. For such scenarios, CRADLE can easily leverage tool-using skills, like the LMM's ability to understand actions in sequence of video frames, enabling it to comprehend video content and identify the exact frames for editing.
We exemplify such tasks with task 5 ("Crop the video when the ball enters the goal, and then export the video") for CapCut, as illustrated in Figure 47. This means our agent can effectively execute tool usage to find the specific frame where "the ball enters the goal". After the first round of Task Inference, CRADLE decomposes the task into three subtasks: 1. Identify the exact frame, 2. Crop the video, and 3. Export the video. Action Planning can then plan to execute "get_ine inttinfforatiin from_rriiiene video(event)) from our curated skills and generate "ball enters the goal" as its required argument for execution.
In this skill, we input frame set of the video 1 fps to identify the specific frame where the event occurs. The response is then recorded in Episodic Memory to ensure that subsequent operations can accurately utilize it and target the moment when the action occurs. Across subsequent iterations, CRADLE can then correctly plan and execute the remaining necessary actions for task completion: poo_oo_iimestmmpssccodeeee
delete_righht/ and 'export_project()]
We have integrated few-shot learning into Self-Reflection to ensure CRADLE recognizes that following export project)) the expected screen the CapCut application main window. This information allows to verify the successful execution of the task, leading to success detection for the overall task.
60

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Task Description: Crop the video when the ball enters the goal, and then export the video.
Information Gathering
Self-Reflection
Task Inference
Skill Curation
Action Planning
Image Deseription: CapCut shows various elements, including timeline, media library. and preview windows. It contains video clip that appears to capture a soccer game.
Imuge Same Flag: True
Image Same Flag: False
Mouse Position Same Flag: True
Mouse Position
Flag: False
Image Deseription: The timeline at the
ast Action: get inforeation from video
bottorn shows video clip that is
(event- "ball enters the gool")
currently at the 8-second mark.
"The last executed action was successful
Bounding Boxes Deseription:
because it correctly identified the frame where
Last Action: to_to__ (sccodd))))))))sssa3)) delete right()
[..., 14: Media...., 21: Templates,... 145:...
the ball enters the goal as the 8th second.
1. Identify the exacl moment when the ball enters the goal the video.
Success Detection Flag: False. steps to crop the vidco this point and export have not been completed yet.
The last executed action was successful
because the portion of the video after the Sth
second been deleted as required.
2. Crop the video at that point.
Sucvess Detection Flag: False, The final
3. Export the crepped video.
The current task is to crop the video the 8ch second where the ball enters the goal.
of exporting the video has not been completed yet.
Retrieved Skills:
The current subtask to expori
[ ['get_ing iit__infommtion_ffee feroon_ffoo_oodoeoeeee_ee video"
Retrieved Skills:
the videu to complete the task.
"go_to_timees tttotttmmstappp.... press_key_
['Bo_to_ tirostanp' 'deleto cdilett_right',, 'Be It inforrmtionn fron video " 'souse drag
Retrieved Skills:
tnouse_drag_withe wwtp:_hhhhh_aal____ee__a
('export_projese "press_entter 'cl
get "r__information fron_videeeeer
f------
ese window' : gg to toneetttttrpp...
Execute
(event="ball enters the goal
Exeeute
go_to_ timestamp (eresstanpssconssee. (seconds=
(iiletterrght()
Execute
eexport_project
Frame Set at 16ps
Prompt:
Please answer at which frane does "<Sevent3>" happen, only answer number
The sccond of "ball uniers the gual" 8.
Figure 47: Showcase of Task 5 ("Crop the video when the ball enters the goal, and then export the new video") in CapCut.
J.6. Limitations of GPT-40
Besides the previously discussed limitations of GPT-4o, is important to highlight couple other GUI grounding issues.
Non-standard UI and Noise.
Non-standard UI, be it in visual style or in behaviour, can lead GPT-4o to misinterpret UI item functionality and application context state. The same applies to visual noise in the form of update pop-up, external contents (e.g., ads), new e-mail/chat messages, etc.
CapCut is affected by both factors, further illustrated in Figure 49. Moreover, its UI includes non- standaad layouts involving precise positioning and drag/dropping. Lack of such prior knowledge by GPT4-o and differences in behaviour between similar functions, may also lead to mistakes in trying to decompose actions to perform. E.g.. "Add an effect" requires very different Ul-interaction depending on details. Users can add effects in three different ways: i) dragging an effect to the timeline; ii) click the plus sign in a given effect in the effects panel, which adds the effect to the current place on the timeline; and iii) drag an effect directly onto a video and apply the effect to the entire video.
61

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
god
To
grcccadee@outoookccoo
Suppetted contacts
Add subji
GCC Cradle
Add subje Use this address geccradte@ouutooo.coo
gpc.crale@@gmal..om
pocoodeeBootoooccom
found
People
GCC Cradee get gecondde@ggnalcomm
(a) Pre-existing contact dropdown
(b) Contact search dropdown
(c) Contact suggestions
Figure 48: Visual behaviour in Outlook that may lead GPT-4o to visual understanding mistakes.
Version update
Ligiin car share
X
CapCut Current weedon 3.D
veraien::.0.0
impooedd
Wi thank you supporting CurCui und look ferward so creating beauthul momente togathar.
Install now
Figure 49: Different CapCur pop-ups
Visual Context Detail.
GPT-4o still struggles with detailed visual understanding and over-relies on textual information or hallucinations, which results in insufficient attention visual context and leads understanding and reasoning mistakes.
One such common example is GPT-4o declaring dialog state to be ready to press button like "Save". while ignoring no file name was provided, even if GPT-4o has been prompted to check for such situations. The same applies to it suggesting keyboard shortcuts to open menus that do not exist in the image being interpreted, egg., trying to press 'Alt + F' to open the "File" menu on screenshot that has no "File" menu.
Lastly, this lack of attention context details can also affect understanding the outcome of operations over visual content, leading to incorrect estimation of operation success, e.g., when retouching an image deciding between circle and heart for shape form.
K. OSWorld
K.1. Introduction to OSWorld
OSWorld is scalable, computer environment designed for multimodal agents. This platform provides a unified environment for assessing open-ended computer tasks involving various applications.
K.2. OSWorld Tasks
OSWorld is a benchmark suite of 369 real-world computer tasks (mostly on an Ubuntu Linux environment, but including a smaller set on Microsoft Windows) collected from authors and diverse sources such as forums, tutorials, guidelines. Each task annotated with a natural language instruction and manually crafted evaluation script for scoring.
K.3. Implementation Details
The OSWorld environment uses a virtual machine that takes in Python scripts based on PyAutoGUI for actions and provides screenshots and accessibility tree for observations. We strictly follow the GCC settings. Our agent only uses the screenshot
62

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Slide settings 4. 4.1%
Slide editing 8. 7%
Tab, formatting 3.8%
Doc: settings 66%
Doc. editing 4 %%
Processing 7.0%
Image ops 7.0%
Visualization 1.9%
31.7% Office
Terminal 1.9%
Configuration 3.8%
Settings 2.4%
13.3%
Os 6.5%
Files 2.2%
Professional
Code assist 2.4%
Daily
Video control 4.6%
Workflow
File ops 8. 8.1%
21.1%
Email ops 2. 2.4%
Account ops 1.6%
27 .7.4%
Shopping 2.7 %
Multimedia 4 4.6%
Info query 1%
Data analysis 8.9%
Settings 5.7%
Misc. 5.7%
Figure 50: Task instructions distribution in OSWorld (Xie et al., 2024)
as input and outputs Python scripts using PyAutoGUI methods to control the keyboard and mouse (these operations are analogous to the regular action space for CRADLE). All 369 tasks use a same set of prompt templates.
We employ GPT-4o the framework's backbone model. We use the default experimental settings, as in OSWorld's bascline agent. The executable action space is the same the OSWorld setting, the atomic skills follows:
Mouse Actions
move_mmose to_ _o_positionnx, y): Moves the mouse to a specified position on the screen.
click position(xx y): Performs click at specified position.
mouse_dome mnous__down(button)) Presses the specified mouse button.
mouse (oonse__p(bbutoo)) Releases the specified mouse button.
riight_clickke click(x, y): Right-clicks at the specified position. right
clouble_click_nee at posiiiiio, y): Double-clicks at the specified position.
mouse mouse__rag(xx y): Drags the cursor to the position.
directiii,ection, amount): Scrolls the mouse wheel up or down by specified amount.
Keyboard Actions
rype_teeee ::pe_eex(texx): Types the specified text.
press Reyske)e Presses and releases the specified key.
key down(key):: Holds specified key.
- -ky__pp(eey): Releases a specified key.
press hotkey(keys): Presses combination of keys and releases them in the opposite order (c.g., Ctrl+C), useful for shortcuts.
Task Status
63

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
lask_is_me _ik_is_not iisinss___eeeeeeeeeeeeee not_Jeassble() Indicates that the task cannot be completed, providing feedback for scenarios where the agent encounters infeasible tasks.
Many of these basic skills require GPT-4o directly output an (x.y) position based on screenshot. Given that the current grounding tool to augment the screenshot. This way, GPT-4o
GPT-4o not able achieve such precise control, only needs to choose object ID. With the object ID and the bounding box of the object, we automatically convert it to the (x.y) position needed for skill execution, Instead of having GPT-4o directly choose the executable skills that require (x.y) position input, we provide several skills that only require a label ID as input for GPT-4o.
Actions with Grounding Tools
liai__oo_abelllbbe_iiie Clicks on specified label in the grounding result.
double Laauble_llikk_ue _lick_on label(label
Biick_on_labellaael_ pnner_ovvr__abel(tie _dd:: Double- clicks on specified label in the grounding result.
Moves the mouse to hover over specified label in the grounding result. mouse drag to label(label id): Drags the mouse to specified label the grounding result.
Information Gathering, Tasks in OSWorld require pixel-level mouse control. While GPT-4 exhibits grounding ability, using tools like SAM can further augment the screenshot with the grounding of icons in complex computer control tasks. The bounding box is helpful for GPT-4 to understand the occurrence of objects on the screen and can also be used to calculate the precise position for mouse control.
B
Libreotfic MERC
May
14 6
15
SEBI
18
38
40
20
21
22
23
41
24
28
29
30
31
32
33
34
39
35
36
37
Ω
55
63
56
57
58
59
47
48
60
61
50
62
norm
L-po
p1
L
A
4
A
65
66
68
69
Etale
Libreot
92
76
78
82
83
84
85
94
90
93
98
96 Drlecte
95
101 102
97
104
105
words, 1,900 characters
Figure 51: Augmented screenshot using CRADLE'S SAM2SOM
Reff-Refeecton. The reflection module evaluates whether previous actions have been successfully executed and determines the entire task was successful. The self-reflection module important for tasks in OSWorld, which are sequential decision -aainn problems that require re- re-planning based on the current state and previous actions. The self-reflection module also helps to identify infeasible tasks.
64

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
K.4. Application Target and Setting Challenges
Evaluations within WoWd reveal notable challenges in agents' abilities, particularly in GUI understanding and operational knowledge (Xie al., 2024). To further complete tasks in OSWorld, the agent needs advanced visual capabilities and robust GUI interaction abilities. Furthermore, the agents face challenges in leveraging lengthy raw observation and action records. The next-level approach encompasses designing more effective agent architectures that augment the agents' abilities to explore autonomously and synthesize their findings.
K.5. Case Studies
K.5.1. INFORMATION GATHERING
Seoge
Hay2 2324
Searh
LLM Powered Autonomous Agents
#buicCT Exttassted and Rallcdi
Agent System Overview
eformation exbendedd periods, often inveraping external vedior store
APls beell Exemmase thet
afhir rwwurriiiion Invivation mumeee Informatiinn vidi
Figure 52: Case Study of robust and precise GUI interaction via information gathering
With SAM the grounding tool, we prompt the agent to identify the objects in each bounding box to determine the exact position of each object. As shown in Figure 52, the agent recognized the GUI element in box 32 the Save button. In the planner, the agent chose to click on box 32 to save the PDF, resulting in success.
K.5.2. PLANNING WITH SELF-REFLECTION
We showcase how self-reflection combined with planning helps the agent complete a task by coming up with an alternative plan and validating its success.
The current task instruction is "Copy the file 'filel' each of the directories 'dirl', 'dir2', 'dir3'." " As shown Figure 53, the agent made two attempts at implementing the command but encountered emrors and warnings.
As shown in Figure 54, after observing the errors and warnings in the previous steps, the agent checked the files in the directory to debug. After confirming the file structure, the agent tried different commands.
As shown in Figure 55, after executing the new command without receiving an error message, the agent checks whether the files have been copied to the folders. After observing the result, it marks this task as a success.
65

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
A
user usergeser www.allmammmee......
ssarbeser cortual 5 EP Filet
filet diz2
dtricp
Fites dira
Set Up
cannet
Jep's Fite directory
ca
riiie3 spect
To use your carte cirtuat
diri rils fille 4122
ritet diztep
Thunderbird Files dirhhp Files dirs files dirs <p Files dirs
dirbpp
cat warnteg: source File "rttes
spectPied nore than once
benember par
Cancel
Cantrue
credentiats only stored leaaly
24
Home
Figure 53: The agent fails copy the files due using incorrect commands
A
usen wengeeer mathel-
diriep
Fitet Files fites dir
Set Up
cannet Septe
file directory
To use your cer sertiaa bochiter: cp Filet diri filet dies files diriee
Files etrice files dirs a4 riles dirs ca Files dira
cat warnteg: source file "ettes" spectFted more than once
Pictures
screenshets
Templaees
Fite3
setup-sh
cp dirt filet c fites dra Files
machtan::
project
#t-2
Fited
wwr.tall Mechhter: Filet diri filet dVz Filet dv/3
Bemember par anker
credential only stared lecilly
Home
Figure 54: The agent reflects the errors, checks the file structure and tries debug
66

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
Antuiin
Torenol
29 01229
Hserquser vertual Rachtmes
Set Up YO
project
corver
Vides
requirement txt
Thunderbird will
To your carre
Boeer strteal machtnes 5 cp Filet di1 Filed dizz cp filei du3
Fi3es
setup.sh
Jibe Doe
mechhtee:: Fitet
5 cp
filet
abdress
vetrtaa mechtness 5 mudtr dirt 44 sketr dtrz nettr
jobe dordeeeaann
witr Merchtie:: <p Files dirs Files dizz files durs
Pastaoed
virtual machhiees: ts dira; dir2: la dira
rLlet
assen wwr.teal machhtees
Cancel
Cantinue
Figure agent checks if files have already been copied
Table 15: Detailed success rates divided by domains: OS, LibreOffice Cale, LibreOffice Impress, LibreOffice Writer, Chrome, VLC Player, Thunderbird, VS Code, GIMP, and Workflow (i.e, involves multiple applications).
OS
Cale
Impress (47)
Writer (23)
VLC (17)
TB
Chrome (46)
VSC (23)
GIMP
Workflow (101)
Method
(24)
(47)
(15)
(26)
GPT-4o
8.33
0.00
6.77
4.35
16.10
0.00
4.35
4.35
3.85
5.58
GPT-40+SoM
20.83
0.00
6.77
4.35
6,53
0.00
4.35
4.35
0.00
3.60
CRADLE
16.67
0.00
4.65
8.70
6.53
0.00
8.70
0.00
38.46
5.48
67

---

CRADLE: Empowering Foundation Agents Towards General Computer Control
K.6. Quantitative Evaluation
The detailed success rates for each application are listed in Table 15. We followed the same experimental settings as the OSWorld paper, running the experiment only once. Our results show that agent performs better in the Chrome and GIMP domains. However, the difference in performance in the OS, Writer, and VSC domains is less statistically significant due to the smaller number of tasks. While improved information gathering and self-reflection empowered the agent in these domains, the complex pipeline and limitations of current grounding tools and GPT-4 hindered performance in domains like VLC and VSC. We identify these limitations as future directions for implementing the agent in real-world scenarios.
68


Reviews:
[{"review_id": "d34shHVjUa", "reviewer": "Reviewer_Cfph", "summary": "The paper proposes the General Computer Control (GCC) setting where the input is restricted to screenshots and the output to keyboard and mouse actions. To address this setting, the paper proposes Cradle, an LMM-based framework with six components:  Information\nGathering, Self-Reflection, Task Inference, Skill Curation, Action Planning, and Memory. Given a screenshot as input, Information Gathering parses visual and textual information using LMM. Self-Refelction then reasons about what happened based on the extracted information. Task Inference plans a next task from the reflected result to achieve a desired goal. Given the predicted task, Skill Curation generates necessary skills to complete it and Action Planning retrieves relevant skills to take the next action for the goal. The proposed approach is validated by four video games, five software applications, and the OSWorld benchmark with noticeable margins over the baslines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers for their valuable feedback and insightful comments. We hope our following answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.\n\n---\n\n**Q1**: About the necessity of each module of Cradle. Why not provide an ablation study? \n\n**A1**: Thanks for pointing it out. We indeed provide a comprehensive ablation study by systematically removing each module of Cradle to show the effectiveness in Appendix E3 and Table 5 (page 18-19). The ablation study shows that removing any of the modules will result in a significant performance loss. Due to the limitation of pages, we have to put the result in the appendix. As the camera-ready version will allow one more page, we will move the ablation study to the main paper for better readability. \n\n---\n\n**Q2**: Why not directly ask LMM to generate the next action? The proposed multi-staged approach can be easily affected by even a single failure in intermediate steps.\n\n**A2**: One of our baselines, ReAct, is exactly kind of a one-stage approach. It lets LMM generate the next action with CoT in one step. This method shows much worse performance than Cradle. Additionally, according to our ablation study, removing any of the modules will result in a significant performance loss. From the perspective of designing complex systems [1], systems with no redundancy or error-detection mechanisms are susceptible to single-point failures. A module dedicated to a single function is less likely to cause serious system-wide issues than one that tries to handle everything. For the multi-staged approach, the error still has the potential to be corrected by the following stages, however, the error in the single-staged approach will directly be executed into the environment and cause unrecoverable loss.    \n\n[1] Blanchard, Benjamin S., Wolter J. Fabrycky, and Walter J. Fabrycky. Systems engineering and analysis. Vol. 4. Englewood Cliffs, NJ: Prentice hall, 1990.\n\n---\n\n**Q3**: While agreeing to address the GCC setting, it would be better to see how much Cradle can be improved if it has access to APIs and their documentation for a target program.\n\n**A3**: We thank the reviewers for acknowledging our GCC setting. We want to clarify that Cradle is designed to solve the challenges that GCC presents. If provided with APIs, the agent can directly have access to the internal state and the full action space with the meaning of each action. Modules like information gathering and skill curation are less effective. The agent can even complete tasks with textual observation without visual ability. One of our baselines, Voyager, is a good example to show the performance of a multi-staged agent with API access. Since most of the software applications and video games do not provide API access, this kind of method has limited application scenarios. The setting with API and documentation will largely violate the GCC setting. We expect that the APIs and documentation access can improve the performance of Cradle, but this is beyond the scope of this paper.", "claims_and_evidence": "It seems the claims made in the submission are supported by convincing evidence.", "methods_and_evaluation_criteria": "- Why do we need these six steps? Can this be reduced to fewer steps? The necessity of each step seems not well justified.\n- A naive approach to the GCC setting is to directly ask LMM to output the next step. Why not directly ask LMM to generate the next action?\n- The proposed multi-staged approach can be easily affected by even a single failure in intermediate steps. Can the proposed approach address this? This may be particularly important for some tasks, which are often irrecoverable, such as bank accounts, privacy issues, etc.", "theoretical_claims": "No theoretical claims are made.", "experimental_designs_or_analyses": "- For Table 2, Section 4.2 describes the baselines as models using a subset of the six components used in Cradle. It is unclear whether we can say they are indeed prior work. And why not just providing an ablation study of each component instead?", "relation_to_broader_scientific_literature": "The GCC setting addresses the general format of I/O for computer control. I believe that addressing this setting can be extended to many other domains such as robotics that usually require different I/O formats for different robot bodies.", "essential_references_not_discussed": "It seems essential references are cited in this paper.", "other_strengths_and_weaknesses": "Strengths\n- The paper is generally written well and easy to follow.\n- The proposed GCC setting sounds reasonable and well-motivated. Addressing the general I/O framework seems important and necessary.\n- The paper provides extensive analyses on its method.\n\nWeaknesses\n- While agreeing to address the GCC setting, it would be better to see how much Cradle can be improved if it has access to APIs and their documentation for a target program.", "other_comments_or_suggestions": "I have no other comments.", "questions_for_authors": "All questions are made in the sections above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wIu5pMRAns", "reviewer": "Reviewer_GCyf", "summary": "This paper focuses on building a framework based on a multimodal model, specifically OpenAI’s GPT-4o, for computer use through keyboard and mouse inputs. The proposed framework consists of six distinct modules: information gathering, self-reflection, task inference, skill curation, action planning, and memory, which are employed in a prompting and agentic manner to interact with games and software applications.\n\nThe authors evaluate their proposed framework on four different games and common software applications. The experimental results in gaming show that, except for Stardew Valley, the framework achieves high completion rates in three of the tested games. However, in the case of general software applications, while the framework demonstrates the ability to accomplish specific tasks, it does not exhibit a consistently high level of performance across different applications.\n\n### update after rebuttal:\nI thank the authors for their response and have taken into account the perspectives of the other reviewers. My concerns have been partially addressed, and I have accordingly raised my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers for their valuable feedback and insightful comments. We hope our following answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.\n\n---\n\n**Q1**:  The scale of evaluation in the gaming domain is limited.\n\n**A1**: We appreciate the reviewer’s concern about the scale of evaluation. However, the selected four representative games already cover a broad spectrum of game types and gameplay, from 2D to 3D, from RPG to simulation game, from comic style to realistic style, from first-person/third-person perspective to top-down perspective, etc. To our best knowledge, no previous works were evaluated on video games with so many diverse game types and gameplay before.  Additionally, based on GCC setting, Cradle does not rely on any assumptions of playing these games, which is sufficient to show that Cradle also has the potential to be developed to thousands of games with the same types and even different types. We would like also to note that one of our key related work and baseline methods, Voyager, is only evaluated in one game.\n\nFinally, we also provide some preliminary results of applying Cradle to the extremely challenging Action RPG game, Black Myth: Wukong. Cradle still manages to defeat boss and enemies in the early stages of the game.\n\n|              Task             |   Cradle  |  \n|:-----------------------------:|:---------:|\n|       Defeat Erlang           |    100%    | \n|       Defeat WolfScout      |    30%    |  \n|       Defeat WolfSwornScout      |    20%    |  \n|       Defeat Croaky       |    40%    | \n\n---\n\n**Q2**: The framework does not demonstrate strong performance in general software applications, which is significantly lower than the SOTA model like UI-TARS and Claude. These baselines are not compared in the paper.\n\n**A2**: We deeply understand that compared to the very recent SOTA models, the performance reported by Cradle in OSWorld is not impressive enough.  However, we would like to kindly remind that all the models mentioned by the reviewers emerged very close to the ICML 2025 submission deadline.  UI-TARS is even after the ICML abstract deadline. According to the ICML policy (https://icml.cc/Conferences/2025/ReviewerInstructions), “Authors cannot expect to discuss other papers that have only been made publicly available within four months of the submission deadline. Such recent papers should be considered as concurrent and simultaneous. ” Nevertheless, we agree on the importance of these works and will include additional discussion in the related work section. \n\nImportantly, we would like to clarify that our primary claim is that Cradle represents the first framework capable of achieving strong performance across both video games and software applications. The domain-specific model, UI-TARS, which primarily demonstrates effectiveness only in software tasks, does not undermine Cradle’s broader contribution. Furthermore, UI-TARS benefits from direct training on data collected from OSWorld, naturally resulting in good performance in this specific benchmark. It still struggles with less common software applications like Cupcut, Feishu and Meitu, further validating Cradle's generalization across diverse software tasks.\nAdditionally, comparing models directly with Cradle may not be entirely appropriate. Cradle is a framework instead of a model, which needs to be initialized with a base model. Compared to UI-TARS, the base model used by Cradle (gpt-4o-0513) inherently exhibits much weaker performance on GUI tasks. Cradle's capability to significantly enhance performance relative to its base model clearly illustrates its strong generalizability and adaptability. As the base model improves, the performance of Cradle will also improve. Therefore, the improvement of general-purpose models like GPT and Claude does not have a direct competition relationship with Cradle, but a mutually reinforcing relationship instead.\n\n---\n\n**Q3**: Lack of structured description of the full framework such as presenting its workflow in an algorithmic format. \n\n**A3**: We would like to clarify that Cradle is a flexible framework, which can be customized to different tasks and environments. The main workflow has been illustrated in Figure 3. To further address the reviewers’ concerns, we provide a pseudocode to show the workflow in the following anonymous link: https://drive.google.com/file/d/15r4lveEyGaMEFDhOfZrJ8PT1DXhC6M4i/view .\n\n---\n\nWe also thank the reviewers for catching our minor oversight in the appendix, which will be fixed in the latest version.", "claims_and_evidence": "I believe the paper’s main claim, which asserts that the proposed framework (CRADLE) can effectively operate a computer in complex environments and exhibits strong generalization capabilities, is not sufficiently supported by evidence.\n\nThe validation of the framework primarily relies on experiments in four games, where it demonstrates promising results in three of them. While this provides some support for the claim, I am concerned that the scale of the evaluation is too small to convincingly establish the framework’s generalization ability. Moreover, the framework does not demonstrate strong performance in general software applications, which further weakens the generalization claim.\n\nCurrently, OSWorld is the most widely recognized benchmark for evaluating computer-use agents. The paper reports that the proposed framework achieves a score of 7.81, which is significantly lower than the state-of-the-art models on this benchmark. Even compared to the much smaller UI-TARS 7B model (which scores 18.7), the proposed framework falls short.", "methods_and_evaluation_criteria": "I find the proposed framework reasonable, but a major shortcoming is the lack of clarity in its description.\n\nIn Section 3, the authors provide a high-level overview of each module in the framework. However, they do not clearly explain how these modules interact or provide sufficient details on the end-to-end process of the framework. Even though the authors mention in the appendix that there was limited space to include more details, I believe this does not justify the absence of a structured description of the full framework—such as presenting its workflow in an algorithmic format.\n\nThe lack of detailed algorithmic descriptions makes it difficult to assess the novelty and effectiveness of the proposed approach. Additionally, many components of the framework, such as episodic memory and self-reflection, have already been widely explored in recent agentic frameworks, further raising concerns about its level of innovation.", "theoretical_claims": "The paper does not make any theoretical claims.", "experimental_designs_or_analyses": "Yes, I reviewed the experimental design and analysis used to evaluate the proposed framework.\n\nAs mentioned earlier, one major issue with the experiment is the limited scale of evaluation, particularly in the gaming domain.\n\nAnother experimental design issue is the lack of strong baseline comparisons. The authors state in Section 4.2 that there are no existing models capable of performing computer operations like their proposed framework. However, several computer-use agents have already been evaluated on the OSWorld benchmark, including Claude and UI-TARS. These models have publicly reported performance scores on OSWorld and could have been used for comparison. Furthermore, these existing agents could have been adapted for gameplay evaluations, allowing for a more rigorous comparison.", "relation_to_broader_scientific_literature": "The paper’s focus on GUI-based computer-use agents for gaming applications is novel, as most existing computer-use agents are primarily designed for browser-based tasks or basic software operations.", "essential_references_not_discussed": "The paper adequately covers the key related works.", "other_strengths_and_weaknesses": "As mentioned earlier, many of the modules used in the proposed framework—such as self-reflection—have already been widely implemented in existing agentic frameworks. However, the paper does not provide a clear explanation of its algorithmic contributions or highlight its specific innovations. Based on the current description, the proposed framework appears to be a combination of existing techniques.", "other_comments_or_suggestions": "The authors should consider revising Section A.5 in the appendix. It appears that this section contains responses from a previous submission, as seen in line 764, where the text states:\"The work mentioned by the reviewer…\"", "questions_for_authors": "Why not compare the proposed framework with existing models that have been evaluated on OSWorld?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qgPvfDHLpA", "reviewer": "Reviewer_Wqig", "summary": "The paper presents CRADLE, framework that leverages LMMs designed for General Computer Control (GCC). CRADLE operates directly through visual observations (screenshots) and generates keyboard  and mouse commands, enabling it to interact with diverse software environments without relying on specialized APIs. Its architecture comprises six core modules: 1. Information Gathering; 2. Self-Reflection; 3. Task Inference; 4. Skill Curation; 5. Action Planning, and 6. Memory. These facilitate effective interaction, learning, and adaptability.\n\nExperimentally, CRADLE states that it demonstrated notable generalization and strong performance across challenging tasks in 4 complex commercial video games (including RDR2 and Stardew Valley) and 5 real world software applications (Chrome, Outlook, Feishu, Meitu and CapCut).\n\nKey findings include CRADLE’s ability to complete extended missions in an AAA game environment, generate sophisticated procedural skills, and achieve performance comparable to or surpassing human players in several tasks, thus validating the proposed GCC setting. The authors mention being the first to evaluate and showcase a framework that can interacti with both commercial complex games and software applications.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers for their valuable feedback and insightful comments. We hope our following answers will clear up the doubts about our work, and please let us know if there is any other clarification we can provide.\n\n---\n**Q1**: About the selection of the video game and software application for evaluation. \n\n**A1**: We would like to clarify that Cradle primarily focuses on demonstrating the effectiveness of the GCC setting, enabling agents to interact with software in a unified manner. Thus, we deliberately selected four representative games that do not provide API access, so that they have not been explored in prior studies, to clearly distinguish Cradle from existing approaches. As the reviewer rightly pointed out, Minecraft is already a well-established benchmark with rich API access. Many agents have demonstrated impressive performance on it. Evaluating Cradle on Minecraft would not further strengthen our main claims. Benchmarking against specialized domain-specific agents or models is not the primary objective of our current study.\n\nOn the other side, the selection of software applications tells the same story. We deliberately selected several challenging productivity software (e.g., Feishu, Meitu and Cupcut) that are seldom explored before. Compared to video games, there are already many previous agentic works on software and software tasks are usually closer to daily life and easy to understand. Due to the limitation of pages, it is regretful that we have to put the introduction of these software and tasks, and the quantitative results on OSWorld in the appendix. We will move them back to the main paper in the camera-ready version as one more page will be provided. \n\n---\n**Q2**: Insufficient comparison against strong, existing benchmarks (e.g., OSWorld, AGUVIS, Voyager) diminishes the strength of the results.\n\n**A2**: We would like to kindly remind the reviewers that both OSWorld and Voyager are evaluated in our paper. As for AGUVIS, what they proposed is a trained VLM model instead of an agentic framework. The work is released within two months before the deadline for ICML 2025. According to the ICML policy (https://icml.cc/Conferences/2025/ReviewerInstructions), “Authors cannot expect to discuss other papers that have only been made publicly available within four months of the submission deadline. Such recent papers should be considered as concurrent and simultaneous. ” Nevertheless, we agree on the importance of these works and will include additional discussion in the related work section. \n\n---\n**Q3**: Evaluations conducted only against inexperienced humans neglect expert-level comparisons, potentially inflating agent effectiveness.\n\n**A3**: We deeply understand the reviewers’ concerns and provide supplementary comparisons with expert-level players in the following link: https://drive.google.com/file/d/1NUgtjCFhrV3B8RdCvw65LMr5NJX8pJm7/view?usp=sharing.\n\nThere indeed still a gap between Cradle and expert players, however we argue that comparing these agents to expert-level humans might be inherently unfair. Although LMMs have some high-level gameplay knowledge from large-scale internet pretraining, they lack explicit training on the low-level actions needed for specific game tasks evaluated in our study. Thus, LMMs are closer to novice players who gather some information from the internet or game wikis but have no practical experience. Inexperienced players are introduced to basic gameplay and provided the same prompts used by agents to ensure fairness.\n\nMoreover, Cradle is designed to emulate the experience of a fresh player by progressively acquiring new skills and capabilities as gameplay unfolds. At the start of the game, Cradle possesses only a limited set of atomic control skills. In contrast, expert human players already have complete mastery over all necessary skills from the beginning, providing them with a significant initial advantage. Thus, we believe the comparisons made in our study accurately reflect the agent’s true learning capabilities in relation to novice human players.\n\n---\n**Q4**: About algorithmic novelty\n\n**A4**: We deeply understand reviewers’ high criteria for improving our work to be better. We kindly remind that the selected Primary Area of this submission is Applications. Cradle focues on showing that the challenging GCC setting can be properly handled by current techniques, thus motivating more researchers and developers engaged in this setting. Limited algorithmic novelty is unnecessary to be a weakness of our submission.", "claims_and_evidence": "Work clearly demonstrates feasibility and promising generalization in complex commercial video games & sofwatware application, including initial successes with extended storyline tasks (e.g., RDR2). \n\nHowever, the claims made by the authors regarding CRADLE's strong generalization and high performance across previously unexplored complex commercial video games and software applications are only partially supported. While results in games like RDR2 and Cities: Skylines are convincing, the performance in common software applications (e.g., Chrome, Outlook, CapCut) is less impressive and insufficiently analyzed quantitatively. Additionally, comparisons are limited to inexperienced human players, omitting valuable amateur-level comparisons, which could better contextualize the agent's effectiveness, especially given possible prior LLM knowledge about these games. Most importantly, evaluation results on established software benchmarks (e.g., OSWorld) are absent from the main paper, weakening the generalization claim (are only present in appendix). Finally, relevant comparisons to stronger existing software use frameworks are missing, e.g. AGUVIS - Xu et. al. 2024.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria proposed in the paper generally align well with their stated goal of addressing the General Computer Control (GCC) setting. However, some evaluation choices limit the comprehensiveness of their claims. Notably, Minecraft, a well-established benchmark for agent generalization and lifelong learning would have been a valuable addition but is missing. Additionally, the 5 software applications they chose to analyse are not common for computer use benchmarks, and neither are they properly introduced or evaluated. Nonetheless, the chosen environments and methods generally align well with their proposed GCC setting.", "theoretical_claims": "The paper does not contain any theoretical claims.", "experimental_designs_or_analyses": "The experiments and applications are generally appropriate for the proposed GCC framework chosen. \n\nExperimental design issues: \n\n- comparison baseline was conducted by inexperienced human players; this limits insights into the agent's relative performance, especially since the used LMM  (GPT-4o) might have prior knowledge of certain games.\n- the evaluation of software tasks laks detailed quantitative analyses, such as ablation studies or stronger baseline comparisons on more common benchmarks. These omissions make it hard to fully assess CRADLE’s claimed superiority and generalization capabilities.", "relation_to_broader_scientific_literature": "The paper builds  extends recent advances in foundation agents, multimodal learning, and embodied AI. It positions CRADLE within the broader context of LMM-powered agents like Voyager (Wang et al. 2023), which demonstrated lifelong in-context learning in Minecraft, and other multimodal web agents such as WebVoyager (He et al., 2024) and Mind2Web (Deng et al., 2023). Unlike these prior works, which typically rely on domain-specific APIs or simpler interaction spaces, CRADLE introduces the General Computer Control (GCC) setting, utilizing general-purpose screenshots and executable mouse and keyboard actions without API dependence. This paper also relates closely to recent LMM-based agents designed for GUI interaction tasks (ScreenAgent, Niu et al., 2024; Voyager, Wang et al., 2023) but aims for broader generalization across both gaming environments and practical software. Given the limited algorithmic novelty, maybe the paper could be adapted and focus on strictly proposing a general purpose benchmark.", "essential_references_not_discussed": "Not specifically.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- Effectively combines multimodal inputs, reflection, skill generation, and memory into a modular and coherent framework.\n- Demonstrates practical significance and general-purpose computer-use capability through challenging commercial video games and everyday software tasks, practical significance.\n\n**Weaknesses:**\n\n- Limited quantitative evaluation, particularly in common software tasks, weakens the generalization claims.\n- Insufficient comparison against strong, existing benchmarks (e.g., OSWorld, AGUVIS, Voyager) diminishes the strength of the results.\n- Evaluations conducted only against inexperienced humans neglect expert-level comparisons, potentially inflating agent effectiveness.\n- Excessive focus on video games over common software tasks limits broader applicability insights.\n- Limited algorithmic novelty", "other_comments_or_suggestions": "No", "questions_for_authors": "1. Why did you choose to evaluate CRADLE against only human players who had never played the corresponding games before?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on the interesting and ambitious application of general computer control, and demonstrated solid experimental results across a variety of games and software applications that these challenging general computer control settings can be properly handled by current VLMs. Concerns about comparison with other methods and concurrent work were addressed by the authors in the rebuttal with no additional follow ups from the reviewer. Despite the work is largely application focused, discussing any algorithmic insights with some additional algorithmic innovations can improve the quality of the work. Overall, the work makes meaningful contributions. Please be sure to open source the framework upon acceptance of the work.
Decision: Accept (poster)