Cradle: Empowering Foundation Agents towards General Computer Control

## Abstract

Abstract Despite the success in specific tasks and scenarios, existing foundation agents, empowered by large models (LMs) and advanced tools, still cannot generalize to different scenarios, mainly due to dramatic differences in the observations and actions across scenarios. In this work, we propose the General Computer Control (GCC) setting: building foundation agents that can master any computer task by taking only screen images (and possibly audio) of the computer as input, and producing keyboard and mouse operations as output, similar to human-computer interaction. The main challenges of achieving GCC are: 1) the multimodal observations for decision-making, 2) the requirements of accurate control of keyboard and mouse, 3) the need for long-term memory and reasoning, and 4) the abilities of efficient exploration and self-improvement.
To target GCC, we introduce Cradle , an agent framework with six main modules, including: 1) information gathering to extract multi-modality information, 2) self-reflection to rethink past experiences, 3) task inference to choose the best next task, 4) skill curation for generating and updating relevant skills for given tasks, 5) action planning to generate specific operations for keyboard and mouse control, and 6) memory for storage and retrieval of past experiences and known skills.
To demonstrate the capabilities of generalization and self-improvement of Cradle , we deploy it in the complex AAA game Red Dead Redemption II, serving as a preliminary attempt towards GCC with a challenging target.
To our best knowledge, our work is the first to enable LMM-based agents to follow the main storyline and finish real missions in complex AAA games, with minimal reliance on prior knowledge or resources.

## 1 Introduction

Achieving Artificial General Intelligence (AGI) has long been a north-star goal for the AI community []. Recent foundation agents, *i.e., *agents empowered by large multimodal models (LMMs) and advanced tools, have been touted as a promising approach in pursuing AGI.
While recent research has demonstrated the success of such agents in specific scenarios or tasks, including web browsing [, , , , ], operating mobile applications [], crafting and exploration in Minecraft [, ], and some robotics scenarios [, , ]; current foundation agents still cannot generalize across different targets, mainly due to different observation and action spaces, environment dynamics (*e.g., *Minecraft and robotics), dependence on target-specific resources, or other semantic gaps (*e.g., *lack of environment feedback for actions) [].

Computers(^1^11Throughout the paper we use the term *computer* as a synonym for any user-focused computational device, *e.g., *PC, smartphone and tablet. While our description focuses on keyboard and mouse operations, it can be easily generalized to control handles and touchscreens.) are the most important and universal interface in the increasingly digital world. Computer tasks cover a wide variety of scenarios, including complex software to create digital artifacts, *e.g., *Photoshop, everyday productivity software, *e.g., *spreadsheets and word processors, apps and websites with effects in the real world, *e.g., *banking or travel booking apps, video games, *e.g., *The Sims 4 and Red Dead Redemption II (RDR2), and control of external devices, *e.g., *3D printing devices and network routers. By providing standardized universal observation (*i.e., *screen and audio) and abstract actions (*i.e., *keyboard and mouse operations), computer control is an ideal testbed to develop foundation agents for varied complex tasks in dynamic scenarios and thus have the potential to unlock insights on the path to AGI. Therefore, we propose the *General Computer Control* (GCC) setting:

There are many challenges to achieve GCC. i) observations in GCC are multimodal, which requires the alignment of the data in different modalities for better understanding and decision-making.
ii) GCC requires accurate control of device operations (keyboard and mouse) to interact with the computer. iii) It requires long-term memory to store past experiences due to the partial observability of GCC tasks and reasoning ability over experiences to reuse the knowledge to solve novel tasks. iv) GCC requires efficient exploration of the environments in a structured manner to discover better strategies and solutions autonomously, *i.e., *self-improving, which allows agents to generalize across the myriad tasks in the digital world. We discuss GCC in more detail in Section [2](#S2).

In this work, we introduce Cradle, a novel framework targeting GCC, and make a preliminary attempt towards it with a challenging target.
As shown in Figure [1](#S0.F1), distinctly from previous methods focused on controlling a web browser or software with easy access to their internal APIs and states [, ],
Cradle only takes screen images as input and outputs keyboard and mouse operations through a comprehensive reasoning process, enabling it to effectively understand the current situation and make reasonable decisions. Furthermore, we deploy Cradle in the highly acclaimed AAA game(^2^22In the video game industry, AAA (Triple-A) is a term used to classify games with typically high development budgets and production values for the technology available in their time.), Red Dead Redemption II (RDR2). We select RDR2 for our case study due to its complex blackbox control system, which epitomizes the most demanding computer tasks and enables us to evaluate the performance boundaries of our framework in such virtual environments. RDR2 is characterized by its rich and diverse information, encompassing elements like dialogues, unique icons, in-game prompts, and instructions, thus requiring the capture and interpretation of various information forms. Additionally, the game requires a broader range of keyboard and mouse interactions than typical software, such as using the mouse for navigation and combining mouse buttons and keyboard keys interactively to realize actions in the game world (which necessitate not only precise key selection, but also determination of timing, and multi-iteration combinations). We argue that by demonstrating the feasibility of playing this complex game, Cradle sheds light on its potential for GCC.

Our major contributions are summarized as follows:

- •
We propose the novel setting of General Computer Control (GCC), serving as a milestone towards AGI in the digital world, where agents take multimodal observations as inputs and output keyboard and mouse operations, similar to human-computer interactions.
- •
We propose a novel foundation agent framework (Cradle) for the GCC setting(^3^33Cradle’s codebase is open-sourced at: [https://github.com/BAAI-Agents/Cradle](https://github.com/BAAI-Agents/Cradle)), which has strong reasoning abilities, including self-reflection, task inference, and skill curation, to ensure its generalizability and self-improvement across various computer tasks.
- •
To demonstrate the capabilities of Cradle, we incorporate the powerful Large Multimodal Model (LMM) GPT-4V into our framework and deploy it in the famous AAA game RDR2, serving as a preliminary attempt towards GCC. To our best knowledge, our work is the first to enable LMM-based agents to follow the main storyline and finish real missions in complex AAA games, with minimal reliance on prior knowledge.

## 2 General Computer Control

Figure: Figure 2: Taxonomy of GCC.
Refer to caption: /html/2403.03186/assets/x2.png

##### Definition.

General Computer Control (GCC) is a setting where *an agent control a computer through only standardized human-like interactions, i.e., using the screen and (optionally) audio as input and keyboard and mouse operations as output*.
GCC provides a universal interface to any computer-based task without access to the application source code nor APIs. Compared with robotics [], another promising direction toward AGI, GCC provides a more cost-effective and controllable experimental framework, also with significant impact in real scenarios. Due to its universality, achieving GCC can be seen as a significant milestone towards AGI, starting from the digital world, and can be further combined with robotics to address also physical environments.

##### Taxonomy.

As illustrated in Figure [2](#S2.F2), there are three main categories of GCC targets: i) manipulating web software, *e.g., *websites and cloud-based services [], ii) manipulating traditional desktop software, *e.g., *office suites [], and iii) playing video games, *e.g., *Minecraft []. Among them, digital video games are believed to be the most difficult tasks mainly due to i) environment complexity, *e.g., *the open-ended world in RDR2, ii) the long-horizon decision making (not always linear), *e.g., *1K+ semantic steps in RDR2 tasks, and iii) the partial observability of tasks, *i.e., *agents need to store their past experiences and reason upon them for better decision making. Digital games have been a testbed for research on decision making, *e.g., *Atari games in reinforcement learning research []. But complex more modern games present a much more challenging target for the development of the foundation agents, due to their similarities to virtual reality applications and the real world.

##### Challenges.

There are several key challenges in targeting GCC. First, observations in GCC are multimodal, including images (video frames or screenshots), text (from the command line or included in images), and audio (*e.g., *instructions, effects, and music). For example, when playing digital games, the agent commonly needs to follow instructions or dialogue (text/audio) and interpret the world (screen images) to complete tasks.
Second, GCC requires the accurate control of keyboard and mouse to interact with the computer, where the action space of the mouse is continuous, *i.e., *locations on the screen to move and varying speed of movement, and the action space of the keyboard control is discrete, but very large, considering combinations of keys, timing, and other attributes. Furthermore, the syntax contracts for the code generated by the agent to execute such IO operations introduce additional difficulties. Third, GCC tasks are usually partially observable. For example, the player in RDR2 cannot know the information of towns until he reaches them. Therefore, GCC requires the agent to have long-term memory to store past experiences, such as past actions, outcomes, and the knowledge learned from the past experiences, for fast and better decision making. Fourth, GCC requires efficient exploration of the environments in a structured manner and discovering better strategies and solutions autonomously, *i.e., *self-improving. The ability to explore and self-improve enables agents to generalize across various tasks in GCC autonomously. More details on such challenges are discussed in Case Study [5.1.3](#S5.SS1.SSS3) and Appendices [B.1](#A2.SS1) and [B.2](#A2.SS2).

## 3 Related Work

### 3.1 LLM-based Agents for Computer Tasks

The prosperity of LLMs has broadened the potential of deploying powerful foundation models as autonomous agents for completing complex tasks in various computer applications, such as web navigation [, , ], software manipulation [, , ] and game playing [, , , ].
While previous LLM-based web agents [, , , ] show some promising results in effectively navigating, understanding, and interacting with content on webpages, they usually use raw HTML code and DOM tree as input and interact with the available element IDs, missing the rich visual patterns with key information, like icons, images, and spatial relations.

Empowered by the advanced LMMs, multimodal web agents [, , , , ] and mobile app agents [, ] have been explored. Instead of HTML source code, they usually take screenshots as input, however, they still need to use the built-in APIs to get the available interactive element IDs to execute corresponding actions.

Similar to web agents, recent works attempt to deploy LLM agents to various complex video games, such as Minecraft [, ], Starcraft II [] and Civilization-like game [] with textual observations obtained from internal APIs and pre-defined semantic actions. These domain-specific settings make them struggle with generalizing to other games, let alone websites and other software. Although JARVIS-1[] claims to interact with the environment in a human-like manner with the screenshots as input and mouse and keyboard for control, its action space is predefined as a hybrid space composed of keyboard, mouse, and API.

Though achieving promising results in specific tasks, these methods fail to generalize across various tasks, due to the inconsistent observation and action spaces. This indicates the necessity of the GCC setting, which provides a unified representation of the observation and action spaces for enormous challenging computer tasks.

Concurrent with our work, there are several works [, , , , , ] aiming to scale the web agent to more applications with screenshots as input and keyboard and mouse operations as output. However, none of them takes video games into consideration since they mainly focus on static websites and software, which greatly reduces the need for timeliness and simplifies the setting by ignoring the dynamics between adjacent screenshots, *i.e., *animations, and incomplete action space without considering the duration of the key pressed and different mouse mode.

### 3.2 Decision-Making in Video Games

Video games are ideal environments for validating agent’s various abilities due to their diversity, controllability, safety, and reproducibility, which are also believed to be the most complicated tasks in computer control.
Atari games [],
Super Mario Bros [], Google Research Football [], StarCraft II [, ], Minecraft [, , ] etc, have been the popular environments and benchmarks for reinforcement learning (RL) agents. Besides, RL agents also exhibit impressive performance in Dota II [], Quake III [], Gran Turismo [] and Diplomacy []. However, to abstract complex computer control, these environments usually simplify the whole action space (*i.e., *keyboard and mouse movement) to pre-defined domain-specific actions, which vary from game to game, exacerbating the poor generalization of RL agents across environments.

Discarding the semantics in the observation and action also leads to low efficiency.
LLMs enable decision-making agents to leverage semantic information in the environments, which dramatically improves the reasoning ability of agents, Without any training process, Voyager [] can efficiently learn to finish long-horizontal complex tasks through code generation. However, it heavily relies on the built-in API tool, Mineflayer, to obtain internal information and execute high-level actions, which are not available in other games. TextStarCraft II [] and CivRealm [] also suffer the same issue. Therefore, closed-source AAA games with rich textual and visual information are rarely to be explored. Pre-trained with videos with action labels, VPT [] manages to output mouse and keyboard control with raw screenshots as input without any additional information. However, collecting videos with action labels is time-consuming and costly, which is difficult to generalize to multiple environments.

In a nutshell, to our best knowledge, there are currently no agents under the GCC setting, reported to show superior performance and generalization in complex video games or across computer tasks. In this work, we make a preliminary attempt to apply our framework to the epic AAA game RDR2, under the GCC setting.

## 4 The Cradle Framework

To pursue GCC, we propose Cradle, illustrated in Figure [3](#S4.F3), an agent framework that can properly handle the challenges GCC presents, *i.e., *observing and interacting with any environment and dealing with corresponding information and semantic gaps, without relying on any special API that is not available to a typical computer user. A framework for GCC should have the ability to understand and interpret computer screens and dynamic changes between consecutive frames (and possibly audio) from arbitrary software and be able to generate reasonable computer control actions to be executed reliably. This suggests a multimodal model with powerful vision and reasoning capabilities, in addition to rich knowledge of computer UI and control, is a requirement. In this work, we leverage GPT-4V as the framework’s backbone model, which is currently one of the most capable LMM models available.

Figure: Figure 3: An overview of the Cradle framework. Cradle takes video from the computer screen as input and outputs computer keyboard and mouse control determined through inner reasoning.
Refer to caption: /html/2403.03186/assets/x3.png

### 4.1 Environment IO

Although the input from the environment may cross multiple modalities (*e.g., *visual, audio), in this work, we cast Cradle to consume videos (sequences of screenshots) of the screen as input and produce keyboard and mouse control commands as output. As this matches the most common setting faced by users, it is a pragmatic scenario in a preliminary attempt towards GCC.

Information Gathering.
As shown in Figure [3](#S4.F3), to capture all information relevant to understanding the recent situation and perform further reasoning, Cradle takes a video recording since the last executed action as input and needs to make sure information is properly extracted from it, which includes both textual and visual information. Textual information usually includes content (headings and paragraphs), navigation labels (menus and links), notifications, and instructions to convey messages and guide users, which usually depend on the OCR ability of LMM models. On the other hand, visual information includes layout, imagery (visual contents of the screenshot itself and icons), animations, and UI elements to enhance user experience and interface design, which poses high requirements for the spatial perception and visual understanding of LMM models.

Skill and Action Generation Under the GCC setting, the only way to interact with the environment is through keyboard and mouse operations. To bridge the gap between actions outputted by the framework and the OS-level executable actions, Cradle uses the LMM to generate code function as semantic-level skills which encapsulate lower-level keyboard and mouse control, *e.g., *def move_forward(duration): key_hold(‘W’, duration).
We then let the LMM instantiate these skill functions into executable action by specifying any necessary parametric aspects, such as duration, position and speed, *e.g., *move_forward(duration = 2).

Action Execution. After the LMM generates actions and decides to execute them in the environment, an Executor is then triggered to map these semantic actions to the final
OS-level keyboard and mouse commands to interact with the environment.

### 4.2 Reasoning

Based on the extracted information from the video observations and relevant information from its memory, Cradle needs to reason taking into account incomplete information and semantic gaps and then make the next decision. This process is analogous to “reflect on the past, summarize the present, and plan for the future”, which is broken down into the following modules.

Self-Reflection. The reflection module initially evaluates whether the last executed action was successfully carried out and whether the task was completed. Sequential key screenshots from the last video observation, along with the previous context for action planning and task inference are fed to the LMM for reasoning. Additionally, we also request the LMM to provide an analysis of any failure. This valuable information enables Cradle to try and remedy inappropriate decisions or less-than-ideal actions. Furthermore, reflection can also be leveraged to inform re-planning of the task and bring the agent closer to target task completion, better understand the factors that led to previous successes, or suggest how to update or improve specific skills.

Task Inference. After reflecting on the outcome of the last step, Cradle needs to analyze the current situation to infer the most suitable task for the current moment. We let the LMM estimate the highest priority task to perform and when to stop an ongoing task and start a new one, prompting it with the key screenshots, the long-term summary of past experiences, and the latest reflection results.

Skill Curation.
As a new task is selected, Cradle needs to prepare the tactics to accomplish it, by retrieving useful skills from the procedural memory, updating skills, or generating new ones. Similarly to [, ], skills in Cradle are represented as code functions; a form is both flexible and interpretable, and easy for LMMs to understand and maintain. Atomic skills are usually made up of simple calls to keyboard and mouse control, *e.g., *key_press() to press a given key, which can then be extended and rewritten into more complex composite skills.

Action Planning.
To complete a given task, the LMM needs to select the appropriate skills from the curated skill set and instantiate these skills into a sequence of executable actions by specifying any necessary parametric aspects (*e.g., *duration, position, and target) according to the inferred task, last action, and long-term summary. The action sequence is then fed to the Executor for interaction with the environment.
It is important to note that in complex digital games, the effective action space is composed not only of keyboard and mouse function calls per se, but also involves timing and cross-action interaction, semantic mapping of term usage on screen to action-specific details, among other factors.
Moreover, performing actions in the environment is non-trivial also as mapping code execution to its effects is not always explicit. That is, action execution can result in no error from the code or game, but still be incorrect or ineffective.

### 4.3 Memory

Cradle stores and maintains all the useful information provided by the environment and the LMM’s output through a memory module, consisting of episodic memory and procedural memory.

Episodic Memory. Episodic memory is used to maintain current and past experiences, including key screenshots from each video observation, and all LMM output, *e.g., *textual and visual information, actions, tasks, and reasoning from each module. To facilitate retrieval and storage, periodical summarization is conducted to abstract recently added multimodal information into long-term summaries. The incorporation of episodic memory enables Cradle to effectively retain crucial information over extended periods, thereby enhancing its decision-making capabilities.

Procedural Memory. This memory is specific to storing and retrieving skills in code form, which can be learned from scratch or pre-defined in procedural memory. Upon skill curation, skills can be added, updated, or composed in the memory. The most relevant skills for a given task and situation will be retrieved to support action planning, so, as Cradle continuously acquires new skills during interaction, it is critical that this memory can effectively calculate skill relevance.

Figure: Figure 4: The detailed illustration of how Cradle is instantiated as a game agent to play RDR2.
Refer to caption: /html/2403.03186/assets/x4.png

## 5 Empirical Studies

As shown in Figure [4](#S4.F4), we deploy Cradle as a game-playing agent in the renowned AAA game RDR2, showcasing our framework for GCC with a complex target. To the best of our knowledge, this is the first work to explore such a challenging game under the GCC setting, without access to any internal game state or API (*i.e., *the agent has to interact with the game in a human-like manner).
RDR2 is a typical 3D RPG-like game with classical keyboard and mouse controls and movie-like realistic graphics.
The game’s progression unveils a mix of story content, dialogues and instructions, and helpful tips for interactive game mechanisms, which are leveraged by our agent to independently expand the skills in the procedural memory from scratch.
Demonstrating our agent’s capability to navigate the world and complete tasks following the main storyline in RDR2 underscores the significant potential of our framework in advancing towards GCC. We provide a more detailed introduction of the game in Appendix [A.1](#A1.SS1). The implementation details of the framework and all LMM prompts can be found in Appendices [B.1](#A2.SS1) and [B.3](#A2.SS3).

Objective. To emulate a new human player learning to play the game from scratch, including in-game beginner tutorials and hints, we mainly focus on the first two missions of the main storyline in Chapter I, *Explore Shelter* and *Rescue John*, including horse riding, NPC following, house exploration, weapons selection, combat with enemies and wolves, etc.
Typically, it takes a novice human player around 40 minutes of the gameplay duration to finish these missions. Seldom previous work attempted this kind of challenge with super long-horizon tasks and rich semantic environments.
Besides the missions in the main storyline, we also evaluate Cradle in the open-ended world in Chapter II with a mission, *Buy supply*, where the agent is instructed to go to the General Store in Valentine town from the camp for supplies preparation. For this open-ended task, seldom in-game guidance will appear. The agent needs to analyze and propose feasible solutions to complete the mission. We provide a brief introduction to these tasks in Appendix [A.2](#A1.SS2).

Observations and Action Space. Strictly following the GCC setting, our agent takes the video of the screen as input and outputs keyboard and mouse operations to play the game.
To lower the frequency of interaction with the backbone model, the video recorder takes a game screenshot every 0.5 seconds, which proves to be sufficient for information gathering without missing any important information.
For the action space, we categorize the keyboard and mouse actions into 4 key categories: press(key, duration), hold(key, duration), release(key), and pointer_move(x, y), which can be combined in different ways to form combos, use keys in fast sequence, or coordinate timings. Skill code needs to be generated by the agent in order to utilize such functions and affordances so executed actions take effect. More details on the action space are provided in [B.1](#A2.SS1).

Figure: Figure 5: Case study of self-reflection on re-trying a failed task. Task instruction and context require the agent to equip the gun. A wrong weapon (knife) is first selected, but the agent equips the gun after self-reflection. Only relevant modules are shown for better readability, though all modules (Figure [3](#S4.F3)) are executed per iteration.
Refer to caption: /html/2403.03186/assets/x5.png

Figure: Figure 6: Skill code generation based on in-game instructions. As the storyline progresses, the game will continually provide prompts on how to use a new skill via keystrokes or utilizing the mouse.
Refer to caption: /html/2403.03186/assets/x6.png

### 5.1 Case Studies

Here we present a few case studies for more in-depth discussion of the framework capabilities and the challenges of the GCC setting.

#### 5.1.1 Self-Reflection

Self-reflection is an essential component in Cradle as it allows our framework reasoning to correct previous mistakes or address ineffective actions taken in-game. Figure [5](#S5.F5) provides an example of the self-reflection module. The task requires the agent to select a weapon to equip, in the context of the “Protect Dutch” task. Initially, the agent selects a knife as its weapon by chance, but since the game requires a gun to be chosen, this is incorrect and the game still prompts the player to re-open the weapon wheel. The self-reflection module is able to determine that the previous action was incorrect and on a subsequent iteration the agent successfully opts for the gun, correctly fulfilling the task requirement and advancing to the next stage in the story.

#### 5.1.2 Skill Curation

For skill curation, we first provide GPT-4V with examples of general mouse and keyboard control APIs, e.g., io_env.key_press and io_env.mouse_click. Figure [6](#S5.F6) shows that GPT-4V can capture and understand the prompts appearing on screenshots, *i.e., *icons and text, and strictly follow the provided skill examples using our IO interface to generate correct skill code. Moreover, GPT-4V also generates comments in the code to demonstrate the functionality of this skill, which are essential for computing similarity and relevance with a given task during skill retrieval. The quality of the generated comment directly determines the results of skill retrieval, and further impacts reasoning to action planning. Curation can also re-generate code for a given skill, which is useful if GPT-4V wrongly recognized a key or mouse button in a previous iteration.

#### 5.1.3 Action Execution and Feedback

Proper reasoning about environment feedback is critical due to the generality of the GCC setting and the level of abstraction to interact with the complex game world. The semantic gaps between the execution of an action, its effects in the game world, and observing the relevant outcomes for further reasoning lead to several potential issues that Cradle needs to deal with. Such issues can be categorized into four major cases:

Lack of grounding feedback.
In many situations, due to the lack of precise information from the environment, it can be difficult for the system to deduce the applicability or outcome of a given action. For example, when picking an item from the floor, the action may fail due to the distance to the object not yet being close enough. Or, if within pick up range, the chosen action may not exactly apply due to other factors (*e.g., *character’s package is full).

Even if the right action is selected and executed successfully, the agent still needs to figure out its results from the partial visual observation of the game world. If the agent needs to pick or manipulate an object that is occluded from view, the action may execute correctly, but no outcome can be seen.

A representative example in RDR2 happens when the agent tries to pick up its gun from the floor after a fight. Getting to the right distance, without completely occluding the object, can lead to multiple re-trials. Figure [7(a)](#S5.F7.sf1) showcases a situation where, though the character is already standing near the gun (as seen in the minimap), it’s still not possible to pick it up.

Previous efforts [, ] that utilize in-game state APIs unreasonably bypass such issues by leveraging internal structured information from the game and the full semantics of responses (data) or failures (error messages).

Imprecise timing in IO-level calls.
This issue is caused by the ambiguity in the game instructions or differences in specific in-game action behaviors, where even the execution of a correct action may fail due to minor timing mismatches. For example, when executing an action like ‘open cabinet’, which requires pressing the ‘r’ key on the keyboard, if the press is too fast, no effect happens in the game world. However, as there is no visual change in the game nor other forms of feedback, it can be difficult for GPT-4V to figure out if an inappropriate action was chosen at this game state or if the minor timing factor was the problem. Pressing the key for longer triggers an animation around the button (only if the helper menu is on screen), but this is easily missed and any key release before the circle completes also results in no effect. Figure [7(b)](#S5.F7.sf2) illustrates the situation.

The same problem also manifests in other situations in the game, where pressing the same key for longer triggers a completely different action (*e.g., *lightly pressing the ‘left alt’ key vs. holding it for longer).

Change in the semantics of key and button.
A somewhat similar situation occurs when the same keyboard key or mouse button gets attributed different semantics in different situations (or even in a multi-step action). GPT-4V may decide to execute a given skill, but the original semantics no longer hold. The lack of in-game effect parallels the previous situations. Worse yet, an undesired effect will confuse the system regarding the correct action being selected or not.

For example, when approaching a farm in the beginning of the game, the agent needs to hitch the horse to a pole to continue. The operation to perform the action consists of pressing the ‘e’ key near a hitching post (as shown in Figure [7(c)](#S5.F7.sf3)). However, the same ‘e’ key press is the only constituting step in other actions with different semantics, like *dismount the horse* or *open the door*. Wrongly triggering a horse dismount at the situation shown in the figure can lead to undesired side effects, *i.e., *it may mislead the system about the actual effects of the action or affect the planning of which next actions to perform.

Interference issues.
Lastly, completion of some actions requires the correct execution of multiple steps sequentially, which could be interrupted in many ways not related to the agent’s own actions. Without the use of APIs that expose internal states or other forms of feedback, it is much harder for the agent to decide when to repeat sub-actions or try different strategies. For example, if the agents gets shot and loses aim while in combat, or an unrelated in-game animation is triggered mid-action, cancelling it.

Since there is no direct environment feedback, the agent needs to carefully analyze the situation and try to infer if any action step needs re-execution.

Figure: (a) ‘Pick gun’ unavailable.
Refer to caption: /html/2403.03186/assets/x7.png

### 5.2 Quantitative Evaluation

Figure: Figure 8: Cradle performance on seven representative tasks in RDR2. The first six tasks are from the two missions in the main storyline and the last one is one of the open-ended missions. Every task is tested five times with a maximum of ten minutes in-game time. The test will also terminate if the mission fails.
Refer to caption: /html/2403.03186/assets/x10.png

To illustrate the effectiveness and importance of different modules in Cradle to its overall performance, we evaluate the framework on seven representative tasks from the main storyline and open-ended missions, compared with two ablation-like baselines: Cradle without Self-Reflection and Cradle without Task Inference. Except for the task of *Protect Dutch* which involves a fast-paced gun battle, *Search house*, which requires the agent to explore a complex indoor environment, and the open-ended task with long-horizon, Cradle can complete all tasks in the main storyline consistently. Moreover, even for these two complex tasks, Cradle also achieves a significantly higher success rate than disabling either Self-Reflection or Task Inference. The most common failure for the *Buy supply* open-world complex task happens when the agent riding a horse in the lively Valentine town crashes into a passerby or a carriage.

Without task inference, the agent can still leverage the extracted instructional text information and long-term summary to infer both task and goal in the action planning process, which can lead to some successful task completions. However, over time the instructional text disappears from the screen and the long-term summary will be diluted as relevant information will be forgotten due to the recently added entries, which explains why it fails across all tasks, but still has a reasonable success rate in the *Equip shotgun* task, where the task has a shorter horizon and its instructional text appears at the bottom of the screen. For the open-ended mission, *Buy supply*, without self-reflection to propose short-horizon goals, the agent even struggles with leaving the camp.

On the other hand, without self-reflection, Cradle struggles with movement, especially when the agent is blocked by obstacles, which is difficult for action planning to notice and address. Without an independent critic to reflect on the current situation, GPT-4V tends to still trust its previous reasoning and believe that the actions were carried out successfully. As *Search House* is a complex indoor task with various pieces of furniture as obstacles, we can see Cradle without Self-Reflection exhibits extremely poor performance in it, as expected. Without self-reflection, the agent also struggles with entering the store and is always confused about whether the task is finished.

Figure: (a) GPT-4V fails to recognize obstacles in the environment and the position of the shopkeeper.
Refer to caption: /html/2403.03186/assets/x11.png

### 5.3 Limitations of GPT-4V

Deploying Cradle in this complex game, RDR2, requires the backbone LMM model to deal with multimodal input, which revealed several limitations of GPT-4V that needs external tools to provide
additional groundings to improve the overall framework performance. More details are provided in Appendix [B.1](#A2.SS1).

Spatial Perception. As shown in Figure [9(a)](#S5.F9.sf1), GPT-4V’s spatial-visual recognition capability is insufficient for precise fine-grained control, particularly in detecting whether the character is being or going to be blocked and estimating the accurate relative positions of target objects.
These led to the need to augment certain images to provide auxiliary visual clues for decision-making, *i.e., *bounding boxes of possible target objects.

Icon Understanding. GPT-4V also struggles with domain-specific concepts, such as unique icons within the game, which may represent specific targets or refer to certain mouse and key actions. As shown in Figure [9(b)](#S5.F9.sf2), GPT-4V fails to recognize the left shift, right mouse button, and space icons. Attempts to incorporate few-shot learning to improve image understanding cannot be generalized. Therefore, we match prepared pattern templates, *e.g., *icon images, against each screenshot to continuously detect and highlight any appearing icons.

Figure [9(c)](#S5.F9.sf3) also demonstrates that GPT-4V cannot understand the mini-map correctly, which usually contains key information about the position and direction of the character. Failure of localization leads the agent to sometimes get lost in the town and miss the task target. Although the above issues can be slightly alleviated by providing additional few-shot examples, it can only have an obvious effect if we crop the image and provide the GPT-4V with the region exactly containing the icon to be recognized, which makes the issue intractable.

History Processing. Moreover, GPT-4V can easily get distracted by irrelevant information in longer contexts, resulting in hallucinations. For example, when action planning utilizes too many historical screenshots, GPT-4V may confuse past and present frames. Additionally, GPT-4V frequently generates output not adhering to the rules in the provided prompts. To mitigate the issue of hallucinations, we more strictly control input information by further summarizing long-term memory.

World Understanding. Lastly, the absence of an RDR2 world model limits GPT-4V’s understanding of the consequences of its actions in the game. This often results in inappropriate action selection, such as overestimating the necessary adjustments for aligning targets or misjudging the duration required for certain actions. To alleviate this problem, we introduced extra prompt rules regarding action parameters and more flexibility into the self-reflection module.

## 6 Conclusions and Future Work

In this work, we introduce GCC, a general and challenging setting with a unified and standard interface for control of diverse software (via screenshots, and keyboard and mouse operations), paving the way towards general foundation agents across all digital world tasks.

To properly address the challenges GCC presents, we propose a novel framework (Cradle) that enables LMM-based agents to succeed in such an impactful setting. We further showcase its effectiveness in a complex AAA digital game, Red Dead Redemption II. Cradle exhibits strong performance in learning new skills, following the game storyline, and accomplishing real missions in the game. To the best of our knowledge, this is the first LMM-based agent that has managed to complete concrete missions from scratch in AAA games.

Although our agent can still face difficulties in some tasks, Cradle serves as a pioneering work to develop more powerful LMM-based general agents across computer control tasks, combining both further framework enhancements and new advances in LMMs.

Future work in Cradle aims to extend it to support a broader spectrum of games, such as simulation and strategy games, as well as various software applications, to demonstrate its exceptional adaptability across diverse environments. This endeavor aims to substantiate the hypothesis that Cradle can seamlessly interact with any software, transforming it into an interactive benchmark to evaluate the multifaceted capabilities of foundation agents. Incorporating audio as input is also key in important tasks and is necessary to meet the GCC setting, which introduces the additional challenge of dealing with the simultaneity of multiple modalities. Furthermore, the interactive data collected through framework development and execution will be instrumental in future efforts, from training better models to enhancing world modeling, thereby enhancing agent capabilities and usefulness for a wide range of users.

## Appendix A Red Dead Redemption II

### A.1 Introduction to RDR2

Red Dead Redemption II (RDR2) is an epic AAA Western-themed action-adventure game by Rockstar Games. As one of the most famous and highest-selling games in the world, it is widely acknowledged for its movie-like realistic scenes, rich storylines, and immersive open-ended world. The game applies a typical role-playing game (RPG) control system, played from a first- or third-person perspective, which uses WASD for movement, mouse control for view changing, first- or third-person shooting for combat, and inventory and manipulation.

For most of the game, players need to control the main character, Arthur Morgan, upon choosing to complete mission scenarios following the main storyline. Otherwise, they can freely explore the interactive world, such as go hunting, fishing, chatting with non-player characters (NPCs), training horses, witnessing or partaking in random events, and participating in side quests. As the main storyline progresses, different skills are gradually unlocked. As a close-source commercial game, no APIs are available for obtaining additional game-internal information nor pre-defined automation actions. Following its characteristics, this game serves as a fitting and challenging environment for the GCC setting.

### A.2 RDR2 Tasks

**Table 1: Tasks in the first two missions of RDR2. *Difficulty* refers to how hard it is for our agent to accomplish the corresponding tasks. Figures [10](#A1.F10) and [11](#A1.F11) showcase snapshots of each task (specific sub-figures marked in parenthesis in the table).**
| Mission 1: Explore shelter | Description | Difficulty |
| --- | --- | --- |
| Follow Dutch (Fig. [10(a)](#A1.F10.sf1)) | Follow Dutch to the small town, by riding the horse. | Easy |
| Hitch horse (Fig. [10(b)](#A1.F10.sf2)) | Dismount at the hitching post, after reaching the town. | Easy |
| Go to shed (Fig. [10(c)](#A1.F10.sf3)) | Move to the nearby shed to take cover. | Easy |
| Choose weapon (Fig. [10(d)](#A1.F10.sf4)) | Choose the correct weapon to prepare for combat. | Hard |
| Protect Dutch (Fig. [10(e)](#A1.F10.sf5)) | Protect Dutch from the enemies through shooting. | Hard |
| Search house (Fig. [10(f)](#A1.F10.sf6)) | Follow Dutch to enter the house and search for supplies. | Hard |
| Eat something (Fig. [10(g)](#A1.F10.sf7)) | Open Satchel and eat a provision to restore some Health Core. | Medium |
| Investigate barn (Fig. [10(h)](#A1.F10.sf8)) | Leave the horse and go to the barn to investigate. | Easy |
| Defeat O’Driscoll (Fig. [10(i)](#A1.F10.sf9)) | Fight with the enemy hidden in the barn. | Medium |
| Pick up equipment (Fig. [10(j)](#A1.F10.sf10)) | Find and pick up gun and hat lost during barn fight. | Hard |
| Lead horse (Fig. [10(k)](#A1.F10.sf11)) | Calm and Lead the horse out of the barn to hitching post. | Medium |
| Mission 2: Rescue John |  |  |
| Follow Javier (Fig. [11(a)](#A1.F11.sf1)) | Follow Javier through the snow mountain to look for John. | Medium |
| Equip shotgun (Fig. [11(b)](#A1.F11.sf2)) | Equip correct shotgun to prepare for combat. | Medium |
| Look for John (Fig. [11(c)](#A1.F11.sf3)) | Crouch down, enter narrow tunnel, and climb up cliffs. | Hard |
| Shoot wolves (Fig. [11(d)](#A1.F11.sf4)) | Shoot wolves that will attack you and companions. | Hard |
| Protect Javier from wolves (Fig. [11(e)](#A1.F11.sf5)) | Ride horse and protect your companions from wolves. | Hard |

**Table 2: Tasks in the open-ended mission, *Buy supply* in RDR2. *Difficulty* refers to how hard it is for our agent to accomplish the corresponding tasks. Figure [12](#A1.F12) showcases snapshots of each task (specific sub-figures marked in parenthesis in the table).**
| Open-ended mission: Buy supply | Description | Difficulty |
| --- | --- | --- |
| Find horse (Fig. [12(a)](#A1.F12.sf1)) | Find and mount the horse in the camp. | Medium |
| Prepare to navigate to saloon (Fig. [12(b)](#A1.F12.sf2)) | Open map, find the saloon and create waypoint. | Medium |
| Go to saloon (Fig. [12(c)](#A1.F12.sf3)) | Ride horse to the saloon. | Easy |
| Prepare to navigate to shop (Fig. [12(d)](#A1.F12.sf4)) | Open map, find the general store and create waypoint. | Medium |
| Go to shop (Fig. [12(e)](#A1.F12.sf5)) | Ride horse to the shop. | Hard |
| Enter shop (Fig. [12(f)](#A1.F12.sf6)) | Dismount the horse and enter the shop. | Hard |
| Talk to shopkeeper (Fig. [12(g)](#A1.F12.sf7)) | Approach the shopkeeper and talk. | Easy |
| Buy target product (Fig. [12(h)](#A1.F12.sf8)) | Open the menu, find and buy the target product. | Medium |

Figure: (a) Follow Dutch
Refer to caption: /html/2403.03186/assets/subfig/01_Follow_Dutch.jpeg

Figure: (a) Follow Javier
Refer to caption: /html/2403.03186/assets/subfig/12_Follow_Javier.jpeg

Figure: (a) Find horse
Refer to caption: /html/2403.03186/assets/subfig/17_Find_horse.jpeg

Tables [1](#A1.T1) and  [2](#A1.T2) provide a brief introduction of each task in the first two missions of the main game storyline and an open-ended mission, along with approximate estimates of their difficulty. Due to GPT-4V’s poor performance in spatial understanding and fine-manipulation skills, it can be challenging for our agent to perform certain actions, like entering or leaving a building, or getting to precise indoor locations to retrieve specific items. Additionally, the high latency of GPT-4V’s responses also makes it harder for an agent to deal with time-sensitive events, *e.g., *during combat.

## Appendix B Cradle in GCC

### B.1 Implementation Details

As shown in Figure [4](#S4.F4), strictly following the GCC setting, our agent takes the video of the screen as input and outputs keyboard and mouse operations to interact with the computer and the game. An observation thread is responsible for the collection of video frames from the screen and each video clip records the whole in-game process since executing the last action.

We employ GPT-4V(ision) [], *gpt-4-vision-preview*, currently one of the most capable LMM models, as the framework’s backbone model. To lower the frequency of interaction with backbone models, the video recorder captures a game screenshot every 0.5 seconds, which proves to be sufficient for information gathering without missing any important information.

Information Gathering. To extract keyframes from the video observation, we utilize the VideoSubFinder tool (^4^44VideoSubFinder standalone tool - [https://sourceforge.net/projects/videosubfinder/](https://sourceforge.net/projects/videosubfinder/)), a professional subtitle discovery and extraction tool. These keyframes usually contain rich meaningful textual information in the game, which are highly relevant to the completion of tasks and missions (such as character status, location, dialogues, in-game prompts and tips, etc.) We use GPT-4V to extract and categorize all the meaningful contexts in these keyframes and perform OCR, and call this processing "gathering text information". Then, to save interactions with GPT-4V, we only let GPT-4V provide a detailed description of the last frame of the video.

While GPT-4V exhibits impressive visual understanding abilities across various CV tasks, we find that it struggles with spatial reasoning and recognizing some game-specific icons. To address these limitations, we add a visual augmentation sub-module within our *Information Gathering* module. This augmentation step serves two main purposes: i) utilize Grounding DINO [], an open-set object detector, to output precise bounding boxes of possible targets in an image and serve as spatial clues for GPT-4V; and ii) perform template matching [] to provide icon recognition grounding truth for GPT-4V when interpreting instructions or menus shown on screen. As LMM capabilities mature, it should be possible to disable such augmentation.

Self-Reflection. The reflection module mainly serves to evaluate whether the previously executed action was successfully carried out and whether the current executing task is finished. To achieve this, we uniformly sample at most 8 sequential frames from the video observation since the execution of the last action and use GPT-4V to estimate the success of its execution. Additionally, we expect GPT-4V can also provide analysis for any failure of the last action (*e.g., *the move-forward action failed and the cause could be the agent was blocked by an obstacle). With such valuable information as input for *Action Planning*, including the failure/success of the last action and the corresponding analysis, the agent is capable of attempting to remedy an inappropriate decision or action execution.

Moreover, some actions require prolonged durations, such as holding down specific keys, which can coexist or interfere with other actions decided by subsequent decisions. Consequently, the reflection module must also decide whether an ongoing action should continue to be executed. Furthermore, self-reflection can be leveraged to dissect why the last action failed to bring the agent close to the target task completion, better understand the factors that led to the successful completion of the preceding task, and so on.

Besides, we observe that instead of providing GPT-4V with sequential high-resolution images for self-reflection

Task Inference. During gameplay, we let GPT-4V propose the current task to perform whenever it believes it is time to start a new task. GPT-4V also outputs whether the task is a long- or short-horizon task when proposing a new task. Long-horizon tasks, such as traveling to a location, typically require multiple iterations, whereas short-horizon tasks, like picking up an item or conversing with someone, involve fewer iterations. The agent will follow the newly generated task for the next $h$ interactions. After $h$ interactions, the agent returns to the last long-horizon task in the stack. Deciding on a binary task horizon is much easier and more robust for GPT-4V, than re-planning at every iteration. Since a long-horizon task frequently includes multiple short-horizon sub-tasks, this implementation also helps avoid forgetting the long-horizon tasks under execution.

To save interactions with GPT-4V, we implement this as part of the *Information Gathering* module. When GPT-4V detects instructional text in the recent observation, it will directly generate a task description and determine whether it is a long- or a shot-horizon task.

Skill Curation.
As shown in Figure [6](#S5.F6), during gameplay, instructions often appear on the screen, such as “press [Q] to take over" and “hold [TAB] to view your stored weapons," which serve as essential directives for completing current and future tasks proficiently. To translate these textual and iconic instructions into executable mouse and keyboard actions, we provide GPT-4V with skill code samples using interfaces for mouse and keyboard manipulation, e.g., io_env.key_press (to press a key), io_env.key_hold (to keep a key held), io_env.mouse_click (to click the mouse button), and io_env.mouse_hold (to keep the mouse button held). GPT-4V is required to strictly follow the provided interfaces and examples to generate the corresponding code for new skills. Moreover, GPT-4V is required to include documentation/comments within the generated code, delineating the functionality of each skill. The *Procedural Memory* sub-module where skills are stored will then check whether the code is valid, whether the format of documentation is right, and whether any skill with the same name already exists. If all conditions are passed, the newly generated skill is persisted for future utilization.

Similarly to *Task Inference*, we also implement a simple version of this module inside *Information Gathering* to reduce interactions with GPT-4V. When GPT-4V detects and classifies some instructional text in the recent observation, which usually contains key/button hints, it will directly generate the corresponding code and description.

Action Planning. Upon execution of this module, we first retrieve the top k relevant skills for the task from procedural memory, alongside the newly generated skills. We then provide GPT-4V with the current task, the set of retrieved skills, and other information collected in *Information Gathering* that may be helpful for decision-making (e.g., recent screenshots with corresponding descriptions, previous decisions, and examples) and let it suggest which skills should be executed. We also request that GPT-4V provide the reasons to choose these skills, which increases the accuracy, stability, and explainability of skill selection and thus greatly improves framework performance. While GPT-4V sometimes may generate a sequence of actions, we currently only execute the first one, and perform *Self-Reflection*, since we observe a tendency for the second action to usually suffer from severe hallucinations.

Action Execution. For the action space, we categorize the keyboard and mouse actions into 4 key categories: press(key, duration), hold(key, duration), release(key), and pointer_move(x, y); which can be combined in different ways to form combos, use keys in fast sequence, or coordinate timings. Skill code needs to be generated by the agent in order to utilize such functions and affordances so executed actions take effect. Table [3](#A2.T3) illustrates Cradle’s action space.

**Table 3: Action space in the Cradle framework, including action attributes. Coordinate system is either *absolute* or *relative*. Actions with durations can be either *synchronous* or *asynchronous*.**
| Type | Action | Attributes |
| --- | --- | --- |
| Keyboard | Key Press | Key name (string), |
| Key press duration (seconds:float) |  |  |
| Key Hold | Key name (string), |  |
| Key press duration (seconds:float), |  |  |
| Wait behaviour (sync/async) |  |  |
| Key Release | Key name (string) |  |
| Key Combo | Key names (strings), |  |
| Key combo duration (seconds:float), |  |  |
| Wait behaviour (sync/async) |  |  |
| Hotkey | Key names (strings), |  |
| Hotkey sequence duration (seconds:float), |  |  |
| Wait behaviour (sync/async) |  |  |
| Text Type | String to type (string), |  |
| Typing duration (seconds:float) |  |  |
| Mouse | Button Click | Mouse button (left/middle/right), |
| Button click duration (seconds:float) |  |  |
| Button Hold | Mouse button (left/middle/right), |  |
| Button hold duration (seconds:float), |  |  |
| Wait behaviour (sync/async) |  |  |
| Button Release | Mouse button (left/middle/right) |  |
| Button Double Click | Mouse button (left/middle/right), |  |
| Button click duration (seconds:float) |  |  |
| Move/Hover/Point | Mouse position (width:int, height:int), |  |
| Mouse speed (seconds:float), |  |  |
| Coordinate system (enum), |  |  |
| Tween mode (enum) (^5^55Tweening is not currently utilized in the prototype. But this is a OS-supported functionality that can greatly enhance user experience.) |  |  |
| Drag | Mouse final position (width:int, height:int), |  |
| Mouse speed (seconds:float), |  |  |
| Coordinate system (enum), |  |  |
| Tween mode (enum) |  |  |
| Scroll | Orientation (vertical/horizontal), |  |
| Distance (clicks:int), |  |  |
| Duration (seconds:float) |  |  |
| Wait Action | Wait time (float) |  |

It is important to note that, while some works (*e.g., *AssistantGUI [] and OmniACT []) use *PyAutoGUI* (^6^66Python library that provides a cross-platform GUI automation module - [https://github.com/asweigart/pyautogui](https://github.com/asweigart/pyautogui)) for keyboard and mouse control, this approach does not work in all applications, particularly in modern video games using DirectX (^7^77Microsoft DirectX graphics provides a set of APIs for high-performance multimedia apps - [https://learn.microsoft.com/en-us/windows/win32/directx](https://learn.microsoft.com/en-us/windows/win32/directx)). Moreover, such work chooses to expose a subset of the library functionality in its action space, ignoring dimensions like press duration and movement speed, which are critical in many scenarios.

To ensure wide game compatibility and avoid interference between different code libraries, in our current implementation we use the similar *PyDirectInput* library (^8^88Python library encapsulating Microsoft’s *DirectInput* calls for convenience manipulating keyboard keys - [https://github.com/learncodebygaming/pydirectinput](https://github.com/learncodebygaming/pydirectinput)) only for keyboard control, and write our own abstraction for mouse control (using the *ctypes* library (^9^99Python library that provides C compatible data types, and allows calling functions in DLL/.so binaries - [https://docs.python.org/3/library/ctypes.html](https://docs.python.org/3/library/ctypes.html)) to send low-level mouse commands to the operating system. For increased portability and ease of maintenance, all keyboard and mouse control is encapsulated in a class, called *IO_env*.

Unlike the conventional mouse operation in standard software, where the cursor is restricted to a 2D grid and remains visible on the screen to navigate and interact with elements, the utilization of the mouse in 3D games like RDR2 introduces a varied control scheme. In menu screens, the mouse behaves traditionally, offering familiar point-and-click functionality. However, during gameplay, the mouse cursor disappears, requiring players to move the mouse according to specific action semantics. For example, to alter the character’s viewpoint, the player needs to map the actual mouse movement to in-game direction angle changes, which differ in magnitude in the X and Y axes. Another special transition applies to shooting mode, where the front sight is fixed at the center of the screen, and players must maneuver the mouse to align the sight with target enemies. This nuanced approach to mouse control in different contexts adds an extra layer of challenge to general computer handling, showcasing the adaptability required in game environments, compared to regular software applications.

Procedural Memory. Procedural memory stores pre-defined basic skills and the generated skills captured from the *skill curation*. However, as we continuously obtain new skills during game playing, the number of skills in procedural memory keeps increasing, and it is hard for GPT-4V to precisely select the most suitable skill from the large memory. Thus, we propose first to retrieve a subset of skills, that are relevant to the given task, and then let GPT-4V select the most suitable one from the subset. In the skill retrieval, we pre-compute the embeddings of the documentations (code, comments and descriptions) of skill functions, which describe the skill functionality, and compute the embedding of the given task. Then we compute the cosine similarities between the skill documentation embeddings and the task embedding. The higher similarity means that the skill’s functionality is more relevant to the given task. We select the most similar 10 skills as the subset. Using similarity matching to select a small candidate set simplifies the process of choosing skills. We use OpenAI’s *text-embedding-ada-002 model* to compute embeddings for the skill information.

In our target setting, We intend to let the agent learn all skills from scratch, to the extent possible for the main storyline missions. The procedural memory is initialized with only preliminary skills for basic movement, which are not clearly provided by the in-game tutorial and guidance.

- •
*turn(degree)*, *move_forward(duration)*: Since the game does not precisely introduce how to move in the world through in-game instructions, we provide these two basic actions in advance, so GPT-4V can perform basic mobility, while greatly reducing the number of calls to the model.
- •
*shoot(x, y)*: RDR2 also does not provide detailed instructions on how to aim and shoot. Moreover, due to limitations with GPT-4V spatial reasoning and the need to sometimes augment images with object bounding boxes, we provide such basic skill for the agent to complete relevant tasks.
- •
*select_item_at(x, y)*: Similarly to *shoot()*, due to the lack of instructions, we provide such skill for the agent to move the mouse to a certain place to select a given item.

Beyond these basic atomic low-level actions, we introduce a few composite skills to facilitate the game playing progress. The agent should be able to complete tasks using only the basic skills above and the skills it learns, but these composite skills streamline the process by greatly reducing calls to the backend model.

- •
*turn_and_move_forward(degree, duration)*: This skill is just a simple composition of *turn()* and *move_forward()* to save frequent calls to GPT-4V in a common sequence.
- •
*follow(duration)* and *navigate_path(duration)*: In RDR2, tasks often guide players to follow NPCs or generated paths (red lines) in the minimap to certain locations. This can be reliably accomplished via the basic movement skills, but requires numerous interactions with GPT-4V. To control both cost and time budgets involving GPT-4V’s responses, we leverage the information shown in the minimap to implement a composite skill to follow target NPCs or red lines for a short set of game iterations.
- •
*fight()*: As output of an interaction with GPT-4V, the agent will only take one action per step. However, though the action is generated correctly, specifically in fight scenarios, the action frequency may not be high enough to defeat an opponent. In order to allow sub-second punches, we provide a pre-defined action that wraps this multi-action punching, which can be selected by GPT-4V to effectively win fights.

For the open-ended mission, since the agent skips all the tutorials in Chapter I, we provide all the necessary skills in the procedural memory at the beginning of the mission.

Episodic Memory stores all the useful information provided by the environment and LMM, which consists of short-term memory and long-term summary.

The short-term memory stores the screenshots within the recent k interactions in game playing and the corresponding information from other modules, *e.g., *screenshot descriptions, task guidance, actions, and reasoning. We set k to five, and it can be regarded as the memory length. Information stored over k interactions ago will be forgotten from direct short-term memory. Empirically, we found that recent information is crucial for decision-making, while a too-long memory length would cause hallucinations. In addition, other modules continuously retrieve recent information from short-term memory and update the short-term memory by storing the newest information.

For some long-horizon tasks, short-term memory is not enough. This is because the completion of a long-horizon task might require historical information from a long steps ago. For example, the agent might do a series of short-horizon tasks during a long-horizon task, which makes the original long-horizon task forgotten in short-term memory. To maintain the long-term valuable information while avoiding the long-token burden of GPT-4V, we propose a recurrent information summary as long-term memory, which is the text summarization of experiences in game playing, including the ongoing task, the past entities that the player met, and the past behaviors of the player and NPCs.

In more detail, we provide GPT-4V with the summarization before the current screenshot and the recent screenshots with corresponding descriptions, and GPT-4V will make a new summarization by organizing the tasks, entities, and behaviors in the time order with sentence number restriction. Then we update the summarization to be the newly generated one, which includes the information in the current screenshot. The recurrent summarization update, inspired by RNN, achieves linear-time inference by preserving a hidden state that encapsulates historical input. This method ensures the compactness of summarization token lengths and recent input data. Furthermore, the incorporation of long-term memory enables the agent to effectively retain crucial information over extended periods, thereby enhancing decision-making capabilities.

### B.2 Application Target and Setting Challenges

Choosing a complex game like RDR2 introduces its own set of challenges beyond just the GCC setting and a complex application target.

#### B.2.1 Model latency and Game pause

RDR2 is a dynamic game where events happen in a real-time-like manner. As such, it is unfeasible to simply wait for the latency in GPT-4V responses. During this time any event could happen in the game and invalidate the state passed to the backend model. To sidestep this issue we utilize the *pause* feature in the game, similar to works like Voyager [], to freeze the game until a response from the backend model is received. However, differently from games like Minecraft, RDR2 doesn’t provide instantaneous pause/resume. This leads to additional latency before and after actions. Moreover, pausing the game interferes with the ongoing keyboard/mouse state, which must be reset on the resume.

#### B.2.2 Mouse modes

Differently from traditional applications which use a flat 2D grid-based GUI (from web apps to productivity suites), games, and especially 3D games, make heavy use of different styles or modes of mouse interaction. Movement of the mouse can be mapped to arcs in a 3D view cone, distances moved on the X and Y axis may be non-uniform and differ across scenes in the game, mouse coordinates may move beyond the game screen region, etc. In order to successfully manipulate the game, both the action space must provide for functionality to address these issues and the agent itself must be able to effectively leverage such affordances.

In RDR2, for example, the mouse movement behavior is different in map-mode, regular gameplay, and when aiming the weapon in hunting/combat situations.

#### B.2.3 Multi-episodes

Agent development in digital environments can also usually greatly benefit from multi-episode simulation and learning. But this requires being able to precisely control the environment where the agent is executed and reset it to exact conditions between runs.

RDR2 again is a more challenging environment, where: i) saving the state of the game is not freely allowed (only at specific milestones), and ii) saving/re-loading the game does not precisely preserve the state of the game. For example, if an agent moves to a location in the woods, opens the map, and creates a path on it to a third location, even if allowed to save the game, when the game resumes, the map is closed and no path is defined, as well as the location of the agents reverts to the last game checkpoint location (not where the save happened).

### B.3 Cradle Prompts

Figure: Prompt 1: Gather Text Information prompt.

Figure: Prompt 2: Gather Situation Information prompt.

Figure: Prompt 3: Information Summary prompt.

Figure: Prompt 4: Self-Reflection prompt.

Figure: Prompt 5: Action Planning prompt.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: In terms of action planning, whether PDDLstream could have been explored?

If the pipeline so heavily depends on LMM, then if LMM become very superb in near future as encompassing the architecture internally, then the pipeline might not be needed and go towards dissolution.

How would have Deep RL fared under this scenario with game related rewards?

Is there any LMM query prompting level innovation like chain of thought incorporation

The metrics of evaluation vary per dataset / game / software category. How can this be generalized to reduce human input?

What mathematics and philosophy is behind the fact that this chain of 6 modules will work perfectly, and not any other combination?

Were there any robust human user studies done domain and demography nostic?

I was wondering how will it work for CAPTCHA task?
soundness: 3
strengths: The system architecture is well thought of. It uses LMMs to generate code functions as semantic-level skills, which encapsulate lower-level keyboard and mouse control.

Codebase has been shared and the appendix is extensive with figures and supporting material, case studies. Hard work has been put, but mostly on the engineering side.

The usage in software tasks is something fresh, although tech is a join of off the shelf SOTA.

Formulation / design of the template prompts is good with example and format. This seems to be in line with Robogen [https://github.com/Genesis-Embodied-AI/RoboGen] philosophy.
summary: The paper proposes General Computer Control (GCC) setting in a pursuit to build foundation agents that can master ANY computer task via the universal human-style interface by receiving input from screens and audio and outputting keyboard and mouse actions. The solution approach backed by LMM is broken into 6 modules - 1) information gathering 2) self-reflection 3) task inference 4) skill curation 5) action planning. and 6) memory. Each of the module is challenging and having it's own pros & cons and it is good to see a pipeline stitched in the paper, whose efficacy is further demonstrated in different categories of tasks. Overall, this is a good systems paper with limited novelty and contribution in the representation learning space.
weaknesses: To establish the approach firmly, it is recommended that the authors start the paper with clear input & output example. Fig 15 comes later in Appendix, but is cumbersome to fit in front. The system architecture at front part of paper is too much information right at front, without diving in the actual technical challenges. I think a higher level pipeline [input, output, objective] should have been shown and then specific details on sub-objectives and I/O.

Episodic Memory - there is a lot of important recent work, specifically in egocentric video - FPS. What novelty is bought here is not clear as this is one of the driving factors of success of downstream task [in lieu of earlier skill learning].

Reasoning is also handled by LMM as it is, thereby giving credit to the LMM paper, instead of this paper. At least some sort of improvement LMM++ was expected. 

Not sure if the approach is generalization to different domains and how much depends on perception part? More errors there -> more errors downstream.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 1
questions: Are the authors proposing the GCC (General Computer Control) setting as a contribution/novel aspect of this work? Could the authors clarify how the proposed GCC differs from the general approach to learning control from pixels *Human-level control through deep reinforcement learning* [Mnih et al 2016]?
- How is a 'step' defined in the context of the games/cradle? Is one step=one action selected by cradle? How many mouse/keyboard interactions can happen in one step?
- In table 1 are the humans limited in the amount of time they get to spend to complete the tasks?
- How customized do the prompts for each module in Cradle need to be for each domain it is applied to? Is there anything that can be shared across multiple domain without heavy customization?
- What happens to visual information/screenshots during the summarization process into episodic memory?
- In appendix H.4 how is "visual prompting" different from the information gathering used in other scenarios? If the augmented screenshot is not fe d to the information gathering modules, where does it go?
soundness: 3
strengths: I do applaud the immense amount of effort that has been put into this work and a big strength of this work is the evaluation across a wide variety of task types that include game and non-game settings. The results presented also seem to be relatively strong. The primary contribution is an exploration of what ingredients are needed to get an LMM like GPT-4o to complete a range of tasks when limited to input via pixels and output via keyboard and mouse actions (vs api's provided by these software) in a similar manner to the way a human would control these systems. The system also has to some degree infer what actions it can take in the world and how to do so which is also challenging to do.
summary: The paper presents an LLM powered system/framework for solving tasks by direct observation of screen state (i.e. from pixels) and direct manipulation of computer controls such as  mouse and keyboard. The system comprises of 6 high level modules and the authors demonstrate performance on a range of difficult tasks including open-world game playing and UI navigation on a benchmark of typical computer use/productivity tasks. The novelty lies in the development of a particular framework to use pretrained VLM based controller (as opposed to a model learned from scratch) to perform control from pixels (as opposed to a tailored representation of the state of the world).


Note:
I've updated score after the author rebuttal/clarification.
weaknesses: I think my primary concerns are around the presentation of the work, overall I found it quite hard to extract takeaways about the design of the system and to understand the experiment results from the *main body of the paper*. This is partly because of the scope of the work and the evaluation scenarios, but I think more could be done to bring the main arguments and results into the body of the paper rather than in the appendices (and the appendices are quite long). Overall I'd encourage the authors to consider more what are they key things they would want a reader to takeaway from the paper (and potentially apply to future systems). 

Some specific examples of the challenges I had:
- Its quite hard to understand the design/implementation of the 6 modules in Cradle. Aside from high level descriptions of the *purpose* of the module little information is presented about how they function in the main text. I think parts of the *general implementation* that is currently in the appendix need to move to the main body.
- It is hard to understand the critical differences and similarities between this framework and that used in other systems including ones cited such as, Describe, Explain, Plan and Select, Jarvis-1, etc. Is it the addition of external tools to prompt VLM perception of the scene, or a particular component that you would say is the main advancement in the design of such systems.
- Clearer discussion of the bootstrapping ) the system requires to get started in the various scenarios would be helpful to discuss upfront (e.g. initial skills, and domain information provided in the prompts for each component). It seems like there is quite a bit of upfront work to get Cradle working in a new domain. These are listed in one of the appendices, but some guiding principles/base requirements would be useful for readers to know.  Quantitatively the authors could for example present the number of low-level and composite skills provided for each scenario and the number of generated skills to complete the task successfully.
- A clearer presentation of the quantitative task performance before diving into qualitative analysis of the many different behaviors would make it easier to follow:
	- The paper focuses a fair amount on the results from RDR2, but could better contextualize the tasks themselves (e.g. their difficulty) and the behavior of the system when performing them e.g. how many different discrete actions need to be executed to complete each task—does following an NPC only require executing a single *follow* action (that is my impression after reading the appendix).
	- What is the success rate of the system on RDR2 out of the 5 runs? Do they all compete successfully? 
- The baseline comparisons in section 4.1. do not seem all that useful/informative to me. It is not clear what is being 'ablated' compared to Cradle (GPT-4o) in Table 3. While i think the goal was to try and benchmark in some way against an existing system. I think it would be better to focus on ablations of the different parts of the system itself such as in 4.2. My understanding from reading this section is that the main difference in the Reflextion-like is the lack of "Task Inference" stage, but looking at the ablations in the next section, ablating Task Inference had relatively low impact compared to the other components? Am i understanding this correctly?
- The ablations in 4.2 could be expanded to at least one non-game scenario tasks. The authors present the system as one that **generalizes across multiple domains** and it would be useful for readers to know what elements of the system are most helpful for enabling that generalization or indeed if non-video game domains need the same set of components as video game domains. I'd encourage the authors to run the ablations over the OSWorld benchmark or a benchmark like WebArena (which has wider adoption) that enables automatic evaluation
	- Another ablation that I think would make this a stronger paper is a version of Cradle where the LLM doesn't have access to tools such as Grounding DINO, SAM and others. Is that they key ingredient to make systems like this effective.
- Some of the figures and tables are hard to understand
	- Table 1: What are the units for *"Farm Clearup (Grids Num)"*  and *"Avg Haggling (Count)"*? 
	- Table 1: When you use the plus or minus sign, is that reflecting the range in both directions? Cities Skylines Human population is 415+-416, which would imply some player had -1 population? This table might be clearer if it indicated the min and max separately.
	- Table 4: I think this should be be "Figure" 4. I do find it hard to read, what do the shaded areas represent? They seem to overlap a fair amount, how should we interpret this?
	- Figure 6 is quite busy, the annotations look like the data points being plotted, I don't know if the chart is helped by the connecting lines. It might be more helpful to see average number (and std-dev) of steps per task rather than cumulative steps to allow readers to better understand performance on hard vs difficult tasks. The top half of the figure (the two maps) isn't all that helpful and could give you back space to bring in more information about the system.

I am open to increasing my score, particularly if the authors can help articulate what the main things they think the community should learn/takeaway from this paper (other than the existence of the system)

**Smaller nitpicks (does not affect the score)**

L094: "We managed to prove that commercial software is out-of-box testbeds under our framework." — Not very clear what this sentence means.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Line 1197: Could the authors clarify the specific visual prompting techniques employed, particularly the extent of manual curation involved? Greater transparency here would help assess the generalizability of the framework.

Were task-specific prompts manually written, or were they generated—at least partially—by LMMs? If generated, what iterative methods were employed to refine these prompts?

Can the authors elaborate on the design rationale for each module, discussing why each module was considered necessary and the factors underlying their inclusion, rather than directly presenting each module?

For novel tasks, how much time and expertise are required to develop prompts and input videos for CRADLE to generalize effectively? An exploration of these requirements could provide a clearer picture of the framework's adaptability.
soundness: 3
strengths: This framework demonstrates considerable complexity, with extensive evaluations across video games and diverse software applications. The use of a standard action space is particularly effective, allowing the framework to generalize across applications with minimal adaptation.

The system's performance is noteworthy; however, it's somewhat unclear to what extent this success is attributable to the baseline human players' lack of expertise in the evaluated games, as they had limited practice compared to the agent, which was afforded many trial steps.

The ablation study, examining each module’s contribution, provides valuable insights.
summary: The paper introduces CRADLE, a modular framework utilizing large multimodal models (LMMs) for general computer control (GCC). The authors evaluate its capabilities on both video games and software applications, illustrating the system's adaptability across various domains. This presents an interesting contribution to the conference and offers valuable insights for advancing agent-based control systems.
The framework is composed of six distinct modules: information gathering, self-reflection, task inference, skill curation, action planning, and memory.
weaknesses: While the evaluations are extensive, the number of runs and human participants remains relatively small to achieve robust statistical significance. Given the complexity and resources required for such evaluations, the authors might consider acknowledging this limitation explicitly in the paper.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: I only have a general question about how the authors propose to address the issues with impaired visual capability. Specifically, in situations like the one cited below as it seems that even if the agent can recognize every object in the scene, it would still struggle due to a lack of "attention" or focus.

> L359: “Additionally, indoor tasks like Search for Supplies are also challenging due to GPT4-o’s limited spatial perception, which finds it difficult to locate target objects and ends up circling aimlessly around the house. Moreover, the room contains numerous interactive items unrelated to the task, resulting in much more steps for the agent to complete the task.”
soundness: 4
strengths: This paper presents a well-designed framework that effectively leverages LLMs to build generalized agents capable of performing tasks across diverse domains. The agent's modular design makes it easier to extend or modify specific functionalities for different tasks and environments.

By translating language-based capabilities into embodied actions (using mouse and keyboard inputs), the paper showcases an innovative approach to applying LLMs in environments that require physical interaction without relying on a constrained, manually crafted action space.

The paper includes a comprehensive benchmark covering a variety of tasks, along with a thorough and detailed evaluation. The appendix is also highly informative, providing extensive details on every aspect of the system.

The agent's demonstrated ability to perform tasks in widely-used applications like web browsers and media-editing software suggests strong potential for real-world applications and scalability to more practical, non-gaming environments.
summary: This paper presents an LLM-powered agent designed to interact with various virtual environments, processing screenshots as inputs and executing actions through mouse and keyboard controls, encapsulated within functions generated by the underlying language model. The agent is composed of several modules, including self-reflection, memory, and action planning. The authors demonstrate the agent's versatility across a wide range of environments, from complex, long-duration open-world action games to planning and decision-making games, as well as everyday software applications like web browsers and image and video editing tools.
weaknesses: In several parts of the paper, the authors attribute the agent’s poor performance on certain tasks to the limited OCR and visual capabilities of the underlying LLM. However, no formal evaluation of these limitations is presented in the main paper. While Appendix E5 briefly discusses GPT-4's perception limitations, it lacks a systematic analysis or quantifiable assessment.

The visual perception issues are understandable in scenarios with low visibility, such as in the game RDR, but are less justified in standard software applications. When discussing CRADLE's performance on these applications, the authors cite difficulties in recognizing button states and locations due to non-standardized layouts of some tools used, stating that “lacking prior knowledge by GPT-4 leads to the failure of task inference and selecting the correct skills.” This reliance on prior layout familiarity poses a limitation to the agent’s ability to generalize, in my opinion. Ideally, the agent should identify interactive elements based on general characteristics, rather than relying on previously encountered layouts, which undermines its robustness in unfamiliar software environments.

Meta-review: This paper details an ambitious effort to develop foundational agents for "general" computer control. As the reviewers pointed out, the framework is quite complex so the tasks it deals with. Reviewer pm9Q praised that "The system architecture is well thought of." Similarly, Reviewer bghc mentioned that "This paper presents a well-designed framework that effectively leverages LLMs to build generalized agents capable of performing tasks across diverse domains."  The reviewers also see the evaluation to be comprehensive.

That said, I feel it is difficult to grasp the fundamental breakthrough the work brings or a clear takeaway message. Reviewer pm9Q voiced the same concern by saying "I think my primary concerns are around the presentation of the work, overall I found it quite hard to extract takeaways about the design of the system..." The approach of using "a human-style interface (screen input and keyboard/mouse output)" is hardly novel as there have been a rich body of work over the past few years along the same direction. Cradle seems to be a massive integration of many existing components for sub tasks such as information gathering, self-reflection, task inference, skill curation, action planning, and memory. I feel each of these components is unsolved in their own right, as we have seen these papers in the past. 

The reviewers generally agree that the paper presents a well-designed system with strong empirical results across a variety of tasks, showcasing the potential of LLM-based agents in the GCC setting. However, they also raise concerns about the novelty of the individual modules and the clarity of the presentation, particularly regarding the implementation details and the generalization ability of the framework. The authors addressed some of the concerns by clarifying the scope of their contributions, emphasizing the novelty of the GCC setting and the Cradle framework as a whole, and providing more details on the implementation and generalization aspects. They also acknowledged the limitations of their work and suggested future directions.

Overall, AC thinks the paper touches on many critical problems in the space of GCC. Yet, the work remains at the level of a coarse integration and lacks crisp research takeaways that can benefit following researchers to further explore the topic,

Decision: Reject