Large Action Models: From Inception to Implementation
Authors: Lu Wang
Fangkai Yang
Chaoyun Zhang
Junting Lu
Jiaxu Qian
Shilin He
Pu Zhao
Bo Qiao
He Huang
Si Qin
Qisheng Su
Jiayi Ye
Yudi Zhang
Jian-Guang Lou
Qingwei Lin
Saravan Rajmohan
Dongmei Zhang
Qi Zhang
Venue: Accepted by TMLR ()

Abstract: As AI continues to advance, there is a growing demand for systems that go beyond language-based assistance and move toward intelligent agents capable of performing real-world actions. This evolution requires the transition from traditional Large Language Models (LLMs), which excel at generating textual responses, to Large Action Models (LAMs), designed for action generation and execution within dynamic environments. Enabled by agent systems, LAMs hold the potential to transform AI from passive language understanding to active task completion, marking a significant milestone in the progression toward artificial general intelligence.

In this paper, we present a comprehensive framework for developing LAMs, offering a systematic approach to their creation, from inception to deployment. We begin with an overview of LAMs, highlighting their unique characteristics and delineating their differences from LLMs. Using a Windows OS-based agent as a case study, we provide a detailed, step-by-step guide on the key stages of LAM development, including data collection, model training, environment integration, grounding, and evaluation. This generalizable workflow can serve as a blueprint for creating functional LAMs in various application domains. We conclude by identifying the current limitations of LAMs and discussing directions for future research and industrial deployment, emphasizing the challenges and opportunities that lie ahead in realizing the full potential of LAMs in real-world applications.

Published in Transactions on Machine Learning Research (07/2025)
Large Action Models: From Inception to Implementation
Chaoyun Zhang" Junting Lu², Jiaxu Qian', Shilin He', Pu Zhao', Bo Qiao', Ray Huang', Si Qin', Qisheng Su², Jiayi Ye Yudi Zhang', Jian-Guang Lou', Qingwei Lin', Saravan Rajmohan', Dongmei Zhang', Qi Zhang'
Lu Wang Fangkai Yang!;
'Microsoft.
PPeking University
ZZhejiang University
'Eindhoven University of Technology
Reviewed on OpenReview: chooymzhhangJJullccrootttcc
{wlu, fangkalyang,
Abstract
As AI continues to advance, there is a growing demand for systems that go beyond language- based assistance and move toward intelligent agents capable of performing real-world actions. This evolution requires the transition from traditional Large Language Models (LLMs), which excel at generating textual responses, to Large Action Models (LAMs), designed for action generation and execution within dynamic environments. Enabled by agent systems, LAMs hold the potential to transform AI from passive language understanding active lask completion, marking a significant milestone the progression toward artificial general intelligence.
In this paper, present comprehensive framework for developing LAMs, offering systematic approach to their creation, from inception to deployment. We begin with an overview of LAMs, highlighting their unique characteristies and delincating their differences from LLMs. Using a Windows OS-based agent as a case study, we provide a detailed, step-by-step guide on the key stages LAM development, including data collection, model training, environment integration, grounding, and evaluation. This generalizable workflow can serve as blueprint for creating functional LAMs in various application domains. We conclude by identifying the current limitations of LAMs and discussing directions for future research and industrial deployment, emphasizing the challenges and opportunities that lie ahead in realizing the full potential of LAMs in real-world applications.
1 Introduction
In recent years, large language models (LLMs) have demonstrated remarkable advancements across range of natural language processing (NLP) tasks (Wei et al. 2022a Brown 2020 Yang et al. 2023). These models, often incorporating multiple modalities such as language, vision, and speech, have become foundational in
numerous AT-driven applications (Thrrunavukarasu et al. 2023 Rubenstein et al. 2023 Wang et al. 2023a
Jiang et al. 2024). Their success is evident in systems like question answering in conversational agents (Ma et al. 2023), ; code generation GitHub Copilot (Yetistireu et al., (023), and improved search capabilities in platforms like Bing (Thomas et al. 2024). The key strengths of LLMs namely their vast knowledge, ability to support multimodal inputs, and capacity for human-like response-have propelled them to the forefront of AI research (Minaee et al. 2024 Their capability to generalize via zero-shot learning has further expanded the horizons of what AI systems can achieve, making significant contributions to the productivity of both everyday tasks and specialized professional activities. These innovations mark an important milestone on the path toward artificial general intelligence (AGI) (Feng et al. 2024
FEqual contribution.

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Action Trajectories
Task:
Create slide base docft
Plan:
Fine- tueint
Open the draft. docy and read the content.
Create new PowerPoint file.
LLM
Action:
Open ("draft.docx")
For addd....
LAM
Textual Action Output
Agent
Click (Button! NNww") "left")
oo
User Queries
Textual Output
Task Requests
Interaction
Chatbot
How the weather todby?
Imprrove the wiiting the thest
Iuy a
style
Code generation
What is the fugune?
Wr Python function for
Set up meeting with
Fliin the form with the Excel...
Polish emai...
Send an ssrall to
O
Environment
Figure 1: The transition from LLMs to LAMs.
While LLMs excel at generating text, they remain limited in their ability to interact with manipulate the real world [Wang et al.) 2024b). Many real-world tasks require actions beyond language-such as automating software operations or controlling physical devices (Gao el al. 2024). Bridging this gap demands models that not only understand language but can act on in dynamic environments.
This transition from understanding exccution is non-trivial. Real-world tasks often require multi-step planning, precise action sequencing, and robust adaptation all areas where current LLMs fall short (Yao
et al. 2020 Kalakonda et al. 2023 Valmeekam al. 2022). LLMs are typically optimized for general-purpost
outputs, lacking the task specificity needed for reliable, grounded execution (Ling et al. 2023h.
To address these challenges, we propose Large Action Models (LAMs) LLMs equipped to perform actions in
both digital and physical environments (He et al. 2024 Zhang et al. 2025ch. LAMs interpret intent, generate structured action plans, and interact with real systems through agents. This represents a critical shift from
passive text generation to active, tasp-comp intelligence.
LAMs are often built upon the foundation of LLMs, but the transition from LLMs to LAMs is neither straightforward nor seamless, shown in Figure [ The process of transforming LLM into functional LAM involves multiple intricate stages, each requiring substantial effort and expertise. First, it is essential to collect comprehensive datasets that capture user requests, environmental states, and corresponding actions Deng et al., 2023). These data serve as the basis for training or fine-tuning LLMs to perform actions hather than merely generate text. This stage involves the integration of advanced training techniques that enable the model to understand and execute actions within specific environments (Hong et al. (024)). While GPT-4o demonstrates strong zero-shot capabilities, trained LAMs are better suited for real-world deployment scenarios where latency, cost, and task specialization are critical. In particular, the latency and inference cost associated with GPT-4o can be prohibitive in production settings (especially in high-throughput interactive environments like GUI automation) making such delays unacceptable. Once the LAM has been trained, it must be incorporated into an agent system that can effectively interact with its environment. This system typically includes components for gathering observations, utilizing tools, maintaining memory, and implementing foedback loops. These components are critical for ensuring that the LAM can not only exccute actions but also adapt its behavior based on real-time feedback and evolving situations (Zhang et al. 2025a The integration of these elements enhances the LAM's capacity to perform tasks autonomously, interact meaningfully with its surroundings, and make decisions that are grounded in the context of its environment.
A critical step in developing Large Action Models (LAMs) is rigorous evaluation (Xie et al. 2024). as LAMs directly impact their environments. This paper presents a practical guide to building LAMs from LLMs, using a GUI agent on Windows OS as a case study. We outline a full pipeline- from data collection to training and grounding- designed for safe, adaptable deployment. Our approach generalizes to other domains and highlights LAMs a key step toward action-oriented AT, bringing us closer to AGT.
2

---

Published in Transactions on Machine Learning Research (07 (225))
LLM
Step 1: Open an online shopping website.
Step 2: Search for "jacket for men "
Step 3: Go through all jackets
Buy jacket for men.
Step
Step
Step 3
LAM
Figure 2: The functional distinction between LLMs and LAMs.
2 From LLMs to LAMs
Large Action Models (LAMs) build on the capabilitics of Large Language Models (LLMs) but are specifically optimized to execule grounded actions in real-world environments. Unlike LLMs that generate text, LAMs translate user intent into executable operations, allowing them to interact with physical or digital systems to complete tasks (Zeng et al. 2023
Figure illustrates this functional shift. For instance, LLM may describe how to buy a jacket, while LAM can complete the task by navigating the website and processing the checkout steps. This transformation from passive understanding to active execution enables Al agents to deliver practical, task-completing outcomes.
Although often smaller than general-purpose LLMs, LAMs achieve superior performance within specific domains. Their specialization allows for faster inference, lower resource usage, and improved deployment in constrained environments. Training a LAM typically involves domain-specific data, targeted fine-tuning, and integration with agent frameworks that support real-time observation, planning, and tool use (Zhang al, 2025a).
2.1 Key Characteristics of LAMs
Interpretation of User Intent. LAMs process abstract multimodal inputs (such as text, images,
voice) and infer intent through dialogue and contextual reasoning (Chen et al. 2024a Shah et al. 2023 They convert user requests into high-level task plans. Action Generation. LAMs produce executable actions grounded in the current system state and task context. These actions span GUI operations, API calls, robot control, and code generation (Carta et al. 2023
Dynamic Planning. LAASs break down tasks into multi-step sequences and dynamically adapt as the environment changes (Guan et al. 2023 Shinn al. 2023 ensuring robust execution in dynamic or uncertain settings.
Domain Specialization. Trained for specific applications, LAMs incorporate domain-relevant constraints and affordances. Their compact architecture and focused functionality enhance efficiency, accuracy, and deployment feasibility (Cheng et. al. 2024).
In summary, LAMs represent a mcaningful advancement beyond traditional LLMs. between passive text-based agents and those capable of performing real-world actions in digital or physical environments.
They bridge the gap
3

---

Published in Transactions on Machine Learning Research (07 2025)
3
3 Offline/Online
Data Collection and Preparation
Test data
Evaluation
Took (sautily the style of the slice.
Tosk:
Sct
Task
pie Sarnell the welioc adfthheeet
Sondanerall to
Ery
Crell
Plan:
Input
Output
Execution results
Deoen toom cosertk.
Action
Action:
Agent
Action:
-Opmn
2
Model
Action Executors
Actions
Training
Memory
Feedback
LAM
Environment
Integration and Grounding
LLM
Figure 3: The process pipeline for LAM development and implementation.
2.2 From Inception to Implementation
LAMs have the potential to significantly extend the impact of LLMs by enabling tangible interactions with real-world environments. To harness this potential, an LAM must be developed from the ground up and deployed within a real-world application, allowing it to operate effectively in a physical environment. This process involves 5 critical steps, as shown in Figure 3
1. Data Collection and Preparation (Section 3): The first step involves gathering and curating the necessary data for the specific use case. This includes not only user queries but also environmental context, potential actions, and any other relevant data required to train the LAM effectively. The data must. undergo cleaning and pre- properocssingg before it is used for training or fine-tuning a LAM.
2. Model Training (Section 4
Using the prepared data, the next step is to train the LAM. This training process can involve various techniques such as supervised fine-tuning and reinforcement learning to ensure the model can perform the desired actions accurately and efficiently.
3. Offline Evaluation (Section 5): After obtaining the LAM, evaluate its performance using an offline dataset to verify its reliability in controlled, static environment.
4. Integration and Grounding (Section 6):
The LAM is integrated into an agent framework that serves as its operational platform. This involves grounding the model with the ability to interact with external tools, maintain memory, and interface with the environment. By equipping the LAM with these capabilities, it becomes capable of making meaningful impacts in the physical world.
5. Online Evaluation (Section 7): Finally, the performance of the LAM must be rigorously evaluated in the real environment from multiple perspectives, including accuracy, efficiency, and effectiveness in completing tasks. This step is crucial to ensure that the LAM functions intended and meets the desired operational standards.
Through these steps, LAMs can be effectively developed and deployed to bring LLMs' capabilities into real-world applications, enabling them to interact with and manipulate the physical environment, thereby making a tangible impact.
In the following sections, use the Windows GUI agent UFO (Zhagg et al. 2025a a case study to illustrate the process of building robust LAM from the ground up. This LAM will serve as the core inference engine for UFO, enabling it to autonomously fulfill user requests within the Windows OS environment. While this example focuses on a Windows GUT agent, the outlined steps can be adapted for developing LAMs in other scenarios or for different applications.
https://github. http:/ttwwb..oom//ccrrreeeeeeeee
4

---

Published in Transactions on Machine Learning Research (07 (025)
Task Action Data
Plan
Task:
Initial Task
Highlihtt Twc: "Hello World" in template doc
Task: How highlight text on word?
Task:
How Highiggtt test on word?
Solution:
Phase
Plan:
Select the tud want to highighh.. Click sighlight button.
"choose "Hello Worls", "cancrollabe":
Grounding
"selert_text wdit")
Phase
"argp": ""wet":
step"""cits "cilss the highlight button', "centrollabel: "33"""""" "sencroffeee"" TTxt Highlight Color
"Sunction" 1: Tokikk Inpun.. "ang": buutton" "lett, "double":
Word Environment
Figure 4: The two-phrase data collection and preparation process.
3 Data Collection and Preparation
Data a cornerstone in training LLMs, where high quality data significantly enhances their performance (Wang ct al. 2023c Li al. 2024). Similarly, LAMs require well-prepared, high-quality action-oriented data during the supervised fine-tuning phase. Off-the-shelf LLMs oflen face challenges when interacting with real-world environments. These difficulties typically arise from either lack of domain-specific knowledge the generation of hallucinated outputs that fail to be actionable. To mitigate these issues, adopt two-phase data collection approach:
task-plan collection and task-action collection, as shown in Figure Specifically:
1. Task-Plan Data Collection: this phase, we collect data consisting of tasks and their corresponding plans. Tasks are user requests expressed in natural language, while plans are detailed, step-by-step procedures designed to fulfill these requests. For example, task such as "How to change the fonl size in Word?" would have corresponding plan outlining the steps required to complete the task. This data is used to finc-tune the model to generate effective plans and improve its high-level reasoning and planning capabilities. However, task-plan data cannot be directly executed in the environment, requiring the following data conversion phase.
2. Task-Action Data Collcction: In this phase, the task-plan data is converted into task-action data, which includes tasks, plans, and the associated action sequences needed to execute those plans. Tasks and plans are refined become more concrete and grounded within specific environment. Action sequences are generated at this stage, such as select_ select_text text (text="hello") or click how="left", double=False), which represent actionable instructions capable of directly interacting with the environment. This enriched data provides the necessary granularity for training an LAM to perform reliable and accurate task exccutions in real-world scenarios.
The task-plan data aims at enhancing the model's high-level planning capabilities, allowing it to generate detailed, step-by-step plans based on user requests. Mcanwhile, the task-action data focuses on refining the model's ability execute these plans by converting each planned step into concrete, executable step sequence while considering environmental feedback. The data collection and preparation pipeline ensures that the model is capable of both high-level planning and low-level action exccution, thereby bridging the gap between natural language plans and executable actions.
In the following sections, we detail the methodologies employed for data collection, pre-processing, and integration of task-plan and task-action data. We illustrate how these steps enable the LLM to LAM transformation.
3.1 Task-Plas Data
outlines a multi-step pipeline for collecting and processing task-plan data, essential for training LAMs. The process begins with gathering data from diverse sources, including application documentation,
Figure
5

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Ouange pir modiio
Data Extraction
Data Preprocessing
Data parsing and selection
Remove non-Engtiah, too short/long and non-relevant semples Data duplication
Format unifying
Documents
wikiHow
Data Evolving Instruct Evolve to generate more complex tesk-pian
Data Construction construct plan data in JSON
How to quickty delete PPT fiio?
Historical Search Queries
All Task-Plan data
Figure 5: The pipeline to construct the task plan data.
WikiHow, and historical scarch queries. This is followed by structured pre-processing to ensure that the data is high-quality and relevant to specific tasks.
3.1.1 Data Sources
1. Application Documentation: Documentation and usage manuals for software applications provide authoritative task descriptions. These resources, maintained by product teams, are considered highly reliable. Relevant documentation, such as M365 documentation P crawled, with outdated or inaccessible pages being filtered out. The HTML content is converted into markdown format, and GPT-4o is used to extract task-plan pairs in the desired structured format.
2. WikiHow: WikiHow 3 hosts a wide range of how-to articles, including application-speccfic operational guides. Webpages related Windows platform applications are crawled, and GPT-4o extracts task and plan components, ensuring the resulting data aligns with the desired structured format.
3. Historical Search Queries: Search engine logs provide insight into real user demands, addressing gaps not covered by formal documentation. From Bing search logs, a 1% sample of queries mentioning application names (e.g., Word, Excel, PowerPoint) from the past year was taken.
3.1.2 Data Extraction and Pre- Pre-Processing
The first step processes raw data by filtering out non non-English, overly short/long or irrelevant samples (e.g., non-actionable smartphone content). Remaining data is then standardized into a unified format for downstream use.
3.1.3 Data Construction
To create structured JSON samples, GPT-4o is employed to extract and format tasks along with their associated plans. For historical search queries, synthetic data is generated to enrich the raw input, addressing the common issue of insufficient context. GPT-4o reformulates these queries into complete, sentence-like user requests, ensuring consistency across all data sources and facilitating effective downstream processing.
The resulting dataset contains structured JSON samples, with cach entry including unique task identifier (Lask id), the task description (lask), and step-by-step plan (plan). An example is shown below:
{ "task_id"", "task": "Add border to page in Word",
wrord_032""
"plan":
1. Go to Design > Page Borders.
https://tearn. http://iaaam...ccoooo..o http://aw...ta.t.oeeee/////ttaaa http:///teettttt/////eeee---r--a https:///ww. wikihow. com/Hain Page

---

Published in Transactions on Machine Learning Research (07 (07/2025)
2. Make selections for how you want the border to look.
3. To adjust the distance between the border and the edge of the page, select Options. Make your changes and select OK.
7
4. Select OK.
8
9
With the above process, initially collected total of 29,182 task-plan data samples.
3.1.4 Data Evolving
With the initial dataset processed, we employ data augmentation techniques enhance diversity and complexity. Inspired by WizardLM (Xu et al., (024)) and AgentGen (Hu et al.) 2025), we use GPT-4o evolve the raw lask generale new lask-plan pairs, improving the model's ability to follow instructions and handle more complex tasks.
The data evolving process generates new tasks from the original ones by introducing additional complexity, constraints, or steps while preserving relevance. The guidelines for task evolution are as follows:
The evolved task must be executable step-by-step on Windows OS or application.
The evolved task should include additional requirements, increasing its complexity without exceeding 20 extra words.
The evolved task must remain concise and related to the original task.
For cach evolved task, GPT-4o generates a corresponding plan adhoring to the following guidelines:
The plan must provide correct and actionable steps for Windows environments applications.
The plan should be concise and highlight critical action objects using bold emphasis.
This augmentation process results in a richer dataset where tasks become progressively more challenging, and plans incorporate domain-specifi knowledge. For example:
Raw task: Create drop-down list in Excel for Office 365.
Evolved Tosk: Create a dependent drop-down list in Excel for Office 365, where selecting an item from the first list filters options in the second list.
Evolved Plan:
Prepare your data by organizing it into two columns. The first column contains items for the primary drop-down list, and the second column contains items for the dependent list.
Name your ranges for the first and second lists.
Create the primary drop-down list using Data Validation.
Use the INDIRECT function to create the dependent drop-down list linked to the first selection.
Using data augmentation, we increased the original lask-plan dataset by 150%, generating a larger pool of samples. This augmentation significantly enhances the diversity and complexity of the dataset, allowing the model to learn from a broader range of sccnarios and develop robust planning capabilities. The augmented data introduces more challenging tasks and detailed plans, further enriching the training process and enabling the LAM to handle complex real-world applications effectively.

---

Published in Transactions on Machine Learning Research (07/ (07/2025)
3.2 Task-Actioon Data
The lask-plan data collected in the previous slage provides high-level, step-by-step plans for resolving user- requested tasks, serving as general guidelines. However, these plans are textual and not directly executable in a real-world environment. For instance, a task-plan data sample for the task "Highlight text in document" outlines the necessary steps but does not translate into actionable instructions for interacting with the application's GUT. This gap highlights the need for actionable task-action data to bridge the divide between planning and execution. To enable LAMs to produce actionable outputs, we generate task-action data derived from the previously collected lask-plan data. Task-action data caplures the granular interactions required to complete task in the application environment, including GUI navigation, button clicks, and responding to environmental feedback.
Traditional approaches for action data collection oflen involve manual or agent-based annotation for each lask, which both costly and labor-intensive To address these limitations, we propose an efficient, fully automated, and low-cost pipeline that leverages LLMs and real-world application interactions. This pipclinc consists of four stages, as depicted in Figure Instantiation, Execution, Evaluation, and Post-Processing. Specifically,
1. Instantiation: In this stage, the task-plan data is transformed into executable trajectory. Using LLM, each task is instantiated with specific operational objects, and related high-level plan is instantiated into a concrete sequence of actions that can be directly executed in the application environment.
2. Execution:
The instantiated trajectory is then executed within the real-world application environment. During this stage, the system interacts with the application's GUI carry out the specified actions. For example, the instantiated trajectory for highlighting text would involve selecting the appropriate text, navigating to the highlight tool, and applying the highlight. The result of this execution is the captured executed trajectory, including any feedback or environmental changes observed during the process.
3. Evaluation: Once the execution complete, the trajectory is evaluated for correctness using an LLM. The evaluation stage verifies whether the executed trajectory successfully accomplishes the intended task. This involves comparing the observed outcomes with the expected results outlined in the task-plan data. Tasks that fail to meet the criteria are flagged for review, while successful executions are retained for further processing.
4. Post-Procee Pro-esseoss:::: In the final stage, successful task-action trajectories undergo post-processing to ensure consistency, completeness, and readiness for training. This includes refining the data format, ensuring compatibility with the training pipeline, and annotating the data with relevant metadata (e.g., task IDs, exccution time, and step-by-ste foodback). The post-processod task-action data is then added to the training dataset, enabling the LAM to learn from real-world interactions.
The pipeline minimizes human intervention and reduces the number of LLM calls required, significantly improving scalability and efficioncy.
3.2.1 Instantiation
The task-plan data are primarily collected from help documents or public websites, creating a gap between the generalized task-plan data and the specific requirements nceded for exccution within particular environment. A common issue the lack of specificity. For instance, the task "highlight terl in document" " does not specify actionable objects, such as "which text" "which document". This lack of detail poses significant challenges in exccuting tasks within real-world applications.
To address this problem, we instantiate the lask-plan data to impute target objects and related functions. First, we prepare template Word files to serve as specific targets for the actions. These template files include various Word components such as paragraphs, tables, and figures. Each template file accompanied by a description indicating its content, providing context for grounding actions. Several sample template files can be found in Appendix A
8

---

Published in Transactions on Machine Learning Research (07/202) 2025)
PLAN
ACTION
1.
LLM
2.
3.
Instantiation
Task-Plan Data
Task-Action
2
Execution
Selected
3
Evaluation
Trajectory
4
Post- Processing
Training Data
HEUD Wordd
Figure 6: The pipeline of lask-action data conversion and collection.
Given a task-plan data sample, the task description is matched with the template file descriptions to select an appropriate template file as the target for actions. GPT-4 is then prompted to instantiate the task- plan with larget objects present the selected template file (detailed prompts can be found in Appendix B.1 Simultaneously, filter relevant functions from the available function pool using the task description, allowing the instantiation process to populate the task-action data with specific functions and their input parameters.
As a result of this process, the task description becomes more concrete and grounded in a specific environment, while the corresponding action sequences nceded to complete the task are generated. Figure 7 provides an example the instantiation process. Notably, the lask-action data not directly generated with GPT-4 due the risk of hallucinations. Instead, instantiating grounded task-plan data ensures the generation of more reliable and faithful step-by-step actions.
Task:
ful fix f(r)
Hightight
Solution:
text click the highlgght button
Word templates
Function Pool
Instantiation
Task:
Hightgghh Text "Hello World"
Highlightt Toxt "Helle World" template.doc
template.doc
Solution:
Solution:
sitep""choosee HHlllo World's Text"
Match with control item
"step" "choose text" HHello World",
"functinn:"" seleccttee text"
"unction": "select text", "argo ": ("toat:: "toxt to edit")
"orge"" ( toxt": "taxt to edi!")
"step"/folio" follck hightligtt buttonn,,
folick "ste"""ollkk
"sopp:
hightight button",
"controlLabel"
"controtLabel" '37"
"controlleat' Highlight"
"controlleat" Text Highlight Color
"function;" click input"
"unction": falck input?
"arge": ("button": "latt, "double": fnise)
"sege" ('button,, "double'' faise)
Figure 7: An example of task instantiation.

---

Published in Transactions on Machine Learning Research (07/ 2025)
3.2.2 Execution
To ensure that the steps in the instantiated task-plan data are accurate and truly actionable, the exccution stage verifies the action sequence by matching control items with the real application environment and performing the specified actions. This process validates the task-action data, ensuring its correctness and compatibility with the application GUI.
For instance, shown in Figure 7 the control item "Tezt Highlight Color" with its associated control label is retrieved using the action text "Highlight" from the control item pool. The corresponding task-action data is then executed in the application without further intervention from the LLM. During execution, if an error occurs (e.g., a mismatch between the predicted control item and the actual environment), the instantiated task discarded. Conversely, if all actions in the task execute successfully, the valion-validated task is forwarded to the evaluation stage described the following section. Additionally, screenshots of the application environment are captured after each step in the execution process, forming detailed trajectory to assist in subsequent evaluation.
It is important to note that the instantiated task-action data is not guaranteed to be valid. Since the data is generated through a single GPT-4 call based on task-plan data, it lacks the step-by-step refinement that might be necessary for certain tasks. In some cases, execution results from previous steps are required to instantiate subsequent steps accurately. In such scenarios, the ono-call instantiated task-action data may fail in validation and is removed from the dataset. This execution stage bridges the gap between planning and action, ensuring that task-action data is actionable, robust, and aligned with real-world application requirements.
3.2.3 Evaluation
Even if a task-action trajectory executes without error, additional validation is needed to ensure it fulfills the original task intent. Some plans may yield executable but incorrect behaviors. To evaluate correctness, we use: (1) the full action sequence, (2) before/after /after screenshots, and (3) environmental changes (e.g., comparing .ml state files in Word We prompt GPT--0 to assess whether the outcome matches the task description, assigning task-complete key ('yes", 'no", 'unsure'). Only tasks marked "yes" are retained for training,
ensuring data quality. Full prompl details are in Appendix B.2
3.2.4 Post-Processing
As noted in Section 3.2.2 a trajectory was recorded during the exccution process. This trajectory includes: (1) Screenshots captured at each step. (2) Environment states before and after each action. (3) Plans and corresponding actions for every step.
During the post-processing stage, these trajectories combined with the original task requests to generate synthetic step-wist wise training data. The resulting data format uses the task request as input and LAM's plan and actions as output. This structured format is critical for training LAMs to map task requests to actionable sequences effectively. The detailed template for the data format can be found in Appendix C
4
Model Training
Our objective is to develop T.AM from zero human-labeled data that can map user inputs to appropriate plans and executable actions, ultimately enabling complex task completion. To achieve this, adopt staged training strategy consisting of four phases, each building upon the previous one. As illustrated in Figure g these phases guide the model from learning structured task plans, to imitating expert demonstrations, to self-boosting from its own successes, and finally leveraging reward-based optimization. Throughout these stages, the model progressively evolves from LAM" to LAM
At a high level, Phase 1: Task Plan Pretraining provides a strong foundation by teaching the model to generate coherent, step-by-step plans for various tasks. Phase 2: Learning from Experts then introduces
4These files represent the underlying document, structure.
10

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Phase 1: Task-Plan Pretraining. (Section 4.1)
Resounce
tade-agennicc planing
for
Heppter, Euelued nete, will
g
Search Quentes
Spirsiid fine-tuning
formae
Oleeciie
Talk highight test
LAN
Shlect
Phase Learning from Experts. (Section 4.2)
Phase Learning from Reward Model. (Section
GP groundos dana
Trasing Merihod Superised turving
Bulding esservial
Euta Rescuroo
successes
In BOR
Word bnw
explocaaiionn failures
Frinnng Methos
Supervssd surking
Eipecioal
Data Formut
GPT-40 falures
Data Formiz
Stati:
Lea'n bede action aetion serueeees
LAM
Static
"Tas"" insert picture Ward
Traines on sacressttslure to predict action quality.
Roward Model
TFask:-rrw highight teet on word/
"Ooservario""
RM
*Sberratioo"..
("Sen"""''pp...
Actinsjl Steg"...
Boward:
Dara Resnurm
Boosting Exploration. (Section 4.3)
Al
Muthos Offine Bi
Near aptrrop rato
Foza Resource
Generalizaion deficat tatics
MAM
AM,
grounded in cases
frarerg Mirrod
Oijectinal
LAM
Ward Env
Sepervsed fie-tuning
Data Format
02 atee refinement and rewrrd-booe zellen.
Dejectven
Statn:
insrt video Word?
Data formar
Sucuu
Ciplrre and refine policy dhslleeggigg 195ks
LAM!
"Obseratien"
"tepp"
"Tasi" insert pktture Wondh "Shveratton"".
Actlon]["staa""
Figure 8: The overview of LAM training pipeline.
action trajectories labeled by GPT-4o, enabling LAMP to align its plan generation with actionable steps. However, relying solely on expert successes limits diversity and adaptability. To address this, Phase 3: Self- Boosting Exploration encourages the model to tackle tasks that even GPT-4o failed to solve, autonomously generating new success cases and evolving into LAM Finally, Phase Learning from a Reward Model incorporates reinforcement learning (RL) principles, allowing LAM* to learn from both successes and failures, refining its decision-making in complex, previously unseen scenarios. Table [ summarizes the data used in each phase. Each phase uses different training objectives, namely (i) task-plan pretraining (phase 1) and (ti)
decision-making training (phase 2-4), as detailed in Appendix E
Table Training data summary for each phase of LAM training.
Model
Data Type
Data Source
Input + Output
Data Size
Application documentation, WikiHoww evolved data historical search queries
LAM
Task-Plan Pairs
4 - P.
76,672 tasks
LAM
Task-Action Trajectories
GPT-4o
8t a1
2,192 trajectories
LAM
Task-Action Trajectories
LAM" + GPT-40
8t - 0c
2,688 trajectories
LAM4
Trajectories
RM + LAM2
(st, (t) +
1,788 trajectories
Reward Model
Task-Action Action-Reward Trajectories
GPT-4o + LAM
(st,0t)
4,476 trajectories
4.1 Phase 1: Task-Plas Pretraining
The initial stage focuses on imparting a broad understanding of how tasks can be decomposed into logical steps. We start with Mistral-7B (Jiang et al. 2023) as the base model. A total of 76,672 task-plan pairs (t,, P.) are collected from various sources, including application help documentation, WikiHow, and historical search queries. Of these, 29.182 pairs are sourced directly, while 47,490 are generated via data evolution techniques (as described in Section 3.1.4.. enriching the dataset with more complex and diverse tasks.
11

---

Published in Transactions on Machine Learning Research (07/2025)
In this phase, LAM' is trained via supervised fine-tuning (SFT) plan sequence P, for given task 4;:
(Degg et al. 2023 to predict the correct
N Σ
1 LSFT (TAAM)) = N Ccr( Pprrdd P!rue).
Here, Lce denotes the cross-entropy loss, and N is the number of tasks. Although no actions are generated at this stage, LAM gains a robust planning capability. This knowledge will prove critical in guiding the modol's action exccution in later phases, ensuring that the agent understands the logical structure tasks before attempting perform them.
4.2 Phase 2: Learning from Experts
While LAM' can produce structured plans, it lacks the ability to execute them. In Phase 2, we introduce expert-labeled task-action trajectories from GPT-4o (Section 3.2 to teach the model how to perform actions. The illustrative application this paper is the Microsoft Word environment, where we have 2,192 successful expert trajectories. Each trajectory consists of sequence of state-action pairs (8,,0), representing observed Cl states and the corresponding actions to progress the task.
We split these 2,192 trajectories into training set of 1,757 and test set of 435 trajectories, providing a total of 3,959 steps for training. By imitation learning LAM on these successful action sequences, we obtain LAM The objective is to minimize:
T. ΣV Σ (c((LA)(())))a))
ESFT (LAT)
N
where N is the number of trajectories and Tj is the number of steps in trajectory i. By imitating the expert's policy, LAM2 transforms from passive planner into model capable of exccuting actions aligned with its plans, grounding its reasoning in the real application environment.
4.3 Phase 3: Self- Baff-BBosting Exploration
Up to Phase 2, LAM only learns from successful trajectories provided by GPT-4o. This limits diversity and adaptability, as the model never sees how to handle situations that even GPT-4o could not deal with. To overcome this limitation, Phase 3 introduces self-boosting exploration.
Here, we revisit failed GPT-4o trajectories, i.e., tasks that GPT-4o did not complete successfully, and let LAM attempt them. Using the ReAct mechanism (Yao et al. 2023 Shinn et al. 2023). LAM interacts with the environment and tries alternative strategies for these challenging tasks. To construct meaningful task trajectories, the GUI agent uses the ReAct mechanism by interleaving observation, reasoning, and action. At each step, the agent perceives the current GUT state, reasons about the next move, executes the corresponding action, and records the updated environment, action taken, and reasoning trace. These step-by-step interactions then aggregated to form coherent trajectories for training L.M..
From these attempts, we sampled 2284 GPT-4o failed tasks and then collect 496 newly successful trajectories generated by LAM2 itself. These self-labeled successes, combined with the original 2,192 GPT-4o successes, form an augmented dataset.
We then fine-tune LAM on this enriched data, yielding LAM':
N T
(LA N (ss(LAM((s)).a)
Σ
This self-boosting step allows the model to Iearn from its own newly discovered solutions, overcoming previous limitations and improving adaptability. By leveraging planning knowledge from Phase and expert strategies from Phase 2. LAM' becomes more resourceful, even in scenarios with sparse absent expert guidance.
12

---

Published in Transactions on Machine Learning Research (07/2025)
4.4 Phase 4: Learning from Reward Model
Despite the improvements, Phases 1-3 focus on sucoesses or expert-like behavior. They offer limited insights into intermediate decision quality and fail to exploit learning opportunities presented by failed attempts. Phase 4, we integrate reinforcement learning (RL) to address these shortcomings (Kasneci et al. 2023).
To this end, we adopt two-stage approach: first, train reward model, and then we further fine-tune the LAM using this model. The reward model (RM) is built using LAM as the base model, with an additional output layer added to produce scalar values representing the quality of actions. Using the trained RM, we fine-tune LAM* in an offline RL setting. Here, the model refines its policy without additional environmental interactions, loveraging proviously collected trajectorics to learn from failures and improve action selection.
4.4.1 Reward Model Training
First, we train a reward model (RM) on both LAM successful (496) and failed (1788) trajectorics and GPT-4o's successful trajectories (2192) gathered in previous phases. All steps in successful trajectories are assigned reward of +1, and all steps in failed trajectories reward of -1. This uniform, binary labeling of outcomes ensures the RM consistently captures overall trajectory quality. Formally:
Vz = RM ac;
where è presents the RM parameters, and r. [+1 is the assigned reward. The RM trained via mean squared error (MSE) approximate these ground-truth rewards.
The training dataset for the RM includes both failed and successful lask-action trajectories generated by LAM", well as the successful trajectories from the collected task-action data. All steps in successful trajectories receive reward of +1, while every step in failed trajectories is assigned a reward of -1. This uniform labeling strategy ensures that the RM consistently reflects overall trajectory quality and effectively guides policy optimization.
4.4.2 Optimizing with Offline PPO
Armed with the RM to evaluate intermediate actions, we fine-tune LAM via offline PPO (Schulman et al. 2017). This stage focuses on the 1,788 failure trajectorics collected during Phase 3, providing unique opportunity to learn from mistakes. The training objective of PPO is:
N T
(arst) A,, clip (aMg(ar(sr) 1 c, + ) TAM, (aelst)
(pPo(LAM)) = N min
LAM, Bold (ar(st))
where A, denotes the advantage derived from RM-generated rewards, and is clipping parameter to ensure stable updates.
By incorporating signals from both successes and failures, LAM gains a deeper understanding of action quality. This RI.based fine-tuning helps the model generalize to complex, previously unseen scenarios, ensuring more robust and reliable decision-making. Our pipeline is fundamentally domain-agnostic and consists of: (1) Data Collection & Task-Plan Pretraining, (2) Imitation Learning from Expert Trajectories, (3) Self- Self-Boosting Exploration, and (4) Reward-Model Guided Refinement.
However, to deploy in a new domain, the environment must support the exccution, observation, and validation of action trajectories (e.g,, via APIs, simulators, or instrumentation). With these capabilit ies in place, the same four-phase structure can be applied: (1) Video Games: Discrete controls (e.g., button presses, joystick directions) can be collected through game APIs or scrcen-state instrumentation. (2) Robotics: The same stages apply, but the model must predict continuous motor commands or trajectorics (c.g., using a different decoder head). Simulation platforms (e.g., MuJoCo, Gazebo) facilitate low-risk and scalable data collection.
13

---

Published in Transactions on Machine Learning Research (07 (025)
Table 2: Offline performance comparison across different models and metrics on decision making.
Metric
LAM
LAM"
LAM
LAM
GPT-4o
GPTT-o Mini
DeepSeek Ri O1-mini
Object Ace (%)
39.4
85.6
87.4
87.8
73.2
68.4
63.4
71.8
Operation Ace (%)
59.9
97.3
97.7
97.7
94.2
89.6
87.8
88.5
Status Ace (%)
32.7
97.8
98.2
99.0
65.3
60.2
68.2
59.4
Step Success Rate (SSR) (%)
33.0
83.6
85.9
86.2
68.8
62.7
59.4
69.3
Task Success Rate (TSR) (%)
35.6
76.8
79.3
81.2
67.2
61.2
55.2
64.6
5 Offline Evaluations
The oflline evaluation results of Task-Plan Pretraining Results (Phase 1) and Task-Action Results (Phases 2-4) will be presented in this section. Offline evaluation allows to systematically assess the performance of LAM and subsequent phases (LAM2, LAM and LAM") without interacting with the environment. This setup effectively provides a controlled and reproducible framework for comparing task success rates, precision, and recall metries across models.
5.1 Experiment Setup
5.1.1 Data Preparation
We took several rigorous steps ensure that the evaluation set is both held-out and semantically distinct from the training set. (1) Redundancy Removal Before Splitting: Prior to partitioning, we applied semantic deduplication to eliminate redundant or highly similar tasks. This ensured that no rephrased paraphrased version of a training task appcars in the test set. (2) Random Sampling After Cleaning: The test set was then randomly sampled from the cleaned data. These test lasks were never seen during training or used for model selection. (3) Semantic Similarity Validation: To validate the distinctiveness of the test set, we followed (Wagg et al. 2023b by computing ROUGE-L similarity between training and test sets. We observed an average ROUGE-L score of 0.27, indicating that the test set semantically distinct and suitable for evaluating generalization. For details on our semantic deduplication process and dataset. splitting strategy, please refer to Appendix F
5.1.2 Evaluation Metrics
To evaluate agent performance in lask execution, we adopt five metrics: Object Accuracy, Operation Accuracy, Status Accuracy, Step Success Rate (SSR), and Task Success Rate (TSR). Object Accuracy measures whether the agent selects the correct UI element. Operation Accuracy checks if the predicted action (c.g., Click, Type) matches the ground truth. Status Accuracy evaluates whether the agent correctly determines task completion us. Step Success Rate (SSR) considers a step successful only if object, operation, and status are all correct. Task Success Rate (TSR) requires all steps in a task to be correct for the task to count as successful.
While task success rate (TSR) is our primary metric, we include step success rate (SSR) as a complementary diagnostic tool. In GUI-based environments like Word, most tasks follow highly constrained and canonical trajectories, making step-level precision both meaningful and necessary. Although multiple trajectories may theoretically complete task, in practice, deviations often lead to errors suboptimal behavior. Thus, LAM's higher SSR reflects not just closer alignment with reference trajectorics but greater consistency, planning quality, and robustness. Also, SSR aligus with standard agent benchmarks (e.g.. Mind2Web (Deng el al., 2023), ArTW (Rawles et al. 2023)). LAM² builds on LAM', which learns structured task decomposition from curated plans-knonn knowledge not explicitly present GPT-4o. Combined with domain-specific fine-tuning, LAM? integrates planning priors and labeled actions to achieve more grounded execution. Unlike GPT-4o's zero-shot approach, LAM2 is adapted to the task interface, reducing hallucinations and improving consistency.
14

---

Published in Transactions on Machine Learning Research (07/ (025))
5.1.3 Performance on Decision Making
Table 2 presents results on 435 Word lasks, showing that our four-phase LAM framework yields cumulative performance gains. LAM* achieves a TSR of 81 81.2%, outperforming GPT-40 (67. (67.2%), GPT-44-mini (61.2%), DeepSeek RI (55.2%), and ol-mini (64.6%). These improvements arise from a stepwise training strategy: task-plan pretraining (LAMM') GPT-to-labeled imitation learning (LAM²), self- coossioss on GPT-4o failure cases (LAM²), and reward-guided fine-tuning (LAM**.
Despite using GPTT-o-llbbled data in early phases, LAM surpasses GPT-40 by learning from its limitations and refining its decision making through domain-specifie supervision. The ReAct mechanism supports this by enabling L.AM to generate new successful trajectories for harder tasks. Across key metrics including step success rate (86.2%), object accuracy (87.8%), and status accuracy (99.0%), LAM consistently outperforms all baselines. These results highlight the effectiveness of our targeted pipeline in producing robust and reliable action models for real- wolld applications. All updated comparisons have been incorporated into the main text.
As for costs, GPT-40 was used for approximately 50K API calls during data instantiation and validation, incurring a total cost of 2.5K (at 0.05 per 1K tokens), compared to an estimated > 20K for equivalent manual annotation. Inference costs are minimal (< 0.01 per task).
5.2 Failure Cases and Error Patterns in LAM
While LAM achieves high performance across most tasks, we identify several recurring error patterns that reveal areas for further refinement.1. Early Termination Before Task Completion LAM may halt execution prematurcly, mistaking intermediate steps for task completion. Ezample: In a task requiring word count calculation, LAM correclly opens the "Review" lab bul fails to click on "Word Count," " slopping short of the final step. 2. Skipping Required Preconditions LAM may attempt to execute goal-directed actions without. verifying or setting required environmental conditions. Example: When instructed to calculate word count for document in A4 format with font size 11, LAM skips the font and layout verification steps and proceeds directly to "Word Count." " 3. Incorrect or Ambignous UT Flement Selection LAM sometimes selects visually similar but incorrect controls due to limited U1 disambiguation. Example: In an Excel autofill task, LAM selects the "Ediling" tub instead of directly interucting with the larget cells.
These failure cases reveal three dominant error patterns in LAM's current behavior: (1) premature task termination due to overconfident stopping, (2) insufficicnt attention to task preconditions, and (3) UI disam- biguation failures when interacting with similar elements. Addressing these issues will require improvements in planning, state verification, and more robust grounding to UI context. These kinds of problems we will consider in the future work.
6
Integration and Grounding
6.1 LAM Agent In a Nutshell
In UFO, the LAM serves as the inference engine within the AppAgent, enabling efficient and accurate task completion. Figure illustrates the architecture of the AppAgent. UFO, equipped with LAMs, is designed for interactive engagement with Windows applications. For simplicity, we focus on automating tasks within Microsoft Word, a widely used productivity tool with a sophisticatod GUI and diverse functionalitics, making it an ideal testbed for training and evaluating LAM.
During each inference step, the agent collects critical contextual information from the application environment, which is then passed to the LAM for decision-making, The LAM performs planning, orchestrates actions, and infers the necessary sleps to fulfill the user request. These inferred actions are grounded in the environment. by mapping them to predefined tools and function calls used by the agent, such as mouse clicks, keyboard inputs, or API calls. This process itcrates, with LAM continuously adjusting its plan based on real-time foedback from the environment, until the task is completed. Additionally, the agent maintains a memory that logs historical actions and plans, providing essential context for the LAM to make more informed and
15

---

Published in Transactions on Machine Learning Research (07 2025)
AppAgent
Memory
action
part-
Environment
Request
Sequence
Grounding
Executor
Action
Feedback
Collection
M"ypac" (Botton, "Uie" psoitees" 10.45, 0.88)
["lgac" "Oucament" "pariiss" 13,5, 64s1]
["teac" (Botion, "hoign" "positio""
["tgee" (Comecto,, Information
Screenshots
Figure 9: The overall architecture of the AppAgent employed in UFO.
adaptive decisions the task progresses. This integration ensures that UFO can efficiently manage and complete complex, real-world tasks in Windows environments.
6.2 Environment
The UFO agent leverages the LAM to interact with applications in the Windows enviromment. At each decision step, UFO employs the UI Automation (UIA) API (Dinh al., 2018) inspect all actionable controls within the target Windows application, retrieving contextual information for each contro This information is passed to the LAM for control selection and action inference. The control data is structured list of dictionaries, where each control is assigned numerical index (as label), along with its title and control type, allowing the LAM to make informed decisions regarding control selection and the corresponding action. This input format mirrors the structure used during offline data collection for consistency in training and execution.
6.3 LAM Inference
Using the environmental observations of application control information, UFO constructs prompls in the same format as the offline training data, using planning and thought generation techniques (Wei et al. 2022b Ding et al. 2024) to enable LAM to make reliable inferences about the appropriate controls and operations to invoke. These inferences target the controls detected by the UIA, where each control is selected from a predefined list. The function calls inferred by LAAM are limited to pre-defined operations, such as mouse and keyboard actions, as well APIs specific Word-related tasks. Once inferred, these operations are parsed and executed within the environment.
6.4 Action Execution
UFO employs a control interactor to ground the action strings generated by LAMs, translating them into tangible impacts within the target application. Each action typically consists of two key components: Control Element: This refers to the specific UI control within the application that will receive the action, such as a button, text box, or scroll bar. Function Call: This represents the operation to be performed on the control element, such as mouse click, keyboard input, invocation of native APIs. By combining the control element and its associated function call, UFO executes the inferred actions within the application.
SUIA is the native Windows OS APIs used dctect actionable controls and provide their metadata, such and locations. For other platforms, UTA be replaced by vision based detectors that analyse screenshots by utilizing alternative accessibility APIs.
16

---

Published in Transactions on Machine Learning Research (07 (075)
6.5 Memory
UFO maintains additional information in its memory assist LAMs in making more informed and accurate decisions. This memory includes: (1) Historical Actions: A log of action trajectories and their exccution results from the initial step onwards. This helps LAM understand the current system state and aids in exploring the next steps based on prior actions. (2) Previous Plan: The textual planning for future actions, generated by LAM in the previous step. This serves as a reference for guiding the currnt and future actions, ensuring consistency across steps.
7
Online Evaluations
With the integration of the Windows GUI agent UFO, evaluate the performance of the LAM in real-world environments. The evaluation process and results are detailed in the following subsections.
7.1 Testing Dataset
The online performance of LAM is evaluated on the same set of 435 test requests used during LAM training. The testing environments, specifically the Word document templates corresponding to each task, also maintained as identical to the training setup to ensure consistency and comparability.
Table 3:
Performance comparison of LAM and baseline models across metrics.
Text-only
Text Visual
Metric
LAM
GPT-1o + UFO GPT-4o Mini + UFO
GPT-1o + UFO GPT-1o Mini
UFO
Task Success Rate (%)
71.0
63.0
57.8
75.5
66.7
Task Completion Time (8)
30.42
86.42
35.24
96.48
46.21
Task Completion Steps
5.62
6,73
5.99
4.98
6.34
Average Step Latency (6)
5.11
12.84
5.88
19.36
7.29
7.2 Implementation
Our LAM was deployed on a virtual machine (VM) configured as NC24s v3. The VM is equipped with 24 virtual cores (vCPUs), 448 GB of memory, and two NVIDIA Tesla V100 GPUs, each with 16 GB of memory, to support efficient inference. This computational setup designed to meet the demanding requirements of LAM's inference processos effectively.
The UFO agent operates on six VMs rumning in parallel using Azure Dedicated Hos to accelerate the testing process. Each VM is equipped with a 15-core Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz, 64GB of RAM, and runs Windows 11 Enterprise version 23H2. Microsoft applications, such as Word and Excel, are installed on version 2410. GUI control is facilitated through the MSTSC too 7 This setup ensures a consistent and controlled environment for evaluating the LAM's performance.
T
7.3 Baselines
To benchmark the performance of LAM, we compared it against two baseline models: GPT-4o and GPT- 4o Mini. These models are widely recognized for their robust natural language processing and reasoning capabilities, making them popular choices in the development of GUI agents. To ensure consistency in evaluation, the top_p and temperature hyperparameters were set 0 for both baseline models.
To further examine the impact of input modalities, conducted an ablation study comparing performance with and without the inclusion of screenshots. Notably, LAM processes only textual inputs, excluding screenshots, while the baseline models were evaluated using both textual and visual modalities.
17

---

Published in Transactions on Machine Learning Research (07/2025)
7.4 Evaluation Metrics
We employ the following metrics to comprehensively evaluate the performance of LAM: Task Success Rate (TSR): The percentage of tasks successfully completed out of the total tasks attempted. Task success is determined by an evaluation agent using GPT-4o, which assesses the full task completion trajectory, including plans, action sequences, and screenshots, to verify task completion. Task Completion Time: The total time taken to complete cach task, mcasured from the initial request the final acti Completion Steps: The total number of action steps performed by the agent to successfully complete each task. Average Step Latency: The average time taken per action step, reflecting the model's efficiency in generating and executing each action.
To ensure the reliability of our LLM-based automatic evaluation, we conducted a small-scale human study focused on validating its accuracy and consistency. Specifically, we randomly sampled 100 task completions from the test set, spanning a diverse range of task types and difficulty levels. Each sample was independently reviewed by three human annotators with domain familiarity, who were asked to judge task success following the same criteria used in our automated evaluation. The results demonstrated 91% agreement between the majority human judgment and the LLM-based labels. Furthermore, the inter-annotator agroement, mcasured by Cohen's kappa, was 0.85, indicating near-perfect consistency among human raters and strengthening the case that the automated evaluation aligns well with human judgment. We ensured cross-validation among annotators, and disagreements were resolved through consensus discussions. These findings confirm that our LLM-based evaluation is a valid and scalable proxy for human judgment in this task setting. Quantitative results, annotator agreement, and evaluation setup for the human study are provided A Appendix G
7.5 Experimental Analysis
The experimental results are presented Table 3 performance compared to the GPT-4o models. While GPT-4o with visual inputs attains the highest TSR of 76.5%, slightly outperforming LAM, its reliance on visual data introduces significant trade-offs in efficiency. Notably, when visual inputs excluded, GPT-4o's TSR drops to 63.0%, an 8.0 percentage point decrease compared to LAM. Similarly, GPT-4o Mini exhibits lower TSRs for both visual and non-visual settings (66.7% and 57.8%, respectively). These results underscore LAM's capability as text-only model to maintain high task success rates, outperforming the text-only variants of the baseline models.
LAM achieves a TSR of 71.0%, demonstrating competitive
Efficiency assessed through Task Completion Time and Average Step Latency, where LAM demonstrates clear superiority. LAM achieves the shortest. Task Completion Time of 30.42 seconds, substantially outperforming all baseline models. In comparison, GPT-4o without visual inputs records completion time of 86.42 seconds, more than 2.84 times longer than LAM. GPT-4o with visual inputs fares even worse, with completion time of 96.48 seconds. Although GPT-4o Mini models show slightly better efficiency than their larger counterparts, they remain less efficient than LAM, with completion times of 35.24 seconds (without visual inputs) and 46.21 seconds (with visual inputs).
LAM also excels in Average Step Latency, achieving the shortest time per action step at 5.41 seconds. Without visual inputs, GPT-4o roduces its step latency to 12.84 scconds but still remains more than twice as slow as LAM. In comparison, GPT-4o with visual inputs exhibits the highest step latency at 19.36 seconds per step, more than triple LAM's latency. GPT-4o Mini models show moderate improvements but still fall short, with step latencies of 7.29 seconds (with visual inputs) and 5.88 seconds (without visual inputs).
These findings highlight LAM's strengths as a text-only model, offering a compelling balance of competitive accuracy and superior efficiency. It achieves rapid task completion and low latency without sacrificing performance, making an effective solution for real-world applications. Its specializod training enables precise action inference and execution, underscoring the potential of LAMs to enhance automation and productivity in agent-based systems.
18

---

Published in Transactions on Machine Learning Research (07/2025)
8 Limitations and Future Directions
Despite their promise, Large Action Models (LAMs) remain in an early slage, facing several key challenges that limit their deployment in real-world applications. We highlight three core areas requiring further research.
Safety Risks LAMs operate in both digital and physical environments, which introduces safety concerns not present in traditional LLMs. Erroncous actions-whether whether controlling software or physical systems result in real-world harm (Liu et al. 2024 Zhou et al. 2024). This necessitates robust safety mechanisms, such as action validation, formal verification, and rollback strategies (Zhang et al. 2023 Koo & Toueg 1986).
Future work should focus on fail-safe architectures that vet actions before execution.
Ethical and Regulatory Concerns As LAMs gain autonomy, questions of accountability, transparency, and bias become critical (Biswas & Talukdar 2023 Li et al. 2023 Ferrara 2024). Misinterpretation of user intent inherited biases from training data result in unfair or unsafe behavior. Regulatory compliance in sensitive domains (e.g., healthcare, finance) further complicates deployment (Karabacak & Margetis 2023). Future directions include developing interpretable decision frameworks and establishing ethical guidelines and compliance standards for safe LAM deployment.
Scalability and Generalization Current LAMs are highly specialized, often limited narrow environ- ments. Changes in application interfaces system updates break performance (Grosse et al. 2023 Zhang et al. 2024 Moreover, domain-specific data collection is costly and labor-intensive (Muennighofi ct al. 2023). Future rescarch should pursuc transfer learning, few-shot adaptation, and automated data collection to enhance generalizability and reduce dependence on manual annotation.
8.1 Mitigation Strategies for Safe Deployment
To ensure the safe and ethical deployment of Large Action Models (LAMs), adopt the following mitigation strategies: (1) Environment Sandboxing: Run LAMs in sandboxed environments to restrict access to critical system functions, minimizing the risk of unintended actions. (2) Human-in-the-Loop Verification:
Require human approval for high-impact or irreversible actions to ensure oversight in sensitive contexts. (3) Action Validation: Validate predicted actions real-time against environmental constraints or through simulation to prevent execution errors. (4) Fail-Safe Defaults and Timeouts: Use safe defaults (e-g., cancel or no-op) and timeout mechanisms to handle ambiguous or stalled executions. (5) Progressive Deployment: Start with low-risk or shadow deployments and gradually scale to full autonomy based on observed reliability.
These strategies follow best practices for trustworthy AI and are essential for deploying LAMs safely in dynamic, real-world environments.
9
Related work
9.1 Data of LAMs
Mind2Web (Deng et al. 2023) is the first dataset developed for web agents that follow natural language instructions complete complex tasks across diverse websites. It includes task descriptions, action sequences, and webpage snapshots, offering rich data for training and testing models in various web-based scenarios. Rawles et al., introduced large dataset called Android in the Wild (AITW) (Rawles et al.
2023 which is
designed specifically for training models to control Android devices. SecClick (Cheng et al.
2024 combines
web, mobile, and general GUI (asks, creating dataset of over million samples for training LAMss Similarly, GUICourse (Chen et al. 2024b) and OmniACT (Kapoor et al.) 2024) provide datasets across web, smartphone, and desktop platforms, containing detailed user requests, environmental states, and action sequences. These datasets are invaluable resources for training LAMs in specific domains and evaluating their task execution abilities.
19

---

Published in Transactions on Machine Learning Research (07/2025)
Several benchmarks have also been developed to evaluate the capabilities of LAMs and their associated agents in different environments. WebCanvas provides 542 tasks with dynamic environments, designed to assess the task completion ability of web agents. AndroidWorld (Rawles et al. 2025) offers a fully functional Android environment, featuring 116 programmatic tasks across 20 real-world Android apps with reward signals for performance evaluation. WindowsArena (Bonatti et al., 2024) focuses on benchmarking LAMs within the Windows GUT, while OSWorld (Xie et al., (024) extends this to a more diverse environment, encompassing Windows, macOS, and Ubuntu. These benchmarks provide standardized settings to measure and compare the effectiveness of LAMs and their agents in various real-world environments, enabling unified evaluation framework for agentic models.
9.2 Training LAMs
Using both open and private domain-specific datascts, significant research efforts have been directed toward training LAMs for specialized purposes, enhancing the action inference abilities of traditional LLMs to enable automation and tangible real-world impact. For example, SeeClick (Cheng et al. 2024) and GUICourse (Chen et al. 2024b), in addition to releasing their own datasets, leverage these resources to train LAMs, grounding real-world data into models that effectively interact with their environments.
Hong et al, trained an which specializes in GUI understanding and navigation tasks across both PC and Android interfaces. By utilizing datasets like Mind2Web and AITW, CogAgent has been optimized for complex navigation and action execution tasks in diverse GUT environments. ScreenAT (Baechler et al. 2024) introduced a textual representation for user interfaces (Uls) to teach models how to understand and interact with Uls. This approach also facilitates automatic generation of large-scale training data, which then used to pretrain and tne-tune models for a wide spectrum of tasks, including UI and infographic understanding and navigation. Additionally, Zhang et ol., released series of large action models (xL.AM) tailored for AT agent tasks Zhhng et al. 2025c), including five models with both dense and mixture-of-expert architectures. By unifying dalasets from diverse environments, xLAM ensures consistency in data format, simplifying model training and enhancing generalization across multiple benchmarks. These models have achieved outstanding performance in diverse scenarios, demonstrating the capability of LAMs to extend beyond traditional LLMs and perform complex real-world (asks.
parllmeer visual language LAM, named CogAgent (Hong al. 2024
9.3 Agents with LAMs
With the development of LAMs, researchers have integrated these models into real-world agent systems, which provide the necessary components and workflows to ensure effective interaction between LAMs and their environments, enabling them to fulfill user requests efficiently. As a pioncer, Zhang et al., dcmonstrated that GPT-V can serve capable LAM for web navigation when coupled with appropriate agent techniques and tools, revealing the patential of LAMSs in complex web interactions. In the mobile domain, MobileAgent (Wang) et al. 2024a) and AppAgent (Zhang et al.] (025bb focus on automating tasks within Android applications by leveraging GUI agents. These systems demonstrate how LAMs can power task automation on mobile platforms, transforming how users interact with applications.
One of the most advanced systems, UFO (Zhang et al. 2025a), is a Ul-focused agent designed for automating tasks on the Windows OS, further enhanced with APIs (Lu et al.) 2024). UFO is composed of two key components: HostAgent that decomposes user requests into subtasks and AppAgent that executes these subtasks within individual applications. This architecture significantly enhances UFO's capability to handle cross-application lasks seamlessly, providing robust (ask automation across diverse software environments. In parallel, ScreenAgent (Niu et al., 2024), Cradle (Tan et al.) 2024), OS-Copilot (Wu et al.) 2024), and MMAC-Copilot (Song et al. 2024) also focus on automating UI tasks in desktop environments. Notably, Cradle and OS-Copilot push the boundaries by enabling agents to learn from their experiences and self-evolve over time, further enhancing their effectiveness and autonomy.
20

---

Published in Transactions on Machine Learning Research (07 (025))
10 Conclusion
"Actions speak louder than words." The transition from generating language responses to executing tangible actions marks the evolution of large language models into large action models, enabling them to make real-world impacts, a critical step towards achieving AGI. This technical report provides a comprehensive introduction to LAMs, covering their conceptual foundations, system architecture, and the step-by-step process of developing a LAM from data collection to model training and deployment systems. We use the Windows OS environment and its GUI agent UFO, as a case study to demonstrate how to build LAM from the ground up. Detailed implementation strategics and evaluation results are presented to offer practical insights into this process.
real-workd agent
However, despite progress, the development of high-quality LAMs is still in its early stages, with several limitations remaining. These include the extensive need for training data and computational resources, inference latency, and the risk of errors during real-world execution. While current LAMs have shown potential, there is substantial room for improvement. We anticipate that as these challenges are addressed, more sophisticated and reliable LAM applications will emerge, bringing us closer to fully autonomous systems capable of meaningful action in complex environments.
References
Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbune, Jason Lin, Jindong Chen, and Abhanshu Sharma, Screenai: vision-language model for U1 and infographics undorstanding. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAT), pp. 3058 3066,, 2024.
Anjanava Biswas and Wrick Talukdar. Guardrails for trust, safely, and ethical development and deployment of large language models (Ilm). Journal of Science & Technology, 4(6):55-88 2023.
Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal OS agents at scale. arXiv preprint arXiv:2409.0820 2024.
Tom B Brown. Language models few-shot learners. arXiv preprint or Xiv::20005... 14165, 2020.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 3676-3713, 2023.
Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuau Lei, Xiaolong Chen, Xingmei Wang, Kai Zheng, Defu Lian, and Enhong Chen. When large language models meet personalization: perspectives of challenges and opportunities. World Wide Web (WWW), 27(4):42, 2024a.
Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjic Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, Yuan Yao, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Guicourse: From general vision language models to versatile GUT agents. Xix preprint arXiv:2406. 13377 2024b.
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the Arnual Meeting of the Association for Computational Linguistics (ACL), pp. 9313-9332, 2024.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 28091-281 14, 2023.
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang, Everything of thoughts: Defying the law of penrose triangle for thought generation. In Proceedings of the Association for Computational Lánguisties (ACL.), pp. 1662, 2024.
21

---

Published in Transactions on Machine Learning Research (07/2025)
Duong Dinh, Pham Ngoc Hung, and Tung Nguyen Duy. A method for automated user interface testing of windows-based applications. In Proceedings of the International Symposium on Information and Communication Technology (SoICT), pp. 337 343, 2018.
Tao Feng, Chuanyang Jin, Jingyu Liu, Kunlun Zhu, Haoqin Tu, Zirui Cheng, Guanyu Lin, and Jiaxuan You. How far are we from AGI: are Iims all we need? Trans. Mach. Learn. Res., 2024, 2024.
Emilio Ferrara. Genai against humanity: nefarious applications of generative artificial intelligence and large language models. J. Comput. Soc. Sci., 7(1):549-50) 569, 2024.
Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian lchter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In Proceedings of the IEEE International Conference on Roboties and Automation (ICRA), pp. 12462 -22669, 2024.
Roger B. Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li. Esin Durmus, Ethan Perez, Evan Hubinger, Kamile Lukosiute, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions. arXiv preprint arx:;22308, 03296, 2023.
Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati. Leveraging pre-trained large language models to construct and utilize world models for model- task planning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 79081-79090 2023.
Jianliang He, Siyu Chen, Fengzhuo Zhang, and Zhuoran Yang. From words to actions: Unveiling the theoretical underpinnings of llm-driven autonomous systems. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: visual language model for GUT agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 14281-14290, 2024.
Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jian- Bian-Guang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. In Proceedings of the ACM SIGKDD Conference Knowledge Discovery and Data Möning (KDD), pp. 496 507, 196-507,
2025.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arv preprint arXiv:2310.000 06825, 2023.
Yuxuan Jiang, Chaoyun Zhang, Shilin He, Zhihao Yang, Minghua Ma, Si Qin, Yu Kang, Yingnong Dang, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Xpert: Empowering incident management with query recommendations via large language models. In Proceedings of the International Conference on Software Engineering (ICSE), pp. 92:1 92:13, 2024.
Sai Shashank Kalakonda, Shubh Maheshwari, and Ravi Kiran Sarvadevabhatla. Action-gpt: Leveraging large-scale language models for improved and generalized action generation. Tn Proceedings of the IEFE International Conference on Multimedia and Expo (ICME), pp. 31-36, 2023.
Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AIShikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist. autonomous agents for desktop and web, In Proceedings of the European Conference on Computer Vision (ECCV), pp. 161-178, 2024.
Mert Karabacak and Konstantinos Margetis. Embracing large language models for medical applications: Opportunities and challenges. Cureus, 15(5)=e39355, 2023.
22

---

Published in Transactions on Machine Learning Research (07/2025)
Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274, 2023.
Richard Koo and Sam Toueg. Checkpointing and rollback-recovery for distributed systems. In Proceedings of the Fall Joinl Compater Conference (FJCC), pp. 1150- 1158, 1986.
Wei Li, William W. Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv preprint 2xxx:240660300 2024.
Yingji Li, Mengnan Du, Rui Song, Xin Wang, and Ying Wang. A survey on fairness in large language models. arXiv preprint (xXi:2908.101433 2023.
Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejje Cui, Xuchao Zhang, et al. Domain specialization as the key to make large language models disruptive: A comprehensive survey. arXiv preprint WxXv:2305.18700 2023.
Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. In Proceedings of the Association for Computational Linguistics (ACL), pp. 1817 1829, 2024.
Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Turn every application into agent: Towards efficient human-agent- computer interaction with api-first llm-based agents. arXiv preprinl arx:v22409. 17140, 2024.
Zilin Ma, Yiyang Mei, and Zhaoyuan Su. Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support. arXio preprint arXiw:2307.18100, 2023.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint arXiv:2002000 06196, 2024.
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A. Raffel. Scaling data-constrained language models. In Proceedings of the Advances in Newral Information Processing Systems (NeurIPS), pp. 50358 50376, 2023.
Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xucyuan Leng, He Kong, Yi Chang, and Qi Wang. Screenagent: A vision language model-driven computer control agent. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAT), pp. 1883-641, 2024.
Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy P. Lillicrap. Androidinthewild: A large-scale dataset for android device control. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 59708-59728, 2023.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E. Bishop, Wei Li, Folawiyo Hampbell-Ajala Daniel Kenji Toyama, Robert James Berry, Divya Tyamagundlu, Timothy P. Lillicrap, and Oriana Riva. Androidworld: dynamic benchmarking environment for autonomous agents. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
Nils Reimers and Iryna Gurevych. sontereeerrasssommr/s//limiiiiii hattiee-rraatttotrrrrllll--aaaaa
https://huggingface.u co/
2021. Accessed: 2023-06-06.
Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg, Tara N. Sainath, Johan Schalkwyk, Matthew Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, Alexandru Tudor, Mihajlo Velimirovic, Damien Vincent, Jiahui Yu, Yonggiang Wang, Vicky Zayats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, and
23

---

Published in Transactions on Machine Learning Research (07/2025)
Christian Havno Frank. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiu:2306. 1 12925, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. aixiv preprint arXiv: 170.003377, 2017.
Chirag Shah, Ryen W. White, Reid Andersen, Georg Buscher, Scott Counts, Sarkar Snigdha Sarathi Das, Ali Montazer, Sathish Manivannan, Jennifer Neville, Xiaochuan Ni, Nagu Rangan, Tara Safavi, Siddharth Suri, Mengting Wan, Leijie Wang, and Longqi Yang. Using large language models to generate, validate, and apply user intent taxonomies. arXiv preprint w:Xi:2339.13006 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 8634-8652, 2023.
Zirui Song, Yaohang Li, Meng Fang, Zhenhao Chen, Zecheng Shi, Yuan Huang, and Ling Chen. Mmac-copilot: Multi-modal agent collaboration operating system copilot. arXiv preprinl arXiv:2404 18074, 2024.
Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Jumpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, Yifei Bi, Pengjie Gu, Xinrun Wang, Börje F. Karlsson, Bo An, and Zongqing Lu. Towards general computer control: A multimodal agent for red dead redemption II as a case study. arXiv preprint arXiv:2403.0911 2024.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Danicl Shu Wei Ting. Large language models in modicine. Nature Medicine, 1900):193-- 2023.
Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. Large language models can accurately predict searcher preferences. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retricval (SIGIR), pp. 1930-1940, 2024.
Karthik Valmeckam, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (A benchmark for Ilms on planning and reasoning about change). arxiv preprint wxXie:2206.10490 2022.
Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXio preprinl arXiv:2401.16155 2024a.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. survey on large language model based autonomous agents. Frontiers Comput. Sci., 18(6):186345, 2024b.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 61501-61513, 2023a.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the Association for Computational Linguistics (ACL), pp. 13484-13508, 2023b.
Zige Wang, Wanjun Zhong, Yufei Wang. Qi Zhu, Fei Mi, Baojun Wang, Lifeng Shang, Xin Jiang, and Qun Liu. Data management for large language models: survey. arXiv preprint arxiv:232.00 011-2012.01700 700, 2023c.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In Proceedings of the International Conference on Learning Representations (ICI.R) 2022a.
24

---

Published in Transactions on Machine Learning Research (07 2225))
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thoughi prompting elicits reasoning in large language models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), pp. 24824 24837, 2022b.
Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhemmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement, arXiv preprint arXiu:2402.074556 2024.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujum Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In Proceedings of the Advances in Newral Information Processing Systems (NeurIPS), pp. 20404, 2024.
Can Xu, Qingleng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung- Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v(ision). arXiv preprint ar Xrxi::2009. 1 17421, 2023.
Shunyu Yao, Rohan Rao, Matthew J. Hausknecht, and Karthik Narasimhan. Keep CALM and explore: Language models for action generation in text-based games. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8736 8754, 2020.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Burak Yetistiren, Isik Özsoy, Miray Ayerdem, and Fray Tüzūn. Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt. arxiv preprint ar Ktt::230411077, 2023.
Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S. Yu. Large language models for robotics: A survey. arXiv preprint arXiv:231..07226, 2023.
Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Ufo: A ui-focused agent for windows os interaction. In Proceedings of the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pp. 597 622, 2025a.
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI), pp. 70:1-70:20, 2025b.
Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Manoj Awalgaonkar, Rithesh R. N., Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio Savarese, and Caiming Xiong. xlam: A family of large action models to empower Al agent systems. In Proceedings of the Conference of the Association for Compulational Linguisties: Human Language Technologies (NAACL), pp- 11583 11597, 2025c.
Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, and Peng Cui. On the nat-of-distribution generalization of multimodal large language models, anXiv preprint Xxt::2402.06599 2024.
Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlic Huang. Safetybench: Evaluating the safety of large language models with multiple choice questions. Xio preprint or arXim:2309.0000 07045, 2023.
25

---

Published in Transactions on Machine Learning Research (07 (07/)025)
Xin Zhou, Yi Lu, Ruotian Ma, Yujian Wei, Tao Gui, Qi Zhang, and Xuanjing Huang. Making harmful behaviors unlearnable for large language models. In Proceedings of the Associalion for Compulational Linguistics (ACL), pp. 10258 10273, 2024.
26

---

Published in Transactions on Machine Learning Research (07 (07/2025)
A Template Word files
Figure 10
11
and 12 show three template word file examples used in the instantiation phase when converting
task-plan data task-action data.
File
Home
Insert
Drow
Design
Loyour
References
Mailings
Review
View
Help
Comments
Edriing
Shate
R
Cambria Body
11
"X
A
Paste
Aa
Ap
21
Norral
AaBbCeb
Spac.
AaBbCc
Heading
Dicato
Sera
Figure 10: word template file with the description "A doc with a rectangle shape."
Referanoer
Mallingr
View
Commenta
Viewing
Share
AaBbC
Bdliig
Nomell
Headtng
Aragach
123
HGH
helo
Figure 11: A word template file with the description "A doc with comments and reviewer."
27

---

Published in Transactions on Machine Learning Research (07 (075)
Home
Insert
Draw
Design
Layout
Refeeences
Maalings
Review
Help
Edittin
Share
Cambria Bidky)
Estting
Sensibirty
Add-is
Exttor
hvi
Voice
Senaibiity
Addira
Chart Title
Canggey
Congory
Cangraaa
Crngyy
Seree Sirres simes
Cwods TD
Figure 12: A word template file with the description "A doc with chart."
B
Prompts
B.1 Instantiation
The instantiation prompt used in the instantiation phase when converting task-plan data to task-action data.
system:
You are a Agent Task Creator and planer.
You will receive <Given Task> that is abstract and your objective
instantiate this task, and give the step-by-stop actions to take. file environment, which contains the canvas content and control information in <Doc Canvas State:> and <Doc Control State
You are provided with
You should review the doc canvas content and control information detail the <Given Task> to a <New Task>. The control information is in dict tree of available control items format.
You are provided with <Available Actions>, you should review the acions carefully and choose the most suitable
step-by-step <Action Plan>. You are also provided with some steps to reference in <Reference Stepa> You should also review these steps carefully, to help you instantiate the original task and give the actions.
Control item
The control iten is the element the page that you can interact with, limit the actionable control item to the folloving:
"Button" is the control item that you can click.
"Edit" is the control item that you can click and input text. click and switch another page. "ListIte=" is the control item that you can click and select.
"TabItem" is the control item that you
click and select. "ScrollBar" is the control item that you can scroll.
"KenuIte=" is the control item that you
"TreeItes" is the control item that you click and select.
28

---

Published in Transactions on Machine Learning Research (07 (025))
"Document" is the control item that you click and select text.
"Hyperlink" is the control item that you can click and open link.
"ComboBor" is the control item that you click and input text. The Google search box is an example of ComboBox.
## Available Actions the control item All the available actions are listed below: {apis}
## The requirements for <New Task>
1. The <New Task> must based on the given task.
2. The <New Task> must be able to be completed step-by-step by Windows
Operating System or an Application on Windows platform
3. You should try your best not to make the <New Task> become verbose, <New Task
> can only add up to 50 words into #Given Task#.
4. The detailed target in <New Task> should be apecific and clear based the doc canvas content and control information.
5. The <New Task> should be able implemented by the available controls and actions.
## The requirements for <Action Plan>
1. The <Action Plan> should step-by-step actions to take in the doc file environment.
2. Each action should be in the available actions from <Available Actions>.
3. Each action should be generated with "step" description which is the function description of the action.
## Response Format
You required reaponse in JSON format, consisting of several distinct parta with the following keys and corresponding content:
{{
"observation": <Outline the observation of the provided doc file environment
based on the given Canvas State and Control State>, "thought": <0utline your thinking and logic of your New Task and the actions to consider the observation of environment and avaiable controls actions>,
"new_tae <ask" the detailed New Task based on Given Task and the observation of doc environment>,
pactions_plam":<Ct the detailed step-by-step actions plan based on the Available Actions and the observation doc environment., The format should be a list of action call format separated by "\n">
}}
### Action Call Format
The action call format is the same as the available actions in the API list.
You are required to provide the action call format in JSON format: {{
"step": <The step description the function the action, which is also the subtask completed by the current action>
"controllabel"; <Specify the precise annotated label of the control item to be selected, adhering strictly the provided options in the field of "label in the <Doc Control State:>. If you believe none of the control item is suitable for the task the task complete, kindly output empty string
>>
"controlText"; <Specify the precise control_tens text of the control item be selected, adhering strictly to the provided options in the field of control_texts in the Control State:>. The control text must match exactly with the selected control label. If the function to call do not
29

---

Published in Transactions on Machine Learning Research (07 (025))2
need specify controlText the task is complete, you kindly output empty string.
If the function call need specify controlTert and of the control item is suitable for the rask,you should imput a possible control name without arguments be
"function": <Specify the precise API function called on the control item to complete the user request, e.g., Llick_input. Leave function is suitable for the task or the task is complete.>
empty string " if you believe none the API
"args": <Specify the precise arguments in dictionary format the selected API function to be called on the control item complete the user request, e.g.,, {("control_id"; "1", "button": "left", "double": false th. Leave it empty dictionary {(} if you the API does not require arguments, or you believe none of the API function is suitable for the task, the task is complete.>
}}
e.g.
{
"step": "change the borders", controllabel": ""
"controlText"; "Borders", "function": "llick_input". "args": {{
"button": "left", "double"; false
}}
}}
{
"step": "change the borders", "controlLabel"; "101", "controlText"; "Borders", "function"; "lickkinpptt,, "args": {{ control_id"" "101", "butoon:: "left", "double": false
}}
}}
c
"step": "select the target text", "controllabel": "",
"controlText"; "",
"function"; "select_text", "args": "text": "Test For Fun" }}
}}
The <actions_plas field muat be strictly in format separated each action call by "\n". The list format should be like this: "action call 1Inaction call 2lnaction call 3"
If you think the original task do not need to be detailed, you can directly copy the original taak the "new_task".
You should review the apis function carefully and if the function to call need specify target control, the "controlText field cannot be set empty.
30

---

Published in Transactions on Machine Learning Research (07/2025)
The "step" description should be consistent with the action and also the thought.
## Here are some examples for you to complete the user request: (examples)
Tips
Read the above instruction carefully. Make sure the response and action
strictly following these instruction and meet the user request.
Make sure you answer must be strictly in J90N format only, without other
redundant text such as jaon header. Your output must be able to able to parsed by json. 1oads().. Otherwise, it will crash the system and destroy the computer.
Your task is very important to improve the agent performance. will tip you 2008 if you do well. Thank you for your hard work!
user: 1-
<Given Task:> (given_task)
<Reference Steps:> (reference_stone steps)
<Doc Canvas State:> {doc_ canvas, state}
<Doc Control State:> boto__conrol_stteee
<Your response:>
B.2 Evaluation
The instantiation prompt used in the evaluation phase when converting task-plan data to lask-action data.
system:
You are an evaluator who can evaluate whether an agent has successfully completed task in the <Original Request>. The agent is an AI model that can interact with the desktop application and take actions.
The thought of agent plan is provided in the <Thought>. You will be provided with task and the <Execution Trajectory> of the agent, including the agent actions that have been taken, and the change of environment.
You will also be provided with final canvas state in <Final Env Status>, You will also be provided with a canvas difference <Canvas Diff>. You will also be provided with the initial control state in <Init Control State
You will also be provided with the final control state after each action in Final Control State>.
Besides, you will also be provided with two acreenshots, one before the agent execution and after the agent execution.
Please judge whether screenshots and the <Execution Trajectory>. 2. You are required to judge whether agent has finished the task or not by observing screenshot differences and the intermediate steps of the agent.
agent successfully completed the based the
# Execution trajectory information
Here are the detailed information about a piece of agent execution trajectory
item:
number: The number of action in the execution trajectory.
action: The action that the agent takes in the current step. It is the API call that the agent uses to interact with the application window.
31

---

Published in Transactions on Machine Learning Research (07/2025)
You will get list of trajectory items the <Execution Trajectory> of the agent actions.
#*# Control State
A control item is the element on the page that can interact with, we limit actionable control item the following:
"Button" is the control item that you can click.
"Edit" is the control item that you can click and input text.
"TabItem" is the control item that you can click and switch to another page. "ListIte=" is the control item that you can click and select.
"MenuItes" is the control item that you can click and select.
"ScrollBar" is the control item that you can scroll.
"TreeIten" is the control item that you can click and select. "Document" is the control item that you can click and select text. "Hyperlink" is the control item that you can click and input text. The Google
click and open link.
"ComboBox" is the control item that search is an example ComboBox.
You are given the information of all available control item in the current application window a hybrated tree format:
{
"control_ :abeel" "label the control item",
"control_text"" name of the control item,
"contro__type: type of the control item,
"selected": False True null, null the control item is not if it is selected,
}}. "children": list of the children control item with format above
### Canvas State Format
The canvas state is in the xml format which transformed from the document
object model (DOM) of the canvas area.
The canvas diff the difference which is in the format of the difference of the xml of the canvas area. Here is an example of xml of a canvas, which show the text content in document:
the canvas area before and after the action
}), [rgpa CK oraeapahhhechahcaaaaaaaaa.aaeeer }}}}, orgpap/.eeeaaaaaaa.r.......arrrr o...0.................0..raraaa.
color" o.o.rreee...0000000000000000aaa1 hra.rhahheeaaaaaa.....raaaaaara"
"440",""u:boti
"(t.aacccrr.o)).....a..o000aaaa. [1tp::ppppppprr.o.oooooo.od0dd"" "336"")))))333)
### Action Explanation
Below is the available API that the agent can use to interact with the
application window. You refer the API usage understand the agent actions.
(apis)
Evaluation Items
32

---

Published in Transactions on Machine Learning Research (07 (025))
You have 2 main items to evaluate:
1. You should also give a overall evaluation of whether the task has been finished, marked "yes", "no" "unsure".
2. You should also give overall evaluation of the quality of task marked as ambiguous", "over detailled or "good".
Criteria for evaluation of the task completion:
1. The <Final Control State and <Final Env Status should be consistent with the task requirements. If the controls canvas content expected to be changed not changed, the task is not completed.
2. The <Execution Trajectory> should be consistent with the task requirements. If the agent actions are not consistent with the task requirements, the task is not completed.
3. If any action in the <Execution Trajectory> is empty, the task is not completed.
Criteria for evaluation of the task quality:
1. The description of the <Original Request:> should be clear and unanbiguous, without the meaning of "selection".
2. The description of the <Original Request:> should not be too detailed like step-by step actions.
## Response Format
You must strictly follow the below JSON format for your reply, and do not change the format nor output additional information.
{
"task "laak_quality"; The quality the <Original Request:>, which is "ambiguous/ over-detailed/ good", ttaas_complete"" The evaluation of the task completion, which is "yes/no/ unsure",
complete_judgeeents your judgment of whether the task has been finished, and the detailed reasons for your judgment based the provided information,
"quality_judgement" your judgment the quality of the task, and the detailed reasons for your judgment based on the provided information
}}
Please take a deep breath and think step by step. Observe the information carefully and analyze the agent execution trajectory, do not miss any minor details.
Rethink your response before submitting it. Your judgment is very important to improve the agent performance. will tip you 200$ if you provide detailed, correct and high-quality evaluation. Thank you for your hard work!
user: 1-
<Driginal Request:> (request}
<Thought:> (thought)
<Execution Trajectory:> (trajectory)
<Canvas Diff:> (canvas_dift
<Init Control State:> Enit__oottro__sttto)
<Final Control State:> lastall_contro_eee
<Final Eny Status:> ifinal_status)
33

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
<Your response:>
C Templates of training format
The following presents a template the training data format. The parts enclosed "" are fields that need to be filled. The "apis" field corresponds the function information in the respective app, while "control control_itemm" iitem contains the control information of the app under the current screenshot. The "user request" field captures the uscr's current request, "step_historr" history" records the agent's previous trajectory history, and "previous_ outlines the agent's 's planning for the lask in the previous state.
system:
You are virtual assistant that can help users to complete their current requests by interacting with UI Window Os. You are provided a list of control items of the current application window for reference
You provided your previous plan of action for reference decide the step, the previous plan is the list of plan for the future actions made before the current action.
You are provided the steps history, including historical actions of your previous steps for reference decide the next step.
You are required to select the control item and take one-step action on it to calling
complete the user request for one step. The one-step action function with arguments for only once.
You are required to decide whether the task status, and detail a list plan of following actions to accomplish the current user request. Do not include any additional actions beyond the completion of the current task.
## Control item
The control item is the element the page that you interact with, limit the actionable control item to the folloving: click.
"Button" is the control item that you
"Edit" is the control item that you can click and input text.
"TabItem" is the control item that you
click and switch another page. can click and select.
"ListItes" is the control item that
"KenuIte=" is
control item that you can click and select. "ScrollBar" is the control item that you can scroll.
"TreeIte=" is the control item that you can click and select. "Document" is the control item that you can click and select text. "Hyperlink" is the control item that you can click and open link. "ComboBox" is the control item that you can click and input text.
## Action the control item
You are able to use pyvinauto to interact with the control item. {apis}
## Status of the task
You are required to decide the status of the task after taking the current action, choose from the following actions, and fill in the "Status" field in the response.
""COTTIUE": means the task is not finished and need further action. "FINISS"" means the current task is finished for the AppAgent and no further actions required.
## Other Guidelines
You are required to select the control item and take open-step action by calling API it complete the request for step
34

---

Published in Transactions on Machine Learning Research (07 (025))
You required to reaponse in JSON format, consisting of distinct parts with the following keys and corresponding content:
{{
"thought": <Outline your thinking and logic of current one-step action required fulfill the given requeat. You restricted provide you thought for only one step action.>
"control_label"; <Specify the precise annotated label the control item to be selected, adhering strictly to the provided options in the field of label" in the control information. If you believe none the control item is suitable for the task or the task is complete, kindly output empty string .>
"control_name"" <Specify the precise control_text of the control item to be selected, adhering strictly to the provided options in the field of paatrol_text" in the control information. If you believe none of the control item is suitable for the task or the task is complete, kindly output empty string The control text must match exactly with the selected control label.>
"function": <Specify the precise API function name without arguments to called on the control item to complete the user request, e.g., click click_inpul input. Leave it a empty string "" you believe none of the function suitable for the task the task is complete.>
"arga": <Specify the precise arguments in dictionary format of the selected API function to be called on the control item to complete the user request, e.g., (f"button": "left", "double": false}). Leave it a empty dictionary (0) if you the API does not require arguments, API function is suitable for the task, or the task is complete.> "status"; <Specify the status of the task given the action.>
you believe none of the
"plan": <Specify the following list of plan of action to complete the user request. You must provided the detailed steps of action user request. If you believe the task is finished and no further actions are required after current action, leave it an empty list.>
complete
}}
user: |-
<Available Control Item:> fcontrol_itenl
<User Request:> fuser request)
<Previous Actions:> {step_hisg history]
<Previous Plans:> (previous_ples plann)
assistant: |-
{output}
Evaluation Prompt for Task-Plan
The evaluation prompt for results from LAM' after task-plan pretraining.
checking quality of the answer would like to invite you to evaluate the performance of two AI assistants in answering users question <Question>, These two answers are in <Answeri> and <Answer2>, respectively. Your evaluation will contain five sub-evaluation tasks:
are helpful precise assistant
1. Can <Ansveri> solve the users question? Your ansver should be "Yes" or "No"
2. Can <Answer2> solve the users question? Your ansver should be "Yes" or "No".
Both two extract action items from the provided steps in both answers. The action item is defined like combination of action and element. Compare the action items
contain list steps marked by numbers. Your task is
35

---

Published in Transactions on Machine Learning Research (07 (025))
to identify similarities. Output the similar action items. Count the count of similar action items.
Your answer should contain the extracted action item sets (in the format as a list of string).
Your
should contain the list of string). Similar action items are those sharing similar intent or achieving similar goals. Each similar action pair in list should be in the format of "similar action item from action item seti / similar action item from action item set2"
similar action items (in the format
Your answer should contain the count of similar action items.
4. Which assistant provides a more helpful response? Your answer should be "1" or "2", where "1" represents <Answeri> and "2" represents <Answer2>.
Your answer should contain the reason (s) for your choice. You should focus on the length of the ansver or the details of the anaver, but you should focus whether the steps could solve the question and the quality of the steps.
Your output should be in the folloving format in json: {{
"Subtaski": "Yes" or "No",
"Subtask2": "Yes" or "No",
"Subtask3": {f
"Action items in Answeri": ["action item 1", "action item 2", ...]. "Action items in Answer2": ["action item 1", "action item 2", ...]. "Similar action items": ["similar action item 1", "similar action item 2",
...],
"Count of similar action items": 2
}),
"Subtask4": {f
"More helpful assistant"; or "2"
"Reason"; "reason for your choice"
}}
}}
Here is the users question <Question>: (question) The first answer <Ansveri> is: [answer1} The second answer <Ansver2> is: (answer2)
E LAM Training Objectives
The problem formally structured into two key objectives: (i) lask-plan pretraining and (i) decision-makingg training.
Task-plan pretraining aims to enable the LAM to map given task description to structured sequence of plans necessary for accomplishing the lask. The primary objective of this component is to generate accurate and coherent plans. The training dataset consists of task-plan pairs, defined as:
D Dplan PD)
where The task description, P:: A sequence of plans to complete the task. In decision-making training, the dataset consists of task-action trajectories, defined as::
{(11, 0z), (82, 02), ... (sT, ar))
where:
St (state at time step f), comprising:
36

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Task description: A high-level summary of the task.
Step ID: The current step in the task sequence.
Observations: Information including control elements and the current canvas state.
TToughts: Model-generated reasoning for the current step.
Previous actions and plans: The sequence of actions and plans from prior steps.
dt (action taken at time step 1), consisting
Thought: Model's reasoning for the action.
Control label: A label for the control element.
Control name: The name the control to interact with.
Function name: The specific function invoked by the action.
Arguments: Parameters passed to the function.
Status: Indicates action's progress, either ongoing (Continue) or completed (Fimish).
The objective of decision-making training is to train the LAM to predict the appropriate action a, for a given state 8, at each time step. This enables the model map input states corresponding actions across the sequence of steps required to accomplish the task.
Broader Impact Statement
The Large Action Model (LAM) framework provides novel approach for training agents in scenarios where no labeled data exists. By combining lask-plan pretraining, initation learning, and autonomous self-boosting with reward- beward-based fine-tuming, LAM enables agents to progress from zero prior knowledge to proficient task execution. This capability is particularly valuable in data-scarce domains, offering a scalable method to bootstrap agent learning using structured knowledge from resources like help documentation. Such an approach makes it feasible deploy AT systems in specialized environments with limited access to traditional training datasets.
The potential impact of LAM extends to wide range of applications, including automation of complex user interactions, optimization of industrial workflows, and enhancement of accessibility technologies. However, the autonomous learning capabilities of LAM also raise ethical concerns. Misaligned or incorrect actions in critical applications could have unintended consequences, and questions of accountability must. be carefully addressed.
To mitigate risks, it is crucial to implement robust evaluation mechanisms and safeguards during deployment. Transparency, alignment. with user intent, and clear oversight are essential to ensure responsible use. By addressing these challenges, the LAM framework offers pathway to extend the reach of Al into domains previously considered inaccossible due to data limitations.
F
Data Deduplication and Train-Test Similarity Analysis
F.1 Semantic Deduplication and Filtering Pipeline
To improve training efficiency and ensure data diversity, applied semantic deduplication pipeline inspired by Self-Instruct. Specifically: We first parsed all raw instruction data and removed irrelevant samples (c.g., non-English instructions, malformed data, smartphone-related lasks). We began with 76,851 raw tasks samples. After applying heuristic filters and semantic deduplication (using Sentence-BEr BERT all-MiniLM-LA v2 (Reimers Gurevych] 2021) with cosine similarity threshold of 0.95), we removed 179 samples, resulting in 76,672 cleaned tasks. The final split used for training and evaluation 1,757 training tasks and 435 test tasks.
37

---

Published in Transactions on Machine Learning Research (07 (025)
ROUGE-L Distribution Between Train and Test Instructions
We computed the ROUGE-L score between each test instruction and its closest training instruction. The summary statistics are Mean ROUGE-L: 0.27, 25th percentile: 0.19, 75th percentile: 0.34, Standard deviation: 0.13.
F.2 Train- Test Instruction Pairs with ROUGE-L Scores
We include several reprosentative train-test pairs below to show their semantic difference.
PAIR 1: ROUGE-L Score = 0.2000 Test: 'Attach a digital signature in Word by navigating to the
Insert' tab, selecting the 'Signature Line' option, and following the prompts to insert and sign the signature line."
Train:
'Select the 'tert to edit' and attempt to access font settings scale the text within the limitations of the available controls."
PAIR 2: ROUGE-L Score = 0.2069 Test: "Access the 'Add-ins' feature within Microsoft Word to explore or manage add-ins." Train: 'Switch to 'Outline' view in Microsoft Word to make structural changes, such as rearranging paragraphs."
PAIR 3: ROUGE-L Score 0.1818 Test: "Edil the opened document in Word, assurving il the
Google Doc that has been downlonded and opened. Train: "Use the 'Search' control to initiate search for 'chore checklist" template in Word."
PAIR 4: ROUGE-L Score 0.2759 Test: 'Navigate to the 'Insert' tab in Word the initial step towards inserting two images side by side."
Train: 'Locate and access the "Mailings" tab in the Word document interface."
PAIR 5: ROUGE-L Score = 0.4000 Test: 'Click on the 'Insert' tab in Word to begin the process of auto-insee insertineenng a paragraph using a command."
Train: "Navigate to the 'Insert' tab in Word as the initial step to use the "entspricht' symbol."
PAIR 6: ROUGE-L Score 0.1250 Test: 'Save Word document and open the font dialog bor for more options."
Train: 'Navigate to the 'Layout' tab in Word to begin the process of removing page nuumbers from specific section."
These examples confirm that although some instructions share interface elements (e.g., "Insert" tab), they are semnntically and functionally distinct, thus preserving the integrity of our train-test. split. The distribution of ROUGE-L score between train and test is in Figure
G Human Evaluation Protocol and Results
To validate the reliability of our LLM-based automatic evaluation, we conducted human study involving three trained annotators. Each annotator independently reviewed a set of 100 randomly sampled task completions. For each sample, annotators answered the following question:
Does the action sequence complete the intended task as described, given the initial application state?
They provided a rating of and No for each task. Each task was evaluated based on the task description, initial state (screenshots and XMT.), and the action sequence executed by the agent.
38

---

Published in Transactions on Machine Learning Research (07 (07/2025)
ROUGE-L
0.8
ROUGE
0.4
0.0
Figure 13: Distribution of ROUGE-L score between train and test.
Quantitative Summary:
Mean agreement with LLM-based evaluation: 91%
Inter-annotator agreement (Cohen's k): 0.85
Rating distribution:
Yes: 83.3% + 4.7%
No: 16.7% + 4.7%
These results demonstrate strong alignment between human judgment and LLM-based scoring. The high Cohen's 6 score (0.85) indicates near-perfect inter-annotator reliability.
39


Reviews:
[{"review_id": "lgtJERpkxf", "reviewer": "Reviewer_yWy8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(**S1**) Unlike papers limited to simulations or synthetic web interfaces, this work is deployed in a live Windows environment (Microsoft Word), enhancing its practical relevance.\n\n(**S2**) The paper delivers a complete pipeline, from synthetic data generation to live deployment.\n\n(**W1**) The notion of a “Large Action Model” is largely a rebranding of prior work in action-based LLM agents (e.g., OS-Copilot, RT-2). The conceptual framing is not as original as claimed.\n\n(**W2**) Comparisons are only made to GPT-4o. Other relevant strong baselines like ReAct-style agents, open-source GUI agents (e.g., Gorilla, ScreenAgent), or fine-tuned Vicuna models are omitted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents Large Action Models (LAMs), a framework for building language-based agents capable of controlling desktop GUI environments. The authors propose a practical, reproducible workflow for training and evaluating action agents, including synthetic data generation, imitation learning, self-bootstrapping, and offline reinforcement learning. They develop a two-stage, LLM-driven pipeline to automatically produce over 2,000 verified task-action trajectories with minimal human labeling. The detailed comparisons are provided with GPT-4-o, demonstrating that a text-only LAM achieves comparable or better performance at lower execution latency.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(**RC1**) The paper acknowledges failure modes and ethical risks only briefly and lacks detailed analysis of robustness, failure cases, or safety in long-horizon or out-of-distribution tasks.\n\n(**RC2**) Some of the sentences have the risk of over-claiming:\n\n- *\"We are the first to propose Large Action Models.\"* Prior arts, such as RT-2, ScreenAgent, and OS-Copilot, use different names but the same idea. This work sounds more like extend the concept of large language and vision–language models to desktop GUI control.\n\n- *\"Comprehensive pipeline not shown before.\"* Similar multi-stage pipelines (plan / act / reflect / RL) exist in quite a few agentic workflow papers.\n\n(**RC3**) The work claims that their pipeline can scales without expensive human labels. However it still depends heavily on GPT-4-o for both trajectory validation and task augmentation, which involves significant cost and circularity. The authors need to provide cost analysis and discuss how cheaper open-source models could replace GPT-4o in the loop.\n\n(**RC4**) The paper uses inconsistent citation formats.", "broader_impact_concerns": "The Broader Impact Statement should address these concerns by outlining mitigation strategies, such as ensuring transparent decision-making, incorporating privacy safeguards, actively monitoring for biases, and discussing equitable access to LAM technology.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u2W3VDn6jl", "reviewer": "Reviewer_KDSj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\nThe whole pipeline is very concise and well detailed, clear to read and understand. \nSuch detailed instruction to train an LAM (or even other large models) are quite rare and very useful.\n\n*Weakness*\n\nI see two main points that remains unclear and limiting the impact of that paper.\n\n1) Is that method applicable to other type of tasks (robotics, medicine, video games etc)? Or are some fundamental modifications required depending on the task? (for ex in robotics, actions are no longer sequences of discrete tokens but take continuous values)\n\n2) Is the proposed method (and especially the training phases) a generic one applied in many papers (and if yes, it could be nice to remind them at the beginning of each section), or is it a specific blend introduced by the present paper (in that case, the paper could be strongly improved by more justifications of the approach, or a comparison with other existing approaches).\n\nDetails and Questions:\n\nP6, in 2.1 → what do you mean by physical? the example tasks explored in this paper does not seem physical.\n\nTable 3 --> how is it possible that LAM^2 (from phase 2) outperforms GPT4, since it is basically distilling it in phase 2? also GPT4 does not seems very better than GPT4-mini, maybe something is wrong with GPT4's column?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides an end-to-end recipe for training a Large Action Model (LAM) from scratch, using GPT4 as an helper tool for building and cleaning data, as well as expert teacher for initial phases of training.\n\nIn the proposed scenario, the LAM is implemented on a Windows OS environment, and the type of tasks are software manipulation (opening and editing a document, etc).\n\nThe entire procedure is divided into 3 main parts:\nData collection\nTraining\nEvaluation", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My suggestions of changes might depend on the answer to the questions asked above.", "broader_impact_concerns": "As explained in the paper, in 8.2, LAM bring some ethical concerns, which are sufficiently stated in my opinion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JmPRqGKYKX", "reviewer": "Reviewer_CeVy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The authors present quite some detail on their data collection and pre-processing method.\n- The authors collect a large and high-quality dataset, and present an interesting method to ensure validity of the action trajectories\n- The proposed method achieves a high performance on a Word interaction task, comparable to GPT-4o using a 7B model.\n\n## Weaknesses\n\n**Experimental setup**\n\nThe experimental is, as far as I can judge, not sound:\n- It is unclear to me how the offline evaluation tasks are selected, and whether the LAM is trained on them or not. \n- Although the 435 Word environment trajectories are held out from the training set, given that they are generated by LLMs or scraped from the internet, it is unclear how different they are from the training tasks. \n- Except comparing to GPT-4o out of the box, no baselines are evaluated. \n- Evaluations are all done automatically using LLMs, I would expect at least a handful of human evaluations to confirm that the LLM evaluator is a reasonable proxy for actual accuracy. \n- The main improvement of LAM over GPT-4o is on step precision. It is not clear to me why step precision is a reasonable metric to evaluate on. One can presumably achieve task success using different trajectories. The result in Section 5.2.2 then in my opinion does not show that LLMs have limitations in executing actions but rather shows that you can distill GPT-4o performance into smaller action models. It seems more likely to me that the LAM achieves a higher step precision because it is trained on similar steps as it is tested on, whereas GPT-4o was not.\n- A lot of discussion in the introduction about LLM's failure to do complex planning, but it seems like your method is only tested on very short-term planning (less than 2 actions on average per trajectory for Phase 2 training (top of page 15))\n\n**Applicability of the method**.  \nIn essence, you are doing a lot of domain-specific data collection and training to get a model to perform well on a single environment. This means that likely for each new environment you need to repeat most of your pipeline, whereas GPT-4o can seemingly achieve similar results out of the box. This makes the method less scalable.\n\n**Writing**.  \nThe paper is 25 pages before references/appendix, but it should be much shorter. There is a lot of repetition throughout the paper that can be removed or condensed. For example, the task-plan data collection in section 3, *\"This data is used to fine-tune the model to generate effective plans and improve its high-level reasoning and planning capabilities\"* and *\"The task-plan data aims at enhancing the model’s high-level planning capabilities, allowing it to generate detailed, step-by-step plans based on user requests.\"* a few sentences later. I would say the first 8 pages can be condensed into at most 3 pages. There's other issues with the writing, some of the English requires some work (e.g. Figure 1), the citations are not formatted correctly, and sometimes the writing is too unscientific for an academic article (claims are made without evidence, some examples given below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces an approach for fine-tuning Large Action Models (LAMs) from Large Language Models (LLMs), specifically focusing on models that can execute actions in real-world environments. The methodology consists of several key steps. First, the authors curate a dataset by scraping task descriptions from multiple sources, including WikiHow and Bing search queries. They then leverage LLMs to generate detailed plans and step-by-step action sequences for these tasks. To enhance the dataset's complexity and coverage, they use LLMs to create additional variations with added constraints, expanding the dataset by 50% to reach a total of 76,672 tasks (29,182 original + 47,490 augmented). The authors ensure data quality through validation: a subset of tasks are manually paired with an appropriate environment (such as templated Word documents) and verified for action executability. They also implement an LLM-based evaluation system to verify whether the generated plans and action sequences successfully accomplish their intended tasks. To demonstrate their approach, the authors present a case study using Microsoft Word as the target environment. The fine-tuning process involves four distinct phases: three stages of supervised fine-tuning focused on plans and action trajectories, followed by a final phase of offline reinforcement learning using a reward model. They choose Mistral 7B as their base model for fine-tuning. The authors evaluate the resulting LAM both offline and online. In offline evaluations (without environment interaction), their model achieves comparable performance to GPT-4o on general tasks and superior performance on Word-specific tasks, despite its significantly smaller size. In online testing, the LAM outperforms GPT-4o in text-only scenarios. However, when GPT-4o has access to image inputs, it achieves better performance than LAM, albeit at the cost of longer processing times – approximately three times slower than LAM. The authors note that their mini version operates at about 1.5 times the speed of the standard implementation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*Required changes for acceptance*:\n- Discuss how you can guarantee that the test set is held-out for evaluation, and that it is sufficiently different from training data.\n- A small manual evaluation to confirm the automatic evaluation is reasonable\n- More discussion on the trade-offs of this whole pipeline compared to simply using GPT-4o out of the box. Even though GPT-4o is almost 3 times slower at inference time, it does not require the whole task-specific data collection and training pipeline. Further, when using images, it seems like GPT-4o is still better than the LAM in the case study.\n- Shorten the paper significantly\n- Make sure all claims have the required evidence\n\n*Recommended changes, not required for acceptance*\n- Discuss why step precision is a reasonable metric, e.g. when GPT4o gets a lower step precision but still a task success, are the steps wrong or just different also valid steps?\n- Typo: *\"where we first The reward model (RM) is built using LAM3\"* section 4.4\n- Would be useful to give a one-line summary of how ReAct works to generate succesful trajectories (Section 4.3)\n\nExamples of claims without evidence:\n- *\"This evaluation step ensures that only valid, accurate task-action data is included in the training dataset, contributing to the reliability and robustness of the LAM.\"* -> the evaluation is done automatically with an LLM, so it does not ensure validity or accuracy.\n- *\".. gains a robust task-agnostic planning capability\"* -> the model does not gain a task-agnostic planning capability as far as I can see, it's in task-conditioned throughout the paper\n- Several times it is mentioned you develop a LAM from scratch, but then the only experiment you do is fine-tuned from a Mistral base model", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)