Large Action Models: From Inception to Implementation
Authors: name: Lu Wang; name: Fangkai Yang; name: Chaoyun Zhang; name: Junting Lu; name: Jiaxu Qian; name: Shilin He; name: Pu Zhao; name: Bo Qiao; name: Ray Huang; name: Si Qin; name: Qisheng Su; name: Jiayi Ye; name: Yudi Zhang; name: Jian-Guang Lou; name: Qingwei Lin; name: Saravan Rajmohan; name: Dongmei Zhang; name: Qi Zhang
Published: 2024-12-13 11:19:56

Abstract: As AI continues to advance, there is a growing demand for systems that go
beyond language-based assistance and move toward intelligent agents capable of
performing real-world actions. This evolution requires the transition from
traditional Large Language Models (LLMs), which excel at generating textual
responses, to Large Action Models (LAMs), designed for action generation and
execution within dynamic environments. Enabled by agent systems, LAMs hold the
potential to transform AI from passive language understanding to active task
completion, marking a significant milestone in the progression toward
artificial general intelligence.
  In this paper, we present a comprehensive framework for developing LAMs,
offering a systematic approach to their creation, from inception to deployment.
We begin with an overview of LAMs, highlighting their unique characteristics
and delineating their differences from LLMs. Using a Windows OS-based agent as
a case study, we provide a detailed, step-by-step guide on the key stages of
LAM development, including data collection, model training, environment
integration, grounding, and evaluation. This generalizable workflow can serve
as a blueprint for creating functional LAMs in various application domains. We
conclude by identifying the current limitations of LAMs and discussing
directions for future research and industrial deployment, emphasizing the
challenges and opportunities that lie ahead in realizing the full potential of
LAMs in real-world applications.
  The code for the data collection process utilized in this paper is publicly
available at: https://github.com/microsoft/UFO/tree/main/dataflow, and
comprehensive documentation can be found at
https://microsoft.github.io/UFO/dataflow/overview/.