Magma: A Foundation Model for Multimodal AI Agents
Authors: name: Jianwei Yang, name: Reuben Tan, name: Qianhui Wu, name: Ruijie Zheng, name: Baolin Peng, name: Yongyuan Liang, name: Yu Gu, name: Mu Cai, name: Seonghyeon Ye, name: Joel Jang, name: Yuquan Deng, name: Lars Liden, name: Jianfeng Gao

We present Magma, a foundation model that serves multimodal AI agentic tasks
in both the digital and physical worlds. Magma is a significant extension of
vision-language (VL) models in that it not only retains the VL understanding
ability (verbal intelligence) of the latter, but is also equipped with the
ability to plan and act in the visual-spatial world (spatial-temporal
intelligence) and complete agentic tasks ranging from UI navigation to robot
manipulation. To endow the agentic capabilities, Magma is pretrained on large
amounts of heterogeneous datasets spanning from images, videos to robotics
data, where the actionable visual objects (e.g., clickable buttons in GUI) in
images are labeled by Set-of-Mark (SoM) for action grounding, and the object
movements (e.g., the trace of human hands or robotic arms) in videos are
labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show
that SoM and ToM reach great synergy and facilitate the acquisition of
spatial-temporal intelligence for our Magma model, which is fundamental to a
wide range of tasks as shown in Fig.1. In particular, Magma creates new
state-of-the-art results on UI navigation and robotic manipulation tasks,
outperforming previous models that are specifically tailored to these tasks. On
image and video-related multimodal tasks, Magma also compares favorably to
popular large multimodal models that are trained on much larger datasets. We
make our model and code public for reproducibility at
https://microsoft.github.io/Magma.