An Interactive Agent Foundation Model
Authors: name: Zane Durante, name: Bidipta Sarkar, name: Ran Gong, name: Rohan Taori, name: Yusuke Noda, name: Paul Tang, name: Ehsan Adeli, name: Shrinidhi Kowshika Lakshmikanth, name: Kevin Schulman, name: Arnold Milstein, name: Demetri Terzopoulos, name: Ade Famoti, name: Noboru Kuno, name: Ashley Llorens, name: Hoi Vo, name: Katsu Ikeuchi, name: Li Fei-Fei, name: Jianfeng Gao, name: Naoki Wake, name: Qiuyuan Huang

The development of artificial intelligence systems is transitioning from
creating static, task-specific models to dynamic, agent-based systems capable
of performing well in a wide range of applications. We propose an Interactive
Agent Foundation Model that uses a novel multi-task agent training paradigm for
training AI agents across a wide range of domains, datasets, and tasks. Our
training paradigm unifies diverse pre-training strategies, including visual
masked auto-encoders, language modeling, and next-action prediction, enabling a
versatile and adaptable AI framework. We demonstrate the performance of our
framework across three separate domains -- Robotics, Gaming AI, and Healthcare.
Our model demonstrates its ability to generate meaningful and contextually
relevant outputs in each area. The strength of our approach lies in its
generality, leveraging a variety of data sources such as robotics sequences,
gameplay data, large-scale video datasets, and textual information for
effective multimodal and multi-task learning. Our approach provides a promising
avenue for developing generalist, action-taking, multimodal systems.