AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Authors: name: Seungwhan Moon; name: Andrea Madotto; name: Zhaojiang Lin; name: Tushar Nagarajan; name: Matt Smith; name: Shashank Jain; name: Chun-Fu Yeh; name: Prakash Murugesan; name: Peyman Heidari; name: Yue Liu; name: Kavya Srinet; name: Babak Damavandi; name: Anuj Kumar
Published: 2023-09-27 22:50:51

Abstract: We present Any-Modality Augmented Language Model (AnyMAL), a unified model
that reasons over diverse input modality signals (i.e. text, image, video,
audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the
powerful text-based reasoning abilities of the state-of-the-art LLMs including
LLaMA-2 (70B), and converts modality-specific signals to the joint textual
space through a pre-trained aligner module. To further strengthen the
multimodal LLM's capabilities, we fine-tune the model with a multimodal
instruction set manually collected to cover diverse topics and tasks beyond
simple QAs. We conduct comprehensive empirical analysis comprising both human
and automatic evaluations, and demonstrate state-of-the-art performance on
various multimodal tasks.