MoAI: Mixture of All Intelligence for Large Language and Vision Models
Authors: name: Byung-Kwan Lee, name: Beomchan Park, name: Chae Won Kim, name: Yong Man Ro

The rise of large language models (LLMs) and instruction tuning has led to
the current trend of instruction-tuned large language and vision models
(LLVMs). This trend involves either meticulously curating numerous instruction
tuning datasets tailored to specific objectives or enlarging LLVMs to manage
vast amounts of vision language (VL) data. However, current LLVMs have
disregarded the detailed and comprehensive real-world scene understanding
available from specialized computer vision (CV) models in visual perception
tasks such as segmentation, detection, scene graph generation (SGG), and
optical character recognition (OCR). Instead, the existing LLVMs rely mainly on
the large capacity and emergent capabilities of their LLM backbones. Therefore,
we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages
auxiliary visual information obtained from the outputs of external
segmentation, detection, SGG, and OCR models. MoAI operates through two newly
introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the
outputs of the external CV models, the MoAI-Compressor aligns and condenses
them to efficiently use relevant auxiliary visual information for VL tasks.
MoAI-Mixer then blends three types of intelligence (1) visual features, (2)
auxiliary features from the external CV models, and (3) language features by
utilizing the concept of Mixture of Experts. Through this integration, MoAI
significantly outperforms both open-source and closed-source LLVMs in numerous
zero-shot VL tasks, particularly those related to real-world scene
understanding such as object existence, positions, relations, and OCR without
enlarging the model size or curating extra visual instruction tuning datasets.