LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via
  Hybrid Architecture
Authors: name: Xidong Wang; name: Dingjie Song; name: Shunian Chen; name: Chen Zhang; name: Benyou Wang
Published: 2024-09-04 17:25:21

Abstract: Expanding the long-context capabilities of Multi-modal Large Language
Models~(MLLMs) is crucial for video understanding, high-resolution image
understanding, and multi-modal agents. This involves a series of systematic
optimizations, including model architecture, data construction and training
strategy, particularly addressing challenges such as degraded
performance with more images and high computational costs. In this
paper, we adapt the model architecture to a hybrid of Mamba and Transformer
blocks, approach data construction with both temporal and spatial dependencies
among multiple images and employ a progressive training strategy. The released
model LongLLaVA~(Long-Context Large
Language and Vision Assistant) is the first
hybrid MLLM, which achieved a better balance between efficiency and
effectiveness. LongLLaVA not only achieves competitive results across various
benchmarks, but also maintains high throughput and low memory consumption.
Especially, it could process nearly a thousand images on a single A100 80GB
GPU, showing promising application prospects for a wide range of tasks.