LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large
  Multimodal Models
Authors: name: Feng Li; name: Renrui Zhang; name: Hao Zhang; name: Yuanhan Zhang; name: Bo Li; name: Wei Li; name: Zejun Ma; name: Chunyuan Li
Published: 2024-07-10 17:59:43

Abstract: Visual instruction tuning has made considerable strides in enhancing the
capabilities of Large Multimodal Models (LMMs). However, existing open LMMs
largely focus on single-image tasks, their applications to multi-image
scenarios remains less explored. Additionally, prior LMM research separately
tackles different scenarios, leaving it impossible to generalize cross
scenarios with new emerging capabilities. To this end, we introduce
LLaVA-NeXT-Interleave, which simultaneously tackles Multi-image, Multi-frame
(video), Multi-view (3D), and Multi-patch (single-image) scenarios in LMMs. To
enable these capabilities, we regard the interleaved data format as a general
template and compile the M4-Instruct dataset with 1,177.6k samples, spanning 4
primary domains with 14 tasks and 41 datasets. We also curate the
LLaVA-Interleave Bench to comprehensively evaluate the multi-image performance
of LMMs. Through extensive experiments, LLaVA-NeXT-Interleave achieves leading
results in multi-image, video, and 3D benchmarks, while maintaining the
performance of single-image tasks. Besides, our model also exhibits several
emerging capabilities, e.g., transferring tasks across different settings and
modalities. Code is available at https://github.com/LLaVA-VL/LLaVA-NeXT