LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via
  Hierarchical Window Transformer
Authors: name: Yipeng Zhang; name: Yifan Liu; name: Zonghao Guo; name: Yidan Zhang; name: Xuesong Yang; name: Chi Chen; name: Jun Song; name: Bo Zheng; name: Yuan Yao; name: Zhiyuan Liu; name: Tat-Seng Chua; name: Maosong Sun
Published: 2024-12-18 14:07:46

Abstract: In multimodal large language models (MLLMs), vision transformers (ViTs) are
widely employed for visual encoding. However, their performance in solving
universal MLLM tasks is not satisfactory. We attribute it to a lack of
information from diverse visual levels, impeding alignment with the various
semantic granularity required for language generation. To address this issue,
we present LLaVA-UHD v2, an advanced MLLM centered around a Hierarchical window
transformer that enables capturing diverse visual granularity by constructing
and integrating a high-resolution feature pyramid. As a vision-language
projector, Hiwin transformer comprises two primary modules: (i) an inverse
feature pyramid, constructed by a ViT-derived feature up-sampling process
utilizing high-frequency details from an image pyramid, and (ii) hierarchical
window attention, focusing on a set of key sampling features within cross-scale
windows to condense multi-level feature maps. Extensive experiments demonstrate
that LLaVA-UHD v2 achieves superior performance over existing MLLMs on popular
benchmarks. Notably, our design brings an average boost of 3.7% across 14
benchmarks compared with the baseline method, 9.3% on DocVQA for instance. We
make all the data, model checkpoint, and code publicly available to facilitate
future research.