LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via
  Hierarchical Window Transformer
Authors: name: Yipeng Zhang, name: Yifan Liu, name: Zonghao Guo, name: Yidan Zhang, name: Xuesong Yang, name: Chi Chen, name: Jun Song, name: Bo Zheng, name: Yuan Yao, name: Zhiyuan Liu, name: Tat-Seng Chua, name: Maosong Sun

In multimodal large language models (MLLMs), vision transformers (ViTs) are
widely employed for visual encoding. However, their performance in solving
universal MLLM tasks is not satisfactory. We attribute it to a lack of
information from diverse visual levels, impeding alignment with the various
semantic granularity required for language generation. To address this issue,
we present LLaVA-UHD v2, an advanced MLLM centered around a Hierarchical window
transformer that enables capturing diverse visual granularity by constructing
and integrating a high-resolution feature pyramid. As a vision-language
projector, Hiwin transformer comprises two primary modules: (i) an inverse
feature pyramid, constructed by a ViT-derived feature up-sampling process
utilizing high-frequency details from an image pyramid, and (ii) hierarchical
window attention, focusing on a set of key sampling features within cross-scale
windows to condense multi-level feature maps. Extensive experiments demonstrate
that LLaVA-UHD v2 achieves superior performance over existing MLLMs on popular
benchmarks. Notably, our design brings an average boost of 3.7% across 14
benchmarks compared with the baseline method, 9.3% on DocVQA for instance. We
make all the data, model checkpoint, and code publicly available to facilitate
future research.