LoRA-Contextualizing Adaptation of Large Multimodal Models for Long
  Document Understanding
Authors: name: Jian Chen; name: Ruiyi Zhang; name: Yufan Zhou; name: Tong Yu; name: Franck Dernoncourt; name: Jiuxiang Gu; name: Ryan A. Rossi; name: Changyou Chen; name: Tong Sun
Published: 2024-11-02 02:09:01

Abstract: Large multimodal models (LMMs) have recently shown great progress in
text-rich image understanding, yet they still struggle with complex,
multi-page, visually-rich documents. Traditional methods using document parsers
for retrieval-augmented generation suffer from performance and efficiency
limitations, while directly presenting all pages to LMMs leads to
inefficiencies, especially with lengthy documents. In this work, we present a
novel framework named LoRA-Contextualizing Adaptation of Large multimodal
models (LoCAL), which broadens the capabilities of any LMM to support
long-document understanding. We demonstrate that LMMs can effectively serve as
multimodal retrievers, fetching relevant pages to answer user questions based
on these pages. LoCAL is implemented with two specific LMM adapters: one for
evidence page retrieval and another for question answering. Empirical results
show state-of-the-art performance on public benchmarks, demonstrating the
effectiveness of LoCAL.