CREAM: Coarse-to-Fine Retrieval and Multi-modal Efficient Tuning for Document VQA

Authors: jinxu  zhang
yongqi  yu
yu  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 14:51:17

Abstract: Document Visual Question Answering (DVQA) involves responding to queries based on the contents of document images. Existing works are confined to locating information within a single page and lack support for cross-page question-and-answer interactions. Furthermore, the token length limitation on model inputs can lead to the truncation of answer-relevant segments. In this study, we present CREAM, an innovative methodology that focuses on high-performance retrieval and integrates relevant multimodal document information to effectively address this critical issue. To overcome the limitations of current text embedding similarity methods, we first employ a coarse-to-fine retrieval and ranking approach. The coarse phase calculates the similarity between the query and text chunk embeddings, while the fine phase involves multiple rounds of grouping and ordering with a large language model to identify the text chunks most relevant to the query. Subsequently, integrating an attention pooling mechanism for multi-page document images into the vision encoder allows us to effectively merge the visual information of multi-page documents, enabling the multimodal large language model(MLLM) to simultaneously process both single-page and multi-page documents. Finally, we apply various parameter-efficient tuning methods to enhance document visual question-answering performance. Experiments demonstrate that our approach secures state-of-the-art results across various document datasets.

Main paper: 
 introduction: 
 Document visual question answering holds significant practical value, enabling rapid and precise extraction of answers from extensive documents in response to user queries [29, 30, 34] . As one of the most challenging tasks in the current multimodal field, it requires understanding not only text semantics but also visual and layout information in document images. However, the majority of current methods are limited to single-page documents, demonstrating inadequate performance on multi-page documents [36, 38] as well as on lengthy content within single-page documents.Currently, fine-tuning pre-trained visual document understanding models has yielded impressive outcomes in question-answering tasks involving visually rich documents (VRDs) [1, 15, 17, 18, 41, 42, 44] . This indicates that the integration of large-scale, unlabeled training documents during the pre-training phase of document understanding models can significantly enhance their ability to answer questions from VRDs. Despite notable advancements, these approaches heavily invest in comprehending document images. Yet, most can only process single-page documents or a fixed length of document information, leading to a constrained understanding of documents. This limitation introduces irrelevant information noise and may also result in the loss of pertinent information due to truncation. For multi-page documents, the prevalent method [36, 38] involves first identifying the document page relevant to the query and then processing it as a single-page document. This approach fundamentally lacks generalization capability, underscoring a significant journey still ahead toward practical application.Large language models (LLMs), such as GPT-3 [5] , LLaMA [37] , and PaLM [6] , have rapidly developed and demonstrated remarkable results across a broad spectrum of natural language processing (NLP) tasks. Recently, several methods have been explored to integrate visual features of documents into LLMs for reasoning [2, 10, 13, 43, 48] . Although certain achievements have been realized, understanding visually rich documents remains limited. The primary challenge is that relying solely on image modal fails to fully capture the semantic information of document images, and the area of multi-page document VQA remains unexplored.In this paper, we propose CREAM, a framework that combines high-performance retrieval enhancements with multi-image, multimodal, and efficient instruction tuning. Our approach consists of three modules: (1) an OCR engine that extracts text from document images; (2) a retrieval module that locates relevant document text chunks based on a given question; and (3) a MLLM that combines multi-page document images and related document text chunks for comprehensive fine-tuning. Figure1 illustrates the specific pipeline.Specifically, we first propose an innovative algorithm to enhance the retrieval performance. On one hand, the performance of the existing text embedding models, which is based on similarity search, has reached a bottleneck; on the other hand, the LLMs exhibit strong ranking capabilities. By integrating these approaches-initially performing coarse-grained retrieval with the text embedding model, followed by several rounds of fine-grained grouping and reranking by the LLM-the most relevant text chunks are identified. This method can not only filter out a lot of irrelevant information but also realize cross-page question answering or even multi-hop question answering, which has a strong generalization ability. Furthermore, unlike conventional text, document images encapsulate rich visual information, prompting us to utilize a vision encoder to extract this information to aid in comprehending the associated text semantics. However, current vision encoders are limited to processing single-page documents. Therefore, we have developed a vision encoder that accepts multiple document images and is integrated within an LLM. This allows for the amalgamation of relevant text chunks with corresponding document images, facilitating the concurrent consideration of text and image information pertinent to the query, thereby yielding accurate responses. Experiments were conducted on three single-page document datasets and two multipage document datasets, demonstrating that our method achieves state-of-the-art results in comparison with other methods.The contributions of this paper are summarized as follows:• We propose CREAM, a multimodal large language model designed to enhance the performance and generalization capabilities of document VQA. • We design a coarse-to-fine retrieval algorithm to select the most relevant text chunks from document pages through embedding-based similarity retrieval, multi-round grouping, and LLM reordering, thereby enhancing the questionanswering effectiveness. • We introduce a vision encoder capable of processing multiple document images. By incorporating page information and applying attention pooling for a weighted representation of multi-page document images, we enable visual questionanswering capabilities for multi-page documents through integration with a large language model. • We achieved state-of-the-art results on two multi-page document datasets, as well as the best performance on three single-page document datasets compared to similar methods, and provided extensive ablations to each component in our method.2 Related Work 
visually rich document understanding (vrdu): 
 The VRDU task is aimed at interpreting content within document images, recognized as a formidable challenge. Existing approaches to VRDU can be broadly categorized by their use of Optical Character Recognition (OCR) tools. There are two primary types of models:(1) Two-Stage Models Using OCR Tools, which utilize OCR to extract text and layout information from document images. In this approach, specific pre-training tasks are designed to align visual features, layout information, and textual features within a semantic space. Examples include LayoutLMv3 [15] , UDOP [35] , and Doc-formerv2 [1] , which incorporate tasks like masked image modeling and word-patch alignment, aiming to harmonize the relationship between textual content and its spatial arrangement in documents.(2) End-to-End Models Based on Image Features [17, 18] . This category's pre-training objectives typically involve text recognition tasks akin to OCR, focusing on the nuanced understanding of document images. Recently, with the advent of MLLMs [9, 23, 25, 47, 49] , efforts have been made to integrate vision encoders into LLMs, thereby endowing them with image understanding capabilities. Ureader [43] has been fine-tuned on multiple document understanding datasets, including question-answering and document summarization tasks. LayoutLLM [8] enhances document understanding performance by incorporating pre-trained document vision encoders into large language models. MPLUG-DocOwl 1.5 [13] proposes unified structure learning to boost the performance of MLLMs.While the aforementioned methods harness multimodal information from document images, they necessitate substantial resource consumption for pre-training alignment tasks. Moreover, most of these methods can only process documents with limited information on a single page and struggle to fully grasp the nuances of document information, frequently contending with excessive noise. In our work, acknowledging that such tasks are primarily driven by textual information and augmented with image information, we leverage the reasoning capabilities of LLMs. By refining the instruction-tuning approach, we facilitate the precise generation of answers.RAG significantly enhances the input capabilities of LLMs by integrating retrieved text passages [11, 20] , leading to notable improvements in knowledge-intensive tasks. This enhancement is evident post-fine-tuning and even when used with off-the-shelf LLMs [33] . Currently, RAG plays a pivotal role in addressing two key challenges associated with LLMs: the hallucination of knowledge and the need for up-to-date information. A more recent advancement [28] in the field involves instruction-tuning a Language Model (LM) by appending a fixed number of retrieved passages to the input. This approach is designed to enrich the model's context and understanding by providing additional, relevant information upfront. Furthermore, some methodologies involve jointly pre-training a retriever and an LM, which is then followed by few-shot fine-tuning on specific task datasets.These methods, while effective in addressing open-domain questions, also encounter significant shortcomings, notably the retrieval of non-relevant information. To address this, we have innovatively applied RAG to the task of DVQA. Our approach enhances the relevance of the information retrieved by implementing a coarseto-fine retrieval method. This method is meticulously designed to accurately isolate the specific paragraph containing the answer and to eliminate extraneous content. 
approach: 
 We introduce CREAM, a novel approach based on LLaMA Adapter V2 [47] , distinguished by its incorporation of high-performance document retrieval and visual representation of multi-page document images. Initially, we employ an OCR tool to extract text content from multi-page document images. For the text retrieval module, we introduce the coarse-to-fine retrieval method to construct relevant text chunks and corresponding document images. During the MLLM instruction-tuning phase, we expand the image encoder to capture the semantic representation of multi-page document images, integrating these into the MLLM for enhanced instruction-tuning. Ultimately, CREAM is adept at handling the task of single-page and multi-page document visual question answering.RAG has recently been effectively applied across various domains, however, its application in DVQA, particularly in multi-page DVQA, remains underexplored. Given the limitation of existing models on the volume of document content they can process, prioritizing the retrieval of multi-page documents becomes essential. Furthermore, the current reliance solely on text embedding similarity for retrieval proves insufficient, as it may yield irrelevant information while overlooking pertinent details. Consequently, we have implemented further enhancements.Our proposed coarse-to-fine retrieval method effectively selects content and corresponding document images that are most relevant to the question. Initially, the extracted text is segmented into chunks, such as text chunks of 512 characters in length. Subsequently, the query and each text chunk are analyzed using the similarity of pre-trained text embedding to rank the text chunks. Ultimately, RankVicuna [31] , a ranking pre-trained model of an LLM, groups every 𝑚 text chunk, and after several screening rounds, identifies the 𝑚 text chunks with the highest semantic match. Only the top 3 text chunks are selected as input. The algorithm's flow is depicted in Algorithm 1.Require: 𝑄𝑢𝑒𝑟𝑦: The input query Require: 𝐶ℎ𝑢𝑛𝑘𝑠: A set of document chunks, {𝐶 Most recent multimodal LLMs employ a common framework that utilizes distinct vision and text towers to independently encode the two modalities. These encoded representations are then fused-either by projecting the image representation through one or multiple projection layers, or by direct concatenation-before being fed into LLMs. To our knowledge, all existing visual-language models only accommodate a single document image as input. To tackle accuracy and efficiency challenges, we introduce a new attention-pooling mechanism. This mechanism combines multi-page document images with page information, effectively bridging the gap between multiple image inputs and standard vision language models (VLMs). In particular, multiple images are processed by vision encoders, generating several visual representations through the vision tower where the vision encoder used from Pix2Struct [18] . Subsequently, we augment global location representations by adding them to page embeddings of corresponding document pages:EQUATIONAttention Pooling. It is a weighted average of multiple document image representations.EQUATION)EQUATION𝑁EQUATIONWhere 𝐸 indicates the visual vector of the document images.Visual Query. Upon obtaining the weighted feature representation of multi-page document images via the attention pooling mechanism, we generate a visual query vector to capture the global feature representation of these images. Once combined with 𝐸 and processed through 𝑛 layers of vision transformer (ViT) [7] blocks, the visual query ultimately captures a deeper level of multi-page document images.EQUATIONEQUATION𝐷 𝑄 𝑔𝑙𝑜𝑏𝑎𝑙 , as the final representation of document images, is inserted into the LLM through the AdapterMLP [27] to achieve image and text alignment.EQUATIONIn this phase, we integrated multiple parameter efficient tuning techniques, such as LoRA [12, 14, 46] , prefix tuning [19, 21, 26] , and bias tuning, while incorporating the zero-initialization attention mechanism from prior methodologies [9, 40, 47] , enabling us to achieve sophisticated fine-tuning effects with a minimal number of training parameters.LoRA. Trainable low-rank matrices are introduced to modify the query and value matrices within the multi-head attention layer. The specific computation is implemented as follows:Two low-rank matrices, 𝑊 𝑎 and 𝑊 𝑏 are initialized. These matrices have dimensions that are significantly smaller than the original query and value matrices, thereby reducing the number of trainable parameters. The original query and value matrices, denoted as 𝑄 and 𝑉 respectively, are modified using the low-rank matrices. This modification is not a direct replacement but an additive update, which can be mathematically represented as:EQUATION)EQUATIONHere, 𝑄 ′ and 𝑉 ′ represent the updated query and value matrices, respectively.Prefix tuning. A prefix of length 𝑙 is strategically positioned preceding the key and value matrices within each multi-head attention layer. This approach effectively equates to adding 𝑙 additional soft prompt tokens alongside each original token for the computation of similarity measures. In this paper, we add the visual query onto the K-length adaption prompts at all 𝐿 inserted transformer layers. The aggregation of these calculations is conducted as follows:EQUATIONhead 𝑖 = Attn(𝑥𝑊EQUATIONwhere 𝑃 𝑣 𝑙 denotes the adaption prompt incorporating visual information from the given image context [47] . The length is 𝑚.𝑊 (𝑖 ) 𝑞 ,𝑊 (𝑖 ) 𝑘 ,𝑊 (𝑖 ) 𝑣 ∈ R 𝑑 ×𝑑 ℎ , 𝑑 indicates embedding size, 𝑑 ℎ means attention hidden size, 𝑃 (𝑖 ) 𝑘 , 𝑃 (𝑖 )𝑣 ∈ R 𝑙 ×𝑑/𝑁 ℎ , there are 𝑁 ℎ attention heads.To effectively manage the tasks associated with instructionfollowing data, same as LLaMA adapterV2 [9] , we initially unfreeze all normalization layers within LLaMA. For each linear layer in the Transformer, we introduce a bias and a scale factor, both serving as learnable parameters. The input and pre-trained weights of a given linear layer are denoted as 𝑥 and 𝑊 , respectively.EQUATIONwhere 𝑏 = Init(0), 𝑠 = Init(1). ( 14)We initialize the bias and scale factors with zeros and ones, respectively, to stabilize the training process at the early stages. 
experiments 4.1 experimental setup: 
 4.1.1 Multi-page DVQA. Our experiments are conducted on two multi-page DVQA datasets, MPDocVQA [36] and DUDE [38] , both of which limited their answers to a specific page, so accurately locating the page containing the answer is crucial before attempting a response. In MPDocVQA, answers are extractive, whereas answers of DUDE encompass extractive, abstractive, list, and non-answerable types. Furthermore, the MPDocVQA dataset provides both answers and corresponding document pages, whereas DUDE only provides answers, posing challenges in locating answers within large-scale document images.For multi-page documents, since there is no LLM to explore this task currently, we assessed two types of small pre-trained models based on their utilization of varied modal information. The first category encompasses models such as BERT [16] , T5 [32] , Long-Former [3] , and BigBird [45] , which rely on plain text information. BERT and T5 support a maximum sequence length of 512, while LongFormer and BigBird support up to 4096. The second category includes models like T5-2D [38] , LayoutLMv3 [15] , and HiVT5 [36] , which leverage a blend of text, image, and layout information. Among these, T5-2D incorporates text and layout information with a maximum sequence length of 8192. HiVT5, designed specifically for multi-page documents, employs two methods: (1) predicting the document page likely containing the answer, similar to the single-page document process, and ( 2) predicting the answer for each document page, ultimately selecting the most probable one. Thus, it fundamentally addresses single-page document scenarios. This analysis reveals the limitations inherent in current pre-training models when addressing the complexities of multi-page document understanding, highlighting the need for more innovative solutions that can effectively manage the intricate dynamics of document structures across multiple pages.Both methods utilize OCR tools for text extraction. However, existing models input the entire document content, leading to truncation when a document exceeds the maximum acceptable length. This is particularly problematic for spliced multi-page documents, as relevant information may be lost. Therefore, prioritizing the retrieval of relevant information is essential.We conducted experiments across three distinct types of single-page documents, briefly described as follows: the DocVQA dataset [30] consists of a significant collection of scanned and handwritten documents, with answers-usually one or more entities-being extracted from the text. Conversely, the VisualMRC dataset [34] , derived from layout-rich web pages, necessitates that answers be deduced from the original text. The InfographicVQA dataset [29] showcases documents featuring various chart information and complex layouts, posing considerable challenges.For single-page documents, since our method uses a small vision encoder and a large language model, for the sake of fairness, we compare two types of models related to our method comparison, one is small models with a pure visual or text-only modal encoder, and the other is MLLMs, which belong to the same camp as our method. In addition to the T5 [32] described above, Donut [17] and Table 1 : Comparison results of MPDocVQA and DUDE between CREAM and the current pre-training model. In MPDocVQA, GAP indicates the gold answer page is provided, while NGAP indicates the gold answer page is not provided. Instead, the coarse-to-fine retrieval method was used to obtain relevant text and page images. Modality T, L, and V denote text, layout, and vision. Average ANLS results per question type are abbreviated as (Abs)tractive, (Ex)tractive, (N)ot-(A)nswerable, (Li) Pix2Struct [18] focus their pre-training on text parsing from images to understand documents. Qwen-VL [2] , SPHINX [10] , Monkey [22] , and Ureader [43] utilize off-the-shelf pre-trained vision encoders, merging them with LLMs to process multimodal information. However, document images are constituted by various data types, and existing methods fall short of comprehensively understanding them, whether through solely visual or textual analysis. Therefore, our method not only leverages the textual comprehension capabilities of large language models but also employs powerful document vision encoders for visual understanding, enhancing document visual question-answering performance by integrating both aspects without any pre-training task.In our experiments, the effectiveness of the retrieval module is important, leading us to compare several embedding models 1 , including e5-large, instructor-large, and bgelarge. Given the extensive volume of data involved in our testing, we ultimately selected the open-source bge-large as our retrieval model due to its robust performance. For the retrieval framework, we 1 https://huggingface.co/spaces/mteb/leaderboard utilized Langchain2 , known for its efficiency in handling complex retrieval tasks. In the group ranking phase, we set the text chunks for each group to 8.During the instruction-tuning experiment, we opted for Pix2Struct [18] vision encoder and LLaMA2 [37] 7B as the backbone and utilized a single NVIDIA Tesla A100 80G GPU to train each dataset for 5 epochs. The batch size was set to 6, with a learning rate of 5e-5 and a weight decay of 0.02. We also set the visual prefix length to 65, applying insertions exclusively in the last 30 layers of the model. For LoRA, we set the rank r to 8. For the training phase, we chose three pages of relevant documentation as input, while for the test phase, it ranged from one to three pages.It is noteworthy that VisualMRC is measured by CIDEr [39] , and other datasets are evaluated by ANLS [4] . CIDEr is calculated as follows:EQUATIONWhere 𝑐 𝑖 is the candidate sentence, 𝑆 𝑖 is the set of reference sentences, 𝑚 is the number of reference sentences, n is the length of n-gram, 𝑔 𝑛 (𝑐 𝑖 ) and 𝑔 𝑛 (𝑠 𝑖 𝑗) are the vector of candidate sentences and reference sentences.The calculation formula of ANLS can be expressed as:EQUATION)where N is the total number of samples. Levenshtein(𝑠 𝑖 , 𝑡 𝑖 ) is the Levenshtein distance between the source string 𝑠 𝑖 and the target string 𝑡 𝑖 for the 𝑖 -𝑡ℎ sample. In the experiment, we followed the previous approach and set the threshold to 0.5.Table 1 showcases the performance comparison of CREAM against seven models across two multi-page datasets. As demonstrated, BERT [16] , LongFormer [3] , BigBird [45] , and LayoutLMv3 [15] are limited to handling extractive answer types in the MPDocVQA datasets and exhibit poor performance on other kinds of answers, such as those in the DUDE datasets. The T5-based generative models have shown improvement in abstractive answer types, yet they falter with data containing complex structures like tables and lists. Moreover, given that the DUDE dataset does not specify the page containing the answer, all existing models must accurately answer the question when the answer page is identified. Without this information, relevant content may be truncated due to input limitations, leading to suboptimal performance. For instance, HiVT5 significantly outperforms DUDE in the MPDocVQA. Although our method is influenced by certain factors, it surpasses previous approaches across three answer types. However, the LLM continues to struggle with the hallucination problem, resulting in inferior performance on non-answerable answers compared to existing methods. Table 2 presents the performance of seven models across three single-page document datasets. CREAM exhibited superior performance in all datasets. T5 [32] , Donut [17] , and Pix2Struct [18] experience performance degradation as they rely solely on text modal or image modal information for document comprehension. Recently, attempts to integrate off-the-shelf image encoders into LLMs for aligning text instructions have shown that performance in document visual question answering is lacking. On the one hand, this shortfall is attributed to the superior text comprehension ability compared to the visual understanding of LLMs, on the other hand, the current image encoders exhibit limited capability in understanding document images. CREAM combines the strengths of the aforementioned models. For text comprehension, an LLM is employed, while visual understanding benefits from the superior performance of the Pix2Struct [18] vision model. This superiority, particularly against similar multimodal large language models, underscores the distinct advantages of our method.4.4.1 Multi-page DVQA. Our proposed method is more necessary for multi-page document visual question-answering. Therefore, our retrieval method is more explanatory on multi-page document datasets. As shown in Table 3 , we conduct more detailed ablation experiments on the main modules in this paper.Effect of Text Embedding Module. We utilize a pre-trained text embedding model to rank the text chunks of documents based on vector similarity in relation to the query. The experiment reveals that accurately identifying text chunks related to the query is challenging due to the sparse keyword semantic information in the query itself, and the possibility that each text chunk may contain relevant keywords. Consequently, distinguishing the most relevant text chunk is challenging, making reliance solely on the text embedding module insufficient.Effect of RankLLM Module. Despite the large language model lacking retrieval capabilities, it exhibits strong ranking performance. This is largely attributed to its ability to assess the relevance of a query to a text chunk from a perspective akin to human language comprehension. We employed a strategy of multiple grouping rounds for repeated verification, ultimately identifying the most relevant text chunk. The experiment demonstrates that RankLLM significantly outperforms in both datasets. Effect of Vision Module. Our proposed multi-page document image encoder is a crucial component of this study. Given that the text within document images encompasses rich layouts and visual information, comprehending the document itself is essential. However, the capability of existing vision encoders to comprehend large-scale multi-page documents simultaneously is limited; thus, this study focuses solely on document images pertinent to the query. This encoder offers auxiliary support to the semantic information of the text. Experiments demonstrate that the vision module has a significant impact, particularly on layout-related issues. 4.4.2 Single-page DVQA. In this section, we conduct a detailed analysis of CREAM and its components. To evaluate the impact of individual components on model efficacy, we executed ablation studies across three categories of single-page documents, methodically omitting one module at a time, as shown in Table 4 .Effect of Vision Module. The ablation study on the vision module revealed that visual information is pivotal for single-page document VQA. This is attributed to the text in a document image encapsulating not only the semantic content but also its spatial position and color. Integrating this multi-modal information is essential for a deeper understanding of the document image content.Effect of Retrieval Module. The majority of documents exceed 512 tokens, leading most models to employ a truncation strategy that risks omitting crucial answer segments. However, our coarse-to-fine retrieval approach accommodates the input length constraints while retaining the most pertinent sections, significantly enhancing model performance. Table 4 illustrates that the exclusion of the retrieval module results in diminished performance. Notably, During the experiment, we found through the experimental results that the retrieval module exerts the most substantial impact, underscoring the efficacy of our retrieval technique across diverse test datasets.Effect of Prefix & Bias Tuning and LoRA. Prefix & Bias tuning and LoRA are orthogonal approaches that do not interfere with each other. Prefix tuning allows for the injection of distinct information across various layers of the model, while Lora can further minimize the number of parameters required, without compromising on model performance. According to the experimental results, it is found that Prefix & Bias tuning and loRA have different performances on different datasets, and better results can be obtained by combining the two. It is noteworthy that visual information serves as a basis for prefix tuning. Consequently, removing the prefix module entails the removal of the vision module as well.Figure 3 shows some qualitative results produced by our CREAM on different types of documents. CREAM is adept at generating answers from documents with complex layouts (case a) and performing reasoning based on the document's content (case b). Furthermore, in multi-page documents, which typically encompass extensive textual information, CREAM can precisely pinpoint the location of an answer, as exemplified in case d. Nonetheless, the use of OCR tools in CREAM introduces certain constraints. Specifically, text extraction occurs sequentially from the top left to the bottom right, resulting in the omission of layout and visual information of the document content. Therefore, the visual information we introduce can alleviate this problem, as shown in case c. The examples provided demonstrate that CREAM is capable of handling both single-page and multi-page documents. It not only accurately locates the relevant content of the query but also adapts its responses to the style of questions based on varying document formats. 
conclusion: 
 This paper introduces a new model named CREAM, designed to perform single-page and multi-page DVQA simultaneously. Specifically, a coarse-to-fine retrieval algorithm is developed to extract text chunks relevant to the query. In addition, CREAM is structured as a multimodal large language model capable of processing multi-page documents and generating answers by questions, relevant text chunks, and corresponding document images. Within this framework, CREAM not only processes multimodal information but also lays a solid foundation for future multi-document image processing. Extensive experiments across two multi-page and three single-page DVQA datasets demonstrate our method's superiority over existing approaches in terms of accuracy, generalization ability, and efficiency in document visual question answering. Its limitation is that it is affected by the performance of the retrieval and ranking models, otherwise, it would achieve more desirable results.In most document images, text layout and visual information play a key role. Therefore, for vision encoders, comprehending the multimodal information of document images is crucial. Moving forward, we aim to enhance the seamless integration and effective utilization of the model across diverse modalities. 
 
 Appendix: https://www.langchain.com 


Decision: The paper proposes CREAM for Document Visual Question Answering (DVQA) tasks. CREAM introduces a multi-stage retrieval algorithm using coarse-to-fine similarity matching and a vision encoder to enhance performance and generalization. The reviewers generally appreciate the clear presentation and significant experimental results of this work. 

Several points requiring further clarification were noted by the reviewers. The authors addressed most of these concerns during the rebuttal stage. The overall sentiment among the reviewers leans towards acceptance. Although Reviewer C9C1 did not provide a final rating, by reviewing the authors' rebuttal myself, I reckon that most of the raised concerns have been discussed. The majority of reviewers concur that, with revisions, this paper would make a valuable contribution to the field.

Reviews: review: paper_summary: paper_summary: This study introduces a novel method, CREAM, aimed at enhancing the performance and generalization capabilities of Document Visual Question Answering (DVQA). The paper's multi-stage retrieval algorithm employs a coarse-to-fine similarity matching process, followed by multi-round grouping and sorting, as well as Large Language Model (LLM) reordering, effectively filtering out the text chunks most relevant to the query. This overcomes the limitations of existing methods in dealing with cross-page or multi-hop questioning. Additionally, the research designs a vision encoder capable of processing multi-page document images, integrating information from such images through an attention pooling mechanism. This enables the LLM to handle both single-page and multi-page documents concurrently, marking a significant advancement in visual encoder technology. Experimental results across multiple datasets demonstrate that this approach achieves state-of-the-art performance levels, thoroughly validating its efficacy and practical utility.
strength_weakness: strength_weakness: 1.The article is well written, with clear logic and complete structure.

2.The experiments conducted in the paper are comprehensive and have achieved SOTA performance on multiple datasets.
limitations: limitations: 1.In page 3 line 303, “Ultimately, Rank Vicuna, a ranking pre-trained model of an LLM, groups every 𝑚 text chunk.” Why are text chunks grouped during the process? Is this merely to reduce the computational load? The authors are suggested to clarify this point.

2. The authors mention employing a "coarse-to-fine retrieval method." Does this "coarse-to-fine" aspect primarily manifest in the text similarity-matching process?

3. On page 3, line 265,” To address this, we have innovatively applied RAG to the task of DVQA.” In which section do the authors apply the Retrieval-Augmented Generation (RAG) method? It is suggested that the authors dedicate a separate section in the text to outline the application of RAG explicitly.

4. The paper may not adequately address the computational costs of training the model on large-scale datasets or the efficiency in processing lengthy documents.

5. Section 5 of the experiments appears to omit comparisons with notable larger models, such as Qwen, which is also proficient in handling multi-page document processing.
paper_summary: paper_summary: This paper proposed a retrieval based method for both single-page and multi-page document VQA tasks. Based on the usage of off-the-shelf Pix2Struct and Llama models, the method achieved a sota performance with only 93M parameters.
strength_weakness: strength_weakness: 1. The paper is well written and easy to follow.
2. The proposed method is simple and efficient especially for long documents.
limitations: limitations: 1. In Tab. 1, what is "the gold answer page" for MPDocVQA? Do you mean that you provide document pages with the correct page id as input? For DUDE, the results are all in NGAP or GAP? Please explain it with more details.
2. In Tab. 2, the proposed method, CREAM, takes T+V modalities input, why do you only compare with methods either in T or V? In the benchmark website (https://rrc.cvc.uab.es/?ch=17&com=evaluation&task=1), there are a lot more comparable methods with multiple modalities, e.g. LayoutLM2.0 (2020-12-22) with ANLS 86.7%. Please at least add a few multi-modal methods for a fair comparison.
3. In Tab. 3, what is the result w/ Vision, w/ Text, w/o RankLLM? Please add this important result row, and discuss it with more details.
4. The same for Tab. 4, do you have the result w/o RankLLM? The role for RankLLM is to reduce the length of text tokens while maintaining the useful information for a given question, so as to make input fit into the length limitation. 
5. Please use a high contrast theme for Fig.3 for better look.
paper_summary: paper_summary: In this paper, the authors tackle the problem of document VQA, and focus especially on the problem of cross-page question-and-answer interactions.
To this end, they propose the CREAM framework that introduce :
* a coarse-to-fine retrieval method that aims to select the content and corresponding document images that are the most relevant to the question,
* a multi-page vision encoder, that is able to process both single and multi-page document images
* a multimodal efficient tuning that integrates LoRA, prefix tuning and bias tuning with zero-initialization
This framework is evaluated on Document VQA datasets with both single and multi-page documents. 
Along with ablation and case studies, the authors show that CREAM outperforms previous models on multi-page and single-page documents
strength_weakness: strength_weakness: The paper is clear and well written.
Tackling the problem of cross-page question-and-answer interactions is an interesting research topic.
The approach is extensively described and the authors performed a lot of experiments on two kinds of datasets.
limitations: limitations: The authors could have discuss the limitations of the proposed framework.
For example, in Table 1, there is a 10 point performance's drop when the gold answer page is not known, gap that is higher for CREAM than for other approach such as HiVT5. Do the authors have an explanation for this behavior ?
Besides, the authors do not compare the performance of CREAM with LayoutLMv3 for the single-page documents, while they do it for multi-page documents. Why ? The results of the model on DocVQA are easily accessible form the paper of Huang et al. (78.73 or 83.37)
From Figure 4, we can see that the nature of the question can be different, i.e. either relying on the layout ("What is the position of the people picture in the document?") or just the textual information ("What does FSF stands for?"). It could have been interesting to have an analysis of the results according to the nature of the questions. Is there more mistakes for some particular types of questions?

Remarks : 
* documents and texts are too small to be readable on Figures 2 and 4 when printed
* "large language model(MLLM)" --> "large language model (MLLM)"
paper_summary: paper_summary: This paper introduces CREAM, a novel approach incorporating a coarse-to-fine retrieval strategy and a specifically designed vision encoder to address the multi-page document VQA challenges. The proposed CREAM demonstrate SOTA performance on a range of multi-page and single-page document VQA datasets.
strength_weakness: strength_weakness: 1. The specifically designed vision encoder and coarse-to-fine retrieval strategy efficiently address the multi-page document VQA challenges.
2. CREAM outperforms numerous approaches across multi-page document VQA benchmarks such as MPDocVQA and DUDE. Furthermore, it demonstrates impressive results on single-page document VQA datasets including DocVQA, InfoVQA, and VisualMRC, while maintaining a relatively light burden with a mere 93M trainable parameters.
limitations: limitations: 1. To further assess the model's adaptability, the authors may consider evaluating CREAM's performance on general VQA tasks, such as VQAv2 [1]. This would provide insights into whether the model's versatility is compromised outside its specialized domain.

2. While the comparison presented in Table 2 highlights CREAM's superiority over single-modality baselines, it is crucial to ensure a balanced evaluation, since CREAM uses both the text and visual modality. The authors may include a broader range of baseline models that also incorporate T+V information.

[1] Yash Goyal et al. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR, 2017.