Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding
Authors: Jaeyoo Park
Jin Young Choi
Jeonghyung Park
Bohyung Han
Venue: NeurIPS 2024 poster (2024)
Keywords: Document Understanding
Multi-modal Learning
Primary area: machine_vision
TL;DR: An OCR-free document understanding framework that efficiently processes multi-scale visual features while learning to read text with layout by position-aware instruction tuning.

Abstract: We present a novel OCR-free document understanding framework based on pretrained Multimodal Large Language Models (MLLMs). 
Our approach employs multi-scale visual features to effectively handle various font sizes within document images.
To address the increasing costs of considering the multi-scale visual inputs for MLLMs, we propose the Hierarchical Visual Feature Aggregation (HVFA) module, designed to reduce the number of input tokens to LLMs. 
Leveraging a feature pyramid with cross-attentive pooling, our approach effectively manages the trade-off between information loss and efficiency without being affected by varying document image sizes.
Furthermore, we introduce a novel instruction tuning task, which facilitates the model's text-reading capability by learning to predict the relative positions of input text, eventually minimizing the risk of truncated text caused by the limited capacity of LLMs.
Comprehensive experiments validate the effectiveness of our approach, demonstrating superior performance in various document understanding tasks.

Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding
Jaeyoo Park
Jin Young Choi²
Jeonghyung Park2 Bohyung Han
ECE & IPAL Seoul National University
Samsung SDS
Obellos1203 jychoi999, bhhan)@ssu.da ac kr jeong h. park@samsung. com
Abstract
We present novel OCR-free document understanding framework based pre- trained Multimodal Large Language Models (MLLMs). Our approach employs multi-scale visual features effectively handle various font sizes within document images. To address the increasing costs of considering the multi-scale visual inputs for MILLMs, we propose the Hierarchical Visual Feature Aggregation (HVFA) module, designed to reduce the number of input tokens to LLMs. Leveraging a feature pyramid with cross-altentive pooling, our approach effectively manages the trade-off between information loss and efficiency without being affected by varying document image sizes. Furthermore, we introduce novel instruction tuning task, which facilitates the model's text-reading capability by learning to predict the relative positions of input text, eventually minimizing the risk truncated text caused by the limited capacity of LLMs. Comprehensive experiments validate the effectiveness of our approach, demonstrating superior performance various document understanding tasks.
Introduction
In recent years, significant progress has been made in the development of Multimodal Large Language Models (MLLMs) trained large-scale datasets (1-5), demonstrating remarkable performance in various tasks, such as visual question answering 6 77, image captioning [8|10], image-text
retrieval [TК visual grounding [14 [5] achieving robust Optical Character Recognition (OCR) 9 performance by leveraging extensive pretraining data. Such advancements highlight the potential of MLLMS for practical and challenging tasks related to document understanding, by reducing reliance on external OCR engines which often struggle with complex layouts and varied fonts. Additionally, OCR- based models typically require additional modules to process explicit OCR outputs, adding to system complexity.
etc. MLLMs also exhibit the emergent capability
Despite their promising abilities, document understanding frameworks without external OCR en- gines 19 face significant challenges, particularly in considering multiple scales of document images, text and visual elements often vary in size and style. Complex graphical elements, such as charts and diagrams, along with non-standard layouts. pose additional difficulties for the model due to limited receptive fields. This highlights the need for the frameworks effectively handle the variability in document scales to achieve accurate and comprehensive understanding.
Our goal is process visual documents over multiple scales rather than a single fixed scale. However, augmenting visual features for multiple scales inevitably increases costs, which particularly critical for MILMs, where LLMs have quadratic complexity due to the self-attention mechanism. To tackle this issue, we propose the Hierarchical Visual Feature Aggregation (HVFA) module to reduce the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

number of visual tokens by incorporating cross- attentive pooling within feature pyramid. This design accommodates various document image sizes, effectively balancing the trade-off between information retention and efficiency. By reducing the visual features before feeding them into LLMs, our module maintains the original complexity the L.Mss.
In addition to incorporating multi-scale visual inputs, we introduce novel instruction tuning tasks ensure the model's comprehensive text reading ability and capture layout information effectively. While [9 proposes reading all of the texts in images, we observed that certain documents contain more texts than LLMs can hold, resulting challenge, we propose an efficient and effective alternative task of reading, where the model learns read partial texts in images based on their relative positions. Furthermore, teaching the model to predict the relative positions of given texts, which is an inverse task to the aforementioned reading task, enhances its ability to recognize overall layout information.
information loss during training. To address this
The main contributions of our paper are summarized as follows:
We present a novel framework for OCR-free document understanding, built on a pretrained large-scale multi- modal foundation model, which integrates multi-scale visual features to handle varying font sizes in document images.
We introduce the Hierarchical Visual Feature Aggregation (HVFA) module, which employs cross-attentive pooling to effectively balance information preservation and computational efficiency, addressing the escalating costs associated with detailed visual inputs.
We employ novel instruction tuning task, which aims to predict the relative positions of input text, to enhance the model's comprehensive text reading capability.
The proposed approach presents remarkable performance gains the multiple document understanding benchmarks among OCR-free models.
The rest of the paper is organized follows. Section D reviews the related work. The details of our approach are described in Section 3 and the experimental results are presented in Section 4 Finally,
we conclude this paper in Section 5
2 Related Work
This section provides overview of multimodal large language models and their applications to document understanding tasks. Research on document understanding typically falls into two categories: those that incorporate off-the-shelf OCR engines and those that bypass OCR extractors. From the extensive literature on this topic, we highlight the studies most relevant to our work.
2.1 Multimodal Large Language Models
Contrary early multimodal learning approaches that focused on joint embedding learning for diverse downstream tasks [H3]] 24H24), recent frameworks (9] I2
13 25 have shifted toward generating
26 27 This evolution has contributed to various architectural innovations; GiT [9. BLIP 12). and Qwen-VL (28 pioneer encoder-
open-ended language responses by leveraging LLMs
decoder architectures for unified vision-language understanding and generation. Building upon these foundations, BLIP-2 [ and mPL.UGOWl (25) introduce resamplers (Q-Formers) to bridge the vision-language modality gap through query-relevant feature extraction and dimensionality reduction. Further advances have come through InstructBIIIP (29] and LLL.V (30), which enhance multimodal capabilitics by instruction tuning on vision-language conversation data.
Recent studies explore MLLMs' text-reading capabilities and their relationship with image resolution. Wang al. [9] shows MLLMs gradually learn read text in images through large-scale pretraining. However, Liu et al. [3I reveals that most MLLMs are constrained by their visual encoder architecture designed only for 224x224 resolution inputs, significantly limiting their ability to capture fine- grained details in text-centric tasks. While Monkey (32) demonstrates enhanced performance by supporting high-resolution inputs, their analysis was limited to single-scale processing. These investigations collectively indicate that, although MILMs can read text in images, their performance significantly influenced by input resolutions, affecting their ability handle diverse font sizes and capture fine-grained textual details.

---

1979
Large Language Model (LLM)
LoRA
Global View
When was the robot orm developed for assembly lines?
Linear Projection Layer
Hierarchical Visual Feature Aggregation
Local Scale
Reconaler
Aecangler
Retample
Revampler
Recanglter
Recorale
una am
( + VIT
Pe ViT
+ ViT
+ viT
+ vir
+ vir
vir
Local Scale
Global View
Stall
images)
Local Scale (2%) sub-images)
Figure 1: Illustration of the proposed framework. Our model adopts visual features from multiple scales, which aggregated through the Hierarchical Visual Feature Aggregation (HVFA) module. The aggregated features are then fed into an LLM to generate language response in an autoregressive manner. The sub-image highlighted by the red box contains the text relevant to an input question, which requires accurately recognizing visually detailed elements from high resolution image.
2.2 OCR-based Document Understanding Models
Early studies adopted external OCR engines to explicitly model and understand the texts within images [33 39 LayoutLM [33] directly extends BERT [40] jointly model interactions between text, visual and layout information for the first time. Based on LayoutLM, there have been efforts to introduce various pretraining objectives to learn layout information effectively by incorporating image- text alignment (34), cell-level information from OCR outputs [36), and multi-modal masking (7). Despite their strong performance on document understanding benchmarks, these models rely on off-the-shelf OCR engines and struggle with challenging cases such as diverse fonts, handwritten texts, and degraded historical documents. Moreover, they require additional modules for processing OCR results, which increases computational complexity.
2.3 OCR-free Document Understanding Models
To address the drawbacks of OCR-based models, OCR-free models powered by end-to-end multi- modal pretraining have been proposed. Dessurt [16] and Donut [17] combine a learnable visual encoder with autoregressive text decoder for text recognition. Pix2Struct [18] focuses on layout understanding by learning from HTML data and masked webpage screenshots. However, these methods suffer from high computational costs because they require pretraining models from scratch before task-specific fine- sunitunnng. UReader [19] takes a different approach by leveraging pretrained MLLMs, reducing training costs while handling diverse document types through document instruction tuning and ahape-adaptive cropping. Despite these advances, current OCR-fre models still struggle capture diverse visual scales and font sizes, often missing local details in documents.
Method
This section describes our main idea, including multi-scale visual inputs, the Hierarchical Visual Feature Aggregation (HVFA) module, and relative text-position prediction tasks in detail.
3.1 Overview
MILMMs generate instruction-folloviii responses given multi-modal inputs, image 7 and text (instruction) T. The output token sequence with a length of n, is denoted by S (81, )n,), where the probability of generating s; is given by p(s: I, Sz.), 1 < S Ry. Given the correct

---

answer Y fyr,..., Nn, ly, E RRV. where ny is the length of the answer and V is the vocabulary set, the learning objective of the MLLMs, parameterized by 0, is given by
0 = arg min LMILM (T))0)
M
arg min
Vao logp(s; = -,I,T,
(I)
D
(1TT)D
where D the image-text dataset.
MLLMs generally comprise three modules: 1) a vision encoder, 2) a projector, and 3) a large language model (LLM). First, we generate visual inputs multiple scales from a given document image. The vision encoder extracts visual features for detailed image understanding. The resampler-based projector converts these visual features into compact set of query tokens for the language model. Unlike traditional MLLMs, which feed these query tokens directly into the LLM, we introduce the Hicrarchical Visual Feature Aggregation module to reduce the number of visual query tokens while preserving local information. Finally, the LLM processes both the visual and instruction tokens to generate a response autoregressively. Figure iillustrates the overall framework.
3.2
Multi-Scale Visual Features
We focus on developing a document understanding framework that incorporates multi-scale visual inputs to address various image scales for MILMs. However, using pretrained MILMs constrains the input resolution to a small, square-shaped configuration defined by the visual encoder. This limitation hampers capturing detailed information in high-resolution images and leads to spatial distortions when handling document images of diverse resolutions and aspect ratios.
We adopt shape-adaptive cropping (SAC) [19 to generate multiple sub-images, allowing model to handle varying aspect ratios and resolutions. We split each image using a predefined grid {g = (ng, x Rw) |m,, The N), where ng and nww respectively denote the number of rows and columns for the grid g. Then, each sub-image is resized to fit the visual encoder. Note that the optimal grid g" (m,, n for each image selected by metrics; resolution coherence, and shape similarity. Refer to Appendix C for more details.
Although SAC handles diverse aspect ratios and high-resolution images, it may miss local details within each sub-image due to its receptive fields fixed-size. To address this issue, we consider an additional level of detail by applying 2x upscaling each sub-image to capture detailed visual features and small text fonts through enlarged local resolution. To be specific, this augmentation process involves subdividing each sub-image into half along each spatial dimension, resulting in 2ng x 2n sub-images. We again resize the subdivided images to ensure compatibility with the visual encoder. By incorporating these high- resolution representations, the model now observes the input image in three different levels: global, n2, X Yews and 2n, x 2n, scales.
Unlike vision encoders and resampler-based projectors, which process sub-images in batches and thereby exhibit linear computational complexity. LLMS demonstrate quadratic complexity with respect to the number of sub-images. Thus, when the number of visual inputs increases from to 5n, the computational burden LLMs escalates 25 times, while that of vision encoders and resamplers grows 5 times. Because LLMs are represented by networks with significantly more channels than other models, this quadratic increase of computational cost is problematic many practice scenarios.
3.3 Hierarchical Visual Feature Aggregation
The HVFA module leverages a feature pyramid to fuse multi-scale features for LLMSs and reduces computational overhead. We first construct feature pyramid using the visual features from two adjacent scales, 8th and (i 1)st scales. Cross-attentive pooling is then applied to features at scale (i (i+1) 1) to compress and preserve detailed information, and the pooled features are aggregated with low-resolution features at scale i. Figure depicts the mechanism of the HVFA module.
Let F; E IR x xexe be the visual features for the 7th scale extracted from resampler, where (nf, x ney)) is the number sub-images at 4h scale, g denotes the number of queries from resampler, and c is the number of channels. We perform 2 2 max-pooling F E 134 xnft xaxe to

---

stopgradient
MSE Loss
F
Cross Attention
Reconstruction Layer
Pooled Q
Unpooled K,
Cross Attention
Max Pooling Layer
F.C
Fi+
F E R
Fist E R wen Angixgge
Figure 2: Illustration of the Hierarchical Visual Feature Aggregation (HVFA) module. (Left) HVFA aggregates high-resolution visual features low-resolution features leveraging feature pyramid structure. (Right) In cross-attentive pooling, each sub-image-feature attends to all of the fine-grained visual features, compressing and preserving more detailed information.
obtain a pooled feature F- E Rd x xixe, To reduce the information loss by the max max-pooling operation, we conduct the following operations including lightweight cross-attention:
(F i+1
W query ) (Fi+1 i+1 WLey) )
Fi+1 F4411
Softmax
(Fi+) W value proj
(2)
Vd
F = F; F Fi+1,
(3)
where W query. W key, W value E Rexd and W pmj E Rrexcare learnable parameters, and d is the embedding dimension. Equation 33. pooled feature F corresponds to a query and the original feature, F.+1. works expression. We feed F; to LLMs to generate the language responses. Note that our method maintains the original complexity of LLMs, despite considering multiple scales in practice, because the feature pyramid structure aggregates them into the initial local scale of visual features generated by SAC.
key and value. The layer normalization omitted for simplicity of the
We assume that well-compressed features retain essential information necessary for the reconstruction of the original features. After obtaining the pooled features, based on this assumption, we introduce small decoder network, used only for training, to reconstruct the original features from the pooled features. The decoder trained with the mean squared error (MSE) loss between the reconstructed and original features, which is given by
LMSE (F (+1,
[r(((F+11) stopgrad(F:+1))
where r(.) is an MLP-based reconstruction network and stopgrad indicates stop-gradient opera- tion preventing the collapse in training. By the MSE loss, guide the model to retain important visual details while reducing feature dimensionality. The final objective function is given by
CFinal LMIIM ALMSE,
(5)
where À is the weight for the balance between two terms.
Our approach is similar to pooling- noosing-based methods such MVIT 41 and MViT-v2 42 which construct attention mechanisms utilizing pooled features as keys and values to build efficient ViT variants 43 However, unlike these methods, use pooled features queries to reduce query size rather than optimizing attention mechanisms themselves.
Our cross-attentins pooling can be seen as a form of resampler 13, 44 that reduces spatial dimen- sionality. In this interpretation, the pooled features, F act as the query tokens for learning. Unlike traditional query-based resamplers that require new set of trainable query vectors, our method uses pooled features, which are more effective for resampling than randomly initialized query vectors, as discussed further in Section 4.4
5

---

Table I: Examples for Relative Text-Position Prediction Task. Refer to Appendix | for the full list.
Task
Instruction Templates
Human: What's the first 30% of the image text? Al: (corresponding texts).
RPT (first)
Human: Identify words from the first 15% of the image text. Al: [conrsponding texts]-
Human: What are the words locuted between 10% and 55% of the text in the image? Al: (corresponding texts).
RPT (middle)
Human: words found between 20%
in the image text. (corresponding texts].
Human: Identify words from the 16% of the image text. Al: (corresponding texts).
RPT (last)
Human: Which words make up the last 40% of the text in the image? Al: (corresponding texts).
Human: Specify the relative position within the image where (query texts] is found. Al: 15%-30%%
PTP
Human: Where is the text (query texts] located within the image? Al: 40%-80%.
3.4
Relative Text-Position Prediction Task
Another key challenge in document understanding is learning to read text using layout information. Although [19] trains models read entire document text, this approach entails significant compu- tational challenges and information losses due to text truncation caused by the input capacity of LLMs. To address these limitations, we introduce two novel tasks: Reading Partial Text (RPT) and Predicting Text Position (PTP). RPT focuses on reading specific text segments at given positions, while PTP aims predict the positions of given text segments. These tasks facilitate enhanced layout understanding with reduced computational cost.
access full document text, represented by sequence of tokens and adopt standard reading order of text (top-to-bottee bottom, left-to-right). For the RPT task, we randomly select one of three types: 1) "first": reading up to Pend% of the text, 2) "middle": reading from Pstort % to Pena% of the text, and 3) "last": reading from Pstar: % to the end. We also sample an instruction template for the selected type, where the instruction template examples presented in Table I and the full list is available in Appendix]] ! text truncation, which is given by
Suppose that
We then compute the maximum text coverage, Cmax, to mitigate
Canax min ml),
(6)
where is the length of the tokenized text and mmxx is the maximum sequence length acceptable by LLMs.
We set a threshold for the minimum coverage as a hyperparameter, Cmins to ensure that sufficient text is read. The range for reading, denoted by franges is given by
Cmax
if Cmax I Cmin otherwise,
Frange
raandom (Cmin Cmax)
where random(a, b) denotes uniform sampling from interval (a, b). Then, the positions to predict are expressed as follows:
(0, trange)
if type 'first'
(Pstact: Pend)
(P'san Pettn + taagg (100- 'range: 100)
else if type 'middle'
(8)
else if type 'last'.
where P'start = random(O, 100 - .range))
For the PTP task, given segment of text, the model aims to infer its relative position within the whole text. We determine text segments and their relative positions using the same process as in the RPT task. Through diverse training examples, the model learns to understand spatial relationships between text segments and other components in document.
Experiments
4.1 Datasets and Evaluation Metrics
We utilize document instruction dataset corpora for training, following [19. The training dataset includes various types of document images, including tables, charts, natural images, and web

---

Table 2: Configurations of -fee document understanding baselines.
Method
Model Params
# Trainable Params
Pretraining Data
# Fine-tuning data
Docaumend specific Pretraining
Dessurt 16
127M
127M
11M
Task- lask-speciffic
Donut
143M
143M
13M
Task specific
Pix2Stmucl
18
282M
282M
80M
Task specific
Pis2Struct/arge
18
1.3B
1.3B
80M
Task specifc
MLLM-based Instruction Taning
Qwen- VL
9.6B
207M
LM
Task lpsc-spciific
Monkey [32
9.813
207M
1.
1.4MM
BLIP-2-0PP-2270 + UReader [9]
3.8B
129M
650K
BLIP-2-OPT-2.7B Ours
3.8B
14M
129M
650K
mPLUG-OWl-7B- + UReader 19
7.2B
86M
I.IB
650K
mPPUU-OWW-7BB Ours
7.2B
96M
1.IB
650K
Table 3: Performance comparison with other OCR-free document understanding baselines. The bold-faced numbers indicate the best performance in each column.
Method
DocVQA InfoVQA DeepForm KI.C WTQ TahFaatt ChariQA VisualMRC TextVQA TextCaps
Document- specific Pretraining
Dessurt
63.2
Donut 17
67.5
11.6
61.6
30.0
18.8
54.6
41.8
93.91
43.5
74.4
Pix2Stricgoas
K
72.1
38.2
56.0
88.0
Pix2Struct Lange
76.6
40.0
58.6
95.5
MI..M based Instruction Tioning
Qwen-Vi VL (88
65.1
35.4
4.1
15.9
21.6
65.7
63.8
Monkey 32
36.1
40.6
32.8
25.3
65.1
67.6
BLIP-2-OPT-2.7B UReader [9]
22.9
3.6
18.3
17.4
58.5
37.1
214.3
43.2
126.3
BLIP-2-OPT-2.7B Ours
51.4
29.6
14.6
23.8
21.2
59.9
50.4
228.7
57.3
135.2
mPP.UGG-OW UReeder
19
65.4
42.2
49.5
32.8
29.4
67.6
59.3
221.7
57.6
118.4
mPLUUG-OW1-7B Ours
45.9
53.0
34.5
68.2
63.3
226.4
59.2
123.1
page screenshots, The datasets used are Doc 45
Infographics [46 DeepForm 47
KleisterCharity [48], WikiTaableQuestions [49], TabFact [50]. ChartQA [51], VisualMRC
[22
TextVQA [53], and TextCaps [54]. The combined dataset comprises approximately 650K image-
instruction pairs. For more details about the datasets, please refer to Appendix
We evaluate our model on the test splits of each dataset. For VoA and InfoVQA, employ the
ANLS score (55) while we adopt the FI score for DeepForm and KI.C The performance on WTQ. TabFact, and TextVQA is measured using accuracy, whereas ChartQA utilizes relaxed accuracy [56].
The metric for TextCaps and VisualMRC is the CIDEr score [57]. Evaluations for TextVQA and TextCaps are conducted on their respective official challenge websites.
4.2 Implementation Details
We conduct experiments on two pretrained resampler-based MLLMs: BLIP-2-OPT-2.7B and mPLUG-OWl-7B 25 We utilize LoRA (58] to fine-tune the language model efficiently while keeping the vision encoder frozen. For the BLIP-2-basee model, we freeze the resampler and apply LoRA, whereas for the
model, we train the resampler to ensure a fair comparison with [19]. Regarding the SAC, we set the maximum number of sub-images to 9 the BLIP-2-based model and 20 for the mPLUG-OWW--aasee model. Refer to Appendix D for more details.
4.3 Quantitative Results
We compare our framework to other OCR-free baselines, including document-specific pretraining approaches [16 18 for completeness. Given that previous methods utilize varying numbers of model parameters, trainable parameters, and training data, we provide the detailed configurations in Table 2 It is important to note that our comparisons are made with OCR-free methods based on comparable MILMs in terms model size and dataset scale. Table presents the overall results of the proposed algorithm across 10 document understanding benchmarks. When integrated with both BL.IP-2 and mPL.G-OWl, our approach consistently outperforms UReader [19] by a significant margin across all benchmarks. Among MLLM-based methods, framework outperforms Qwen-
7

---

Table Results of main ablation studies with the BLIP-2-based model. Note that the reconstruction loss (Recon) only comes with MS + HVFA.
MS HVBA Recon RTPP DocVQA
InfoVQA
DeepForm
KLC.
WTQ
TahFact CharQA
VisuaMMRC
TexVQQ
TextCaps
35.17
20.34
4.14
18.50
15.36
53.37
31.76
201.39
43.20
126.34
45.22
25.87
8.23
19.82
18.14
58.28
44.16
221.38
48.56
129.98
40.18
25.19
6.49
17.80
56.85
38.88
219.74
47.54
128.39
47.36
27.37
10.92
19.17
18.24
58.
46.70
222.56
52.65
130.86
50.46
28.48
13.18
23.82
20.06
59.83
49.48
226.65
56.04
134.06
V
v
$1.38
29.60
14.58
23.78
21.15
39.87
50.41
228.65
57.32
135.24
Table 5: Ablation study on the variations of hierarchical visual feature aggregation techniques. We employ BLIP-2-based model for experiments.
Variations
DecVQA Info DeepForm KLC WTQ TabFact CharQQ VisualMRC Tex TextCap
(a) Spatial Dimension Reduction
Max Pooling-only
43.23
25.44
7.88
19.32
18.14
58.82
44.36
222.49
45,91
126.34
Lincar Projectors
46.11
26.44
10.62
21.32
18.74
58.52
46.62
224.74
48.27
129.68
Cross-Attentie Pooling (Ours)
51.38
29.60
14.58
23.78
21.15
59.87
50.41
228.6
57.32
135.24
Cross-Local- Attentive Pooling
50.32
28.60
14.38
23.56
22.15
59.66
50.22
228.82
56.78
134.34
(b) Query Token Initialization
Random Vectors
48.07
27.34
11.40
21.82
19.85
58.42
48.01
223.37
53.48
130.22
Max-Pooled Features (Ours)
51.38
29.60
14.58
23.78
21.15
59.87
50.41
228.65
57.32
135.24
(c) Stop Gradient
w/o stopgrad
49.65
26.80
19.73
19.14
59.11
48.85
223.42
55.24
128.44
w/ stopgrad (Ours)
$1.38
29.60
14,58
23.78
21.15
59.87
50.41
228.65
$7.32
135.24
(d) Reconstracttion Lews
1.0
49.90
28.51
14.41
22.75
20.13
59.03
49.76
225.51
56.79
133.03
0.1 (Ours)
51.38
29.60
14.58
23.78
21.15
59.87
50.41
228.65
57.32
135.24
= 0.01
51.84
28. 14
13.84
23.87
20.39
59.54
50.05
227.57
57.03
135.17
20.06
56.04
in most benchmarks, despite using a smaller set pretraining data and smaller model. The results show that our model successfully transfer the knowledge of pretrained MLLMs to understand document samples.
VL-based methods 28
32
4.4 Ablation Study
We perform several ablation studies with the BLIP-2-based model validate the effectiveness of each component of our approach.
Effeet of each component Based on the results in Table we observe several key findings. First, our multi-scale hierarchical visual feature aggregation (MS + HVFA) significantly improves performance, by leveraging the benefits of multi-scale features. Second, incorporating a reconstruction layer to learn how to reconstruct (Recon) the original features from the pooled features aids in the information compression process, in most cases. Finally, the Relative Text- Post-Position Prediction Task (RTPP) enhances overall performance by improving the model's text-reading ability.
Spatial dimensionality reduction methods Table 5la) compares different operations for reducing the spatial dimensions of finer-scale features within the HVFA module. Our cross-attentie pooling outperforms simple mux-pooling, which suffers from significant information loss due to its non- learnable nature. We also compare our method against a stack of linear projection layers with the same parameter count. The results demonstrate that Pross-attentive pooling achieves superior performance by learning more effective feature relationships through its attention mechanism, enabling context- aware information aggregation. Additionally, investigate docal-attention-baat mechanism, where the cross-attention is restricted to each sub-image and its corresponding scaled-up features within the feature pyramid. Its slightly inferior results indicate that the pooling operation needs to aggregate information from various regions to effective.
cross-attention
Query token initialization strategy Our cross-attentive pooling can be regarded as a special form of resampler since it refines high-resolution features while reducing spatial dimensionality. In this

---

Table Ablation study on text reading tasks, We conduct our ablation study with the BLIP-2-based model. The bold-faced numbers indicate the best performance in each column.
RFT
19
RPT
PTP
DoeVQA
Info
DeepForm
KLC
WTQ
TabFact
CharQA
VisualMRC
TeaVQA
TextCaps
47.36
27.37
10.92
19.17
18.24
56.14
43.88
222.56
52.65
130.86
49.23
16
12.58
21.29
19.42
58.46
46.05
224.39
55.78
133.55
50.62
29.49
12:55
22.88
20.51
59.28
48.41
226.55
57.19
134.44
50.46
28.44 28.44
14.88
22.55
20.88
59.12
48. 28
226.56
56.91
134.92
51.38
29.60
14.58
23.78
21.15
59.87
50.41
228.65
57.32
135.14
80
35
Avg. Score
Throughout (imgls)
RPT (Ours)
.... RFT
Truncated boxt (%6
1.4
30
70
Trunc Tuncated text 1090
1.0
Nog. Store
Aug. Scoree
2.66
62 92 Iwto TRU
0.2
30
0.0
5,+5g
5g HVFA
sequence length Figure 3: Performance analysis on visual and textual inputs. (Left) Impact of visual input scale on model performance. We compare four variants our model: with the first seale (S1), with the second scale (S2), with multiple scales (S1 S2). and with multiple scales with HVFA (S1 + S2 w/ HVFA, ours). (Right) Impact of truncated text the text reading task on model performance by varying sequence length capacity of LLM.
context, the pooled features, denoted as F' 1+ serve query tokens 13
44 for the learning process.
To explore this perspective, we compare our approach with randomly initialized queries of the same size F F As demonstrated in Table b)), using max-pooled features for query initialization yields superior results, even without tintroducing additional parameters for training.
Role of stop-gradient Table 3jc presents the role of the "stop-gradient" operation in Equation where the architectures and all hyperparameters remain unchanged. Empirically, LMSE drops to zero at an early iteration without the stop-gradient operation due to the convergence at a trivial solution; it leads to a significant degradation in performance.
balance the loss We tested several different values of determine the optimal balance for the final loss function, CFinal, in Equation (5). Results in 5a indicate that À 0.1 achieves the best performance. A value of x 0.01 also improves performance compared to À whereas X = 1.0 yields worse results than À 0 on some benchmarks. These findings suggest that, while LSE positively contributes by learning compressed features, excessively strong compression hampers the model's primary goal, extracting text-related features.
Impact
Effectiveness of relative text-position prediction task T illustrates the effectiveness of RPT compared to RFT, implying that concentrating on key text segments guided by their relative positions enhances document understanding. Notably, RFT may lead to information loss due to text truncation, given the limited capacity LLMs. Furthermore, PTP improves the performance of both RPT and RFT tasks, showing that predicting the relative positions of partial text strengthens the model's comprehension document layout and content. This highlights the importance of considering text position in document understanding tasks.
4.5 Complexity Analysis on Visual Input Seale
To analyze the trade-off between performance and computational cost, we compare variants of our model that process different scales of visual inputs: first scale (S) ), the second scale (S2), multiple scales (S) + S2), and multiple scales with HVFA (S₁ + S2 w/ HVFA, ours). Figure Left) presents the average benchmark scores and throughput for each model. The throughput measured on single NVIDIA Quadro RTX GPU using the largest possible batch size for each model. where the input size

---

OUTER SPACE
MEM
MEMO
How many stars our solar system?
What written letters?
Ours: UReader: planets
Ours: MEMO UReader: MEMO
1900
SUN
19.8%
Which meeting in December has the location not finalized ye1?
Q: What contributes to 98 8% of our system?
Ours: REGIONAL HEART COMMITTEE
Ours: Sun UReader: Sun
UReader:
Figure 4: Our method vs. UReader [9] on VOA
45 and Infographic VQA 44
of the visual encoder is 224 x 224. The results indicate that incorporating finer scale significantly improves performance, albeit with increased computational cost (S1 vs. S2). Additionally, combining features from multiple scales boosts performance over single-scale features, though at the expense of higher computational load (S1, S2 vs. S1 + S2). Finally, our method efficiently integrates finer-scale features into coarser scale, achieving balance between performance gains and computational demands (S) + S2 vs. S + S2 w/ HVFA).
4.6 Analysis on Text Reading Task
To demonstrate the effectiivnness and robustness of RPT against truncation, conducted experiments varying the maximum sequence length of LLMs during training. [iguree]Right presents the average benchmark scores and the proportion of truncated text data across the entire dataset. RPT consistently outperforms RFT in all settings, indicating that RPT is robust to the capacity constraints of LLMs. The design of RPT is helpful for mitigating truncation issues while RFT suffers from truncation of text reading data. Notably, with a maximum sequence length of 256, approximately 33.5% of the dataset consists of truncated texts, which is considerable amount. Consequently, model performance declines even further compared to training without any text-reading task, as represented by the red dotted line in Figure 3 Right) Our approach ensures reliable data quality while incorporating stochasticity and positional information, contributing to its effectiveness. These results imply that our partial text reading task can inspire further rescarch on knowledge distillation and the development of compact models for OCR-related tasks.
4.7 Visualization of Results
Figure 4 illustrates question-answer pairs from the DocVQA [5] and InfographicVQA 446 datasets for comparisons between our framework and UReader [19] While both models can read large text, model is more powerful in identifying small text than UReader. This highlights our model's strength in effectively handling text at multiple scales. Furthermore, our method delivers more accurate answers for questions requiring precise reading comprehension. For instance, to answer the question "Which meeting in December has the location not finalized yet?", the model needs to accurately read the text "To be announced" as demonstrated in the rightmost example in Figure 4
5 Conclusion
We present novel OCR-free document understanding framework that leverages a pretrained large- scale multi-modal foundation model. Our approach integrates multi-scale visual features to accom- modate diverse font sizes within document images. To manage the increasing costs for detailed visual inputs, we propose the Hierarchical Visual Feature Aggregation (HVFA) module with cross-attentive pooling, effectively balancing information retention and computational efficiency while adapting to varying document sizes. Moreover, our framework introduces a novel instruction tuning task to enhance the model's text-reading ability by incorporating positional information within images. Extensive experimentation demonstrates the efficacy of our framework, achieving strong performance across range of document understanding tasks.
10

---

Acknowledgement
This work was partly supported by Samsung SDS, and the Institute Information communications Technology Planning & Evaluation (IITP) grants No..S-2002--12200955 (No.202-00000959, (Part 2) Few-Shot Learning Causal Inference in Vision and Language for Decision Making, No.R5, No.RS-2021: 11211343, Artificial Intelligence Graduate School Program (Seoul National University), 12222668, Artificial Intelligence Innovation Hub], and the National Research Foundation of Korea (NRF) grant [No.R4 (No.RS-202) RS-2022-NN070856 Trustworthy Artificial Intelligence], funded by the Korean government (MSIT).
References
[1] Chen, Y.C.,, Li, I.., Yu, I., EI Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Learning universal image-text representations. In ECCV. (2020)
[2] Kim, W., Son, B., Kim, I: Vilt: Vision-and-languagee transformer without convolution or region supervision. In ICML. (2021)
[3] Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.: Align before fuse: Vision
and language representation learning with momentum distillation. In NeurIPS. (2021)
[4] Li, X., Yin, X., Li, C.,, Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F,, et al.: Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV. (2020)
[5] Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic represen- tations for vision-and-language tasks. In NeurIPS. (2019)
[6] Suhr, A., Zhou, S., Zhang, A., Zhang. I., Bai, H., Artzi, Y:: corpus for reasoning about natural language grounded in photographs. In ACL. (2019)
[7] Xie, N., Lai, F., Doran, D., Kadav, A.: Visual entailment: novel task for fine-grained image understanding. arXiv preprint 1rXiv:1901 190,006066 (2019)
[8] Chen, X., Fang. H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.L.: Microsoft coco captions: Data collection and evaluation server. arXiv preprint 100Kv:15040000 (2015)
[9] Wang, J., Yang, Z., Hu, X., Li, L.,, Lin, K., Gan, Z., Liu, Z., Liu, C., Wang, L.: GIT: A generative image-to-tert to-text transformer for vision and language. TMLR (2022)
[10] Yu, J., Wang, Z., Vasudevan, V., Yeung. L., Seyedhosseini, M., Wu, Y.: CoCa: Contrastive captioners are image-text foundation models. TMLR (2022)
[11] Lee, S., Kim, D., Han, B.: Cosmo: Content-style modulation for image retrieval with text feedback. In CVPR. (2021)
[12] Li, Li, D., Xiong, C., Hoi, S.: BLIP: Bootstrapping imgge pre-training for unified vision-language understanding and generation. In ICML. (2022)
[13] Li, J., Li, D., Savarese, S., Hoi, S.: BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML. (2023)
[14] Koh, J.Y., Salakhutdinov, R., Fried, D.: Grounding language models to images for multimodal inputs and outputs. In ICML. (2023)
[15] Yu, L.,, Poirson, P., Yang, S., Berg, A.C., Berg, T.I.: Modeling context in referring expressions. In ECCV. (2016)
[16] Davis, B., Morse, B., Price, B., Tensmeyer, C., Wigington. C., Morariu, V.: End-to-end document recognition and understanding with dessurt. In ECCV. (2022)
[17] Kim, G., Hong, T., Yim, M., Nam, J., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., Park, S.: Oer-free document understanding transformer. In ECCV. (2022)

---

[18] Lee, K., Joshi, M., Ture, I.R., Hu, H., Liu, F., Eisenschlos, J.M., Khandelwal, U.. Shaw, P., Chang, M. M.W., Toutanova, K.: Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In ICML. (2023)
[19] Ye, J., Hu, A., Xu, H., Ye, Q., Yan, M., Xu, G., Li, C., Tian, J., Qian, Q., Zhang, J., al.: UReader: universal ocr-free visually-situated language understanding with multimodal large language model. In EMNLP. (2023)
[20] Tan, H., Bansal, M.: Lxmert: Learning cross-modality encoder representations from transform- ers. In EMNLP. (2019)
[21] Gan, Z., Chen, Y.C., Li, L., Zhu, C., Cheng, Y., Liu, J.: Large-scale adversarial training for vision-and-language representation learning. In NeurIPS, (2020)
[22] Zhang, P., Li, X., Hu, X., Yang, J., Zhang. L.,, Wang. I.., Choi, Y., Gao, J.: VinvI: Revisiting visual representations in vision-language models. In CVPR. (2021)
[23] Huang, Z., Zeng, Z., Huang, Y., Liu, B., Fu, D., Fu, J.: Sceing out of the box: End-to-end pre-training for vision-language representation learning. In CVPR. (2021)
[24] Park, J., Han, B.: Multi-modal representation learning with text-driven soft masks. In CVPR. (2023)
[25] Ye, Q. Xu, H., Xu, G., Ye, J., Yan, M.. Zhou, Y., Wang. J., Hu, A., Shi, P., Shi, Y., Jiang, C., Li, C., Xu, Y., Chen, H., Tian, J., Qi, Q., Zhang, J., Huang, F:: mPLUG-OWl: modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14 14178 (2023)
[26] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al.: OPT: open pre-trained transformer language models. arXiv preprint arXiv:2205.0100 2000006 (2022)
[27] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: LLaMA: open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023)
[28] Bai, J., Bai, S., Yang, S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, Qwen- VL: A versatile vasion-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. 1966 (2023)
[29] Dai, W. Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S.: Instructblip: Towards general- vision-language models with instruction tuning. In NeurIPS. (2023)
[30] Liu, H., Li, C., Wu, Q., Lee, YJ..: Visual instruction tuning. In NeurlPS, (2023)
[31] Liu, Y., Li, Z., Li, H., Yu, W., Huang, M., Peng, D., Liu, M., Chen, M., Li, C., Jin, L., et al.: On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895 (2023)
[32] Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang. J., Sun, Y., Liu, Y., Bai, X.: Monkey: Image resolution and text label important things for large multi-modal models. arXiv preprint arXiv:2311 1.06607 (2023)
[33] Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: LayoutLM: Pre-training of text and layout for document image understanding. KDD. (2020)
[34] Xu, Y., Xu, Y., Lv, T., Cui, L., Wei, F., Wang, G., Lu, Y., Florencio, D., Zhang, C., Che, W., al.: Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. In ACL. (2021)
[35] Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: LayoutLMv3; Pre-training for document ai with unified text and image masking. In ACM. (2022)
[36] Li, C., Bi, B., Yan, M., Wang, W., Huang, S., Huang, F, Si, L.: Structural Structural pre-training for form understanding, In ACL. (2021)
12

---

[37] Appalaraju, S., Jasani, B., Kota, B.U., Xie, Y., Manmatha, R.: DocFormer: End-to-end transformer for document understanding. In ICCV. (2021)
[38] Tang, Z., Yang. Z., Wang, G., Fang, Y., Liu, Y., Zhu, C., Zeng, M., Zhang, C., Bansal, M.: Unifying vision, text, and layout for universal document processing. In CVPR. (2023)
[39] Liao, H., RoyChowdhury.. A., Li, W., Bansal, A., Zhang, Y., Tu, Z., Satzoda, R.K., Manmatha, R., Mahadevan, V:: DocTr: Document transformer for structured information extraction in documents. ICCV. (2023)
[40] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. (2019)
[41] Fan, H., Xiong, B., Mangalam, K., Li, Y., Yan, Z., Malik, J., Feichtenhofer, C.: Multiscale vision transformers. In ICCV. (2021)
[42] Li, Y., Wu, C.Y., Fan, H., Mangalam, K., Xiong, B., Malik, J., Feichtenhofer, C.: Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR. (2022)
[43] Dosovitskiy, A.. Beyer, L., Kolesnikov, A.,, Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition scale. In ICLR. (2021)
[44] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.; End-to-end object detection with transformers. In ECCV. (2020)
[45] Mathew, M., Karatzas, D., Jawahar, C.: DocVQA: A dataset for vqa document images. In WACV. (2021)
[46] Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., Jawahar, C.: Infographic VQA. In WACV, (2022)
[47] Svetlichnaya, S.: DeepForm: Understand structured documents at scale (2020)
[48] Stanislawek, T., Gralinski, F., Wróblewska, A., Lipinski, D., Kaliska, A., Rosalska, P., Topolski, B., Biecek, P:: Kleister: key information extraction datasets involving long documents with complex layouts. In ICDAR. (2021)
[49] Pasupat, P., Liang, P.: Compositional semantic parsing on semi-structured tables. In ACL. (2015)
[50] Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S.. Zhou, X., Wang, W.Y.: TabFact: A large-scale dataset for table-based fact verification. In ICLR. (2020)
[51] Masry, A., Long, D.X., Tan, J.Q., Joty, S., Hoque, E.: ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In ACL. (2022)
[52] Tanaka, R., Nishida, K., Yoshida, S.: VisualMRC: Machine reading comprehension on docu- ment images. In AAAL (2021)
[53] Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., Rohrbach, M.: Towards vqa models that can read. In CVPR. (2019)
[54] Sidorov, O., Hu, R., Rohrbach, M., Singh, A.: TextCaps: dataset for image captioning with reading comprehension. In ECCV. (2020)
[55] Biten, A.F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E.,, Jawahar, C., Karatzas, D.: Scene text visual question answering. In ICCV, (2019)
[56] Methani, N., Ganguly, P., Khapra, M.M., Kumar, P:: PlotQA: reasoning over scientific plots. In WACV. (2020)
[57] Vedantam, R., Lawrence Zitnick, C., Parikh, D.: CIDEr: consensus-band image description evaluation. In CVPR. (2015)
13

---

[58] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z.. Li, Y.. Wang. S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In ICLR. (2022)
[59] Borchmann, L., Pietruszka, M., Stanislawek, T., Jurkiewicz, D., Turski, M., Szyndler, K., Graliñíski, F:: DUE: End-to-end document understanding benchmark. In NeurlPS Datasets and Benchmarks Track. (2021)
[60] Krasin, I., Duerig, T., Alldrin, N., Ferrari, V., Abu-EI-Haija, S., Kuznetsova, A., Rom, H., Uijlings, J., Popov, S., Kamali, S., Malloci, M., Pont-Tuset, J., Veit, A., Belongie, S., Gomes, V., Gupta, A., Sun, C., Chechik, G., Cai, D., Feng, Z., Narayanan, D., Murphy, K.: OpenImages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://storage.e
.tmll (2017)
[61] Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al.: How far we to gpt-4v? closing the gap to commercial multimodal models with open-source suites, arXiv preprint 2XXX:22444.168// (2024)
[62] Wu, J., Hu, X., Wang, Y., Pang. B., Soricut, R.: Omni-smola: Boosting generalist multimodal models with soft mixture of low-rank experts. In CVPR. (2024)
[63] Chen, X., Wang. X., Beyer, L., Kolesnikov, A., Wu, J., Voigtlaender, P., Mustafa, B., Goodman, S., Alabdulmohsin, I., Padlewski, P., al.: Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199 (2023)
[64] Hu, A., Xu, H., Ye, J., Yan, M., Zhang, L.,, Zhang, B., Li, C.,, Zhang, J., Jin, Q. Huang, F, al.: mplug-docow 1.S: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403. 12895 (2024)
[65] Ye, Q. Xu, H., Ye, Yan, M., Hu, A., Liu, H., Qian, Q. Zhang, J., Huang, F: mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. In CVPR. (2024)
[66] Carbune, V., Mansoor, H., Liu, F., Aralikatte, R., Baechler, G., Chen, J., Sharma, A.: Chart- based reasoning: Transferring capabilities from llms to vlms. arXiv preprint arXiv:2403.122 12599 (2024)
14

---

A Broader Impacts
Research on document understanding and large multimodal language models (MLLMs) can sig- nificantly impact various societal domains. In education, they enable personalized learning and greater accessibility to information, while in healthcare, they streamline medical documentation and enhance patient education. Legal and regulatory fields benefit from improved document review and compliance, and businesses gain through advanced customer service and financial analysis tools. However, addressing biases, ensuring privacy, and managing employment impacts are crucial ethical considerations. Moreover, vision system that can understand documents can be used in extracting pri- vacy information from documents easily. Balancing these advancements with societal responsibilities can maximize the benefits for a more informed and equitable society.
B Limitations
Our work has some drawbacks inherited from LLMs, since our work directly employs pretrained LLMs. First, our work that is vulnerable the bias in LLMs, which can cause misinterpreting the crucial and confidential documents. Next, given some document formats, the model may cause the privacy concerns. Our model has been trained only with the documents written in English and thus may struggle the documents written in other languages. While our work employs multi-scale visual features, we still constrains from using resampler-baset ML.MMs, and thus cannot fully exploit the characteristics of vision encoder, VIT. We believe that our method can directly be adopted to resampler-free pretrained models, since the proposed approach is simple.
C Shape-Adaptive Cropping
Shape-adaptive cropping (SAC) [9 is an image preprocessing method that generates multiple crops to handle varying aspect ratios and resolutions, enhancing versatility and performance. Given an image with dimensions H x W, we split the image with one of the predefined grids {g = (ng, X nww |mg, nww E NJ. where nh, nw denote the number of rows and columns of the grid g. Each image is then resized to (n,, Ho, W,)) to generate g sub-images, where H,, and W., are the spatial configuration defined by the pretrained vision encoder.
The suitable grid g" (n,, X m for each image is selected based on two criteria; resolution coherence, and shape similarity. It's essential for the grid to maintain the image's resolution to the greatest extent possible, and the grid should conform to the aspect ratio of the input image. Given the predefined set of grids, we denote the resolution-related and resolution-agnostc intersection over union, as Syy and Sya, respectively. Syr and Sra are defined as follows:
Spr (1,9) = IoU((H, W), (n Hn, nh, W)))
(9)
Syo (1, IoU(( Rey H
nww,, (n2, Rw))).
(10)
W
Then the optimal grid g" is selected by:
g = argmax, + (ra(1,)))
(1I)
D Implementation Details
We experiment with two pretrained resampler-based MLLMs: BLIP-2-OPT-2.7B 13 and mPLUG- Owl-7B (25). We utilize LoRA (58) to fine-tune the language model efficiently while keeping the vision encoder frozen. In the case of the resampler, we freeze it and apply LoRA for the BLIP-2-based model, while training it for mPLUG-Owl ensure fair comparison to UReader [19] For LoRA, we set rank r = 8, and = 32. The maximum sequence length of the LLM is set to 2048 for all experiments, except for the one described in Section 4.6. Regarding the shape-adaptive cropping module, we set the maximum number of sub-images to 9 for the BLIP-2-based model and 20 for the mPLUG-OOl-baasee model. We employ the transformers library from huggingfaed! to constuct
15

---

BLIP-2-bu b-seded model. For mPLUG-Owl, we implement our algorithm based on the official repository of UReader
For the Hierarchical Visual Feature Aggregation (HVFA) module, we employed a two-layer multi- head cross-attention layer with d 256 and 12 heads. For LMSE, we utilized two-layer MIP for the decoder layer and set À = 0.1. Regarding the minimum coverage of the Relative Text-Position Prediction Task (RTPP), we the minimum coverage Cmin to 30% to ensure that the selected region is not too small.
We trained our model with learning rate of 10 for 10 epochs, incorporating linear warmup of 50 steps, followed by cosine decay to O. The total batch size was set to 256, and we conducted training on 8 A100 GPUs. The entire training time is approximately 2 days for the BLIP-2-based model and 5 days for the mPL.UG-OW--base model.
E
Datasets
For all of the benchmarks, we used the train/val/split set provided by UReader 19 which is basically built on DUE benchmark [59].
DoeVQA Doc [45 consists of 50,000 question and answer (QA) pairs derived from 12,000 document images sourced from the UCSF Industry Documents Library. It comprises 40,000 training questions, 5,000 validation questions, and 5,000 test questions. The evaluation metric utilized is ANLS (Average Normalized Levenshtein Similarity), which measures similarity based on edit distance.
InforgraphicsvQA The Infographic VQA dataset [46 comprises a total of 30,035 questions and 5,485 images sourced from 2,594 distinct web domains. The data is randomly split into 23,946 questions and 4,406 images for training, 2,801 questions and 500 images for validation, and 3,288 questions and 579 images for testing. These questions demand approaches that integrate reasoning across document layout, textual content, graphical elements, and data visualizations simultaneously. We use ANLS for the evaluation metric.
serves as an information extraction dataset comprising 1100 socially significant documents pertaining to election spending. The objective entails extracting contract numbers, advertiser names, payment amounts, and air dates from advertising disclosure forms submitted to the Federal Communications Commission. The benchmark is evaluated by F1 score.
DeepForm DeepForm [7]
KleisterCharity KleisterCharity (KI.) 48 represents another information extraction dataset, featuring 2,700 documents sourced from published reports of charity organizations. It tackles significant challenges, including high variability in layout (due to the absence of templates), the necessity of performing Optical Character Recognition (OCR), the presence of lengthy documents, and the existence of various spatial features such as tables, lists, and titles. The evaluation metric utilized is F1 score.
WikiTable (WTQ) comprises 2.1k table images from Wikipedia and is annotated with 23k question and answer pairs demanding comparison and arithmetic operations. The task necessitates deeper compositionality. leading to a combinatorial explosion within the logical forms space. We use accuracy for the evaluation metric.
WikiTTalleQuestoons
TabFact TabFact [50] is Natural Language Inference dataset comprising 112k statements labeled as 'entailed' or 'refuted' based on information from 16k Wikipedia tables. The dataset is divided into 92.283 statements for training, 12,792 for validation, and 12.779 for testing. The evaluation metric utilized is accuracy.
ChartQA ChartQA 51 is designed to assist users by taking chart and a natural language question as input to predict the answer. The dataset comprises a total of 21,000 chart images and 32,000 QA pairs, divided into 28,200 for training, 1,920 for validation, and 2,500 for testing. We use relaxed accuracy for the evaluation metric.
16

---

Table Results of the ablation studies on SEL-UG--Ol--rae model. Note that the reconstruction loss (Recon) only comes with MS + HVFA.
MS+HVFA Recon RFT
19 RPTPTE DocVQA InfoVQA DeepForm KLCC WTQQ TahFact ChariQA VsualMMRCTexxVV) TextCaps
67.64
43.42
50.34
34.45 49
65.75
60.77
211.03
56.47
119.65
68.14
34.58 29.02
59.25
120.55
71.33
45.21
52.42
36.88 34.14
68.02
62.90
224.34
58.73
122.32
71.77
45.92
51.89
36.37 33.71
67.86
62.35
225.60
59.16
122.79
72.68
53.02
36.73
34.51
68.19
63.28
226.43
59.22
123.11
Table 8: Performance of the BLIP-2-based model with varying scales.
# of scales Throughput (img/s) DocVQA InfoVQA DeepForm
KLC
WTQ
TabFact ChurQQA VisualMRC
TexVQQ
TextCups
1.481
40.18
25.19
6.49
18.97
17.80
56.85
38. 88
219.74
47.54
128.
1.088
51.38
29.60
14.58
23.78
21.15
59.87
50.41
228.65
57.32
135.24
0.494
52.78
31,25
20.59
25,42
23.21
60.12
52.21
230.22
60.22
137.57
VisualMRC VisualMRC [52] consists of 5,000 full screenshots of webpages from 35 websites, with 30,000 annotated QA pairs. The answers, averaging 9.53 words, are expressed in fluent sentences and are notably longer than those other QA datasets. The dataset emphasizes natural language understanding and generation, sourcing its questions and abstractive answers from diverse range of webpage domains. We use CIDEr score for the evaluation metric.
TextVQA Tex(VQA [53] dataset designed for evaluating models' abilities to answer questions about images where the answers are textually present within the images. It consists of images from the OpenImagesV3 [60] dataset paired with questions and answers. The dataset comprises approximately 45,336 images, 28,408 training questions, 3,000 validation questions, and 7,684 test questions. We employ accuracy for the evaluation metric.
TextCaps TexICaps [54] dataset is designed for the task of image captioning with a focus on reading and understanding text present in images. It consists of images sourced from the OpenImages dataset, annotated with captions that require the model recognize and interpret text within the image. The dataset contains over 145,000 images and 230,000 captions. TextCaps challenges models integrate visual and textual information to generate accurate and informative captions that include text found in the image. We employ the CIDEr score as our evaluation metric.
F Additional Ablation Study Results
To validate the effect of each component more robustly, we conduct the additional ablation study for the mPLUG-OWl-based model. We focused on components that might be considered most marginal: the reconstruction process and Relative Text-Position Prediction (RTPP). Note that we compare RPT to RFT for RTPP experiments. Table 7 shows that the reconstruction loss and RPTT consistently improve performance most tasks with the large backbone model, mPLUG-OWl.
G Analysis on using more Scales
Our proposed method inherently flexible, permitting extension accommodate an arbitrary number of scales. For example, a hierarchical configuration utilizing three scales can be implemented by initially integrating features from scales 3 and 2, followed by merging these with features from scale 1. Table prrsee]s the performance across various benchmarks for differing numbers of scales. This experiment employed the BLIP-2-based variant of our model. The findings reveal that the inclusion of third scale continues to enhance document comprehension. albeit with diminishing returns. This partly because, as the model has already captured most of the key information from the initial two scales, the additional benefit is reduced. Note that training the model with all three scales without our hierarchical visual feature aggregation module is not feasible on 100 GPUs due to memory constraints.
17

---

Table Comparison with task-specific state-of tet-thh-rrt arts methods.
of
DoAVQA
InfoVQA
DeepForm
KLC
WTQ
223
75.7
68.8
38.7
40.6
SoTA
(IntemVV.
(IntemVI..1.5 P.a.y.-..PPPusSMPP..... (loS)( DocOwl (.5)m DecOwl 1.5) DocOwl
Ours
12.7
45.9
53.0
36.7
34,3
TabFact
ChartQ 81.3
VisuaIMRC
TexVQQ
TexiCaps
80.4
364.2
82.2
164.3
SoTA (mPLUG-DocOwl 1.5)
(ChartPaLI) 63.3
(LayourT5)
(Omni-SMoLA) 59.2
(PaLI-3)
Ours
68.2
226.4
123.1
H Comparison with Task-specific SoTA Methods
We list the performance of task-specific state-of of--hh--rttt art models at the moment of submission in Table The state-of-the-art models can be categorized into 3 categories.
Large Foundation Models These models leverage extensive pretraining datasets and large model parameters. For DocVQA and InfographicsVOA. VLtem-L--1.5PPur [6]] ranks highest the leaderboard. This model has 26B parameters and is pretrained on 26 dataset corpora before being fine-tuned additional 49 corpora. For TextVQA and TextCaps, variants of PaLI [2] 63 hold the top rank. PaLI, which is trained on the 10B-WebLI dataset, is significantly stronger foundation model than backbone, mPLUG-OWl, which only uses 1B dataset for pretaining. While PaLI-3 633 has 5B parameters, it still benefits from the 10B-WebLI dataset for pretaining. Moreover, direct comparison becomes more challenging when considering the differences in the unimodal encoders of each model. Note that we have compared our model only with OCR-free methods based on comparable MLLMs in terms model size and dataset scale.
Methods Using OCR Engines For VisualMRC, current OCR-free models lag behind Lay- outT5 52 which utilizes external OCR engines. This is likely because VisualMRC data is text-rich and features long documents, indicating significant room for improvement. While LayourT5 has relatively small model size of 770M parameters compared recent MLLMs, incorporating OCR engines in such task provides a substantial advantage, particularly in accurately processing and understanding extensive textual information.
Fine-Tuning Tuning Foundation Models with Document-Specific Data and Methods Our work falls into this category. For DeepForm, KLC, WTQ, and TabFact, the current SoTA model is mPLUG- DocOwl 1.5 [64], which is concurrent work ours. mPLUG-DocOwi 1.5 is based on mPLUG-Owl 665. a direct extension of our backbone, mPLUG-OWl. While models are similar size, mPLUG-OW! 2 is known to perform significantly better than mPLUG-OW Additionally, mPLUG- DocOwl 1.5 is fine-tuned on 4M document datasets, which is several times more than our 650K dataset. For ChartQA, where ChartPaLI (66) is the state-of-the-arts, instruction data specifically designed for chart data is used, and the model is based on PaLI, pretrained with the 1OB-WebLI dataset.
I Instruction Templates for Relative Text-Position Prediction Task
Table 10 illlusstssess the full list of the templates for Relative Text-Position Prediction Task.
18

---

Table 10: Examples for Relative Text-Position Prediction Task
Task
Instruction Templates
Human: What's in the first | 1% of the image text? Al: (oorresponding texts),
Human: Identify words from the first { 1% the image
Al: (corresponding texts).
Human: Which words make up the first ( 1% of the text in the image? AI: (corresponding texts).
Human: List words in the image text's initial 1% segment. Al: [corresponding texts). J.
Human: Extract words found
image text's opening { 1%.. (corresponding texts).
RPT (first)
Human: Words in the initial | 1% of the imuge text? Al: [corresponding texts).
Human: Which words comprise the image text's first { 166 Al: ccorresponding texts).
Human: Identify the words positioned in the image text's initial 1% Al: (corresoonding texts).
Human: List the words found within the first 1% of the image text. Al; [correspondng texts).
Human: What words are situated within the initial 1% segment of the image text? Al: (coresponding texts).
Human: What are the words locuted between 1% and 1% of the text the image? Al: (comresooniig texts).
Human: List words found between 1% and 1% in the image text. Al: [comesponding texts].
Human: Which words fall between [ 15 and 15 in the image text? Al: [corresponding texts].
Human: What words are in the [5--1 range
image
Al: (corresponding (.
Human: Identify words from { |%81| 1% the image text? AI: [corresponding texts).
RPT (middle)
Human: What are the words locuted at ] 18-[ 1% in the image text? Al: [corresponding texts),
Human: Can you extract words from ]%-[[
in the image text? AI: (eorresponding texts).
Human: What's within the |56-| |% runge in the image text? AL: (corresponding texts).
Human: Which words occupy 1 [6--1 1% of the image text? Al: [corresponding texts).
Human: What words lie between 1% and
the image text? Al: (corresponding texts].
Human: Identify words from the last 1%8 of the image text. Al: (corresponding texts).
Human: Which words make up the last 1% of the text in the image? Al: [corresponding texts).
Human: What's
1%
image text? Al: (corresponding texts].
Human: List words in the image text's ss final { 1% segment. Al: (corresponding texts). ).
Human: Extract words found in the image text's 's closing ( 15, Al: (corresponding texts).
RPT (last)
Human: Words in the final 1%
image text?
[comesponding texts).
Human: Which words comprise the image text's 's last |&? AE leoresponding texts). ).
Human: Identify the words positioned in the image text's final [ 1%, Al: |corresponding texts).
Human: List the words found within the last 1% of the image
AL:
texts).
Human: What words are situated within the final 1% segment of the image tex?? Al: (corresponding texts).
Human: Specify the relative position within the image where (query texts] is found. Al: 1%-4 1%.
Human: Where is the text [query texts] located within the image? AI: { }56-( 1%.
Humaa: Locate the relative position within the image where the text [query texts] situated. Al: I ]S-[ 18,
Human: Determine the relative position within the image where the words [query texts) appear. Al: [ ] JS.
Human: Identify the relative position within
image where the phrase (query texts] located.
]S-4 JS.
PTP
Human: Find the relative position within the image where (query texts) is depicted. Al: ( []%-[ 5%.
Human: Where within the image can we find the phrase [query lexts}? Al: ]S- 1S..
Human: what position in the image the words (query texts) appear? AE []6-( (]%,
Human: Where within the image is (query texts) depicted? Al: ]6-[ 1%.
Humaa: Where can we locate (query texts] within the image? Al: [ [96- 19.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of work is stated in the Section in supplementary file.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
We do not have any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Ys]]
Justification: Implementation details are provided in Section 4.2 in the main paper and Section 4 in the supplementary file.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [NA]
Justification: We used the data UReader [9] provided, which are publicly accessible in https://github. www.com////ooo. com//akeeoreverYYong UReader
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
//ipss.cc/pbllicguuiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification: Implementation details are provided in Section 4.2 in the main paper and Section 4 in the supplementary file.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Since our model consists of LLM, which has high computational cost, it is hard to conduct experiment several times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Implementation details are provided in Section 4.2 in the main paper and Section in the supplementary file.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ottll//ttt//////aaaa...////////a
Question:
Answer: [Yes]
Justification:
We do have any violation to the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the broader impacts of our work in section 1 in the supplementary
file.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Since we do not release our model, we do not have risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly cited the original papers for all datasets and codebases used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets for this submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have any experiments on human subjects nor crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "AF8vyC7K0d", "reviewer": "Reviewer_cWFY", "summary": "The paper presents a novel approach to OCR-free document understanding using pre-trained Multimodal Large Language Models (MLLMs). The approach uses multi-scale visual features to handle different font sizes within document images. To address the high computational cost associated with multi-scale visual inputs for pre-trained MLLMs, the authors propose a hierarchical visual feature aggregation module. This module reduces the number of input tokens to LLMs by employing a feature pyramid hierarchy with cross-attentive pooling, thereby balancing information loss and efficiency without being affected by varying document image sizes. In addition, the paper introduces an innovative instruction tuning task that incorporates text position information within images, improving model readability and robustness to text truncation. Extensive experiments demonstrate the effectiveness of the framework in achieving superior document understanding performance in various tasks.", "questions": "- RTP optimisation: The RTP method reads a random percentage of text using a common reading order. Would it be more efficient to read specific, complete sections of the document based on its structure, such as reading entire articles in a newspaper?\n- Text position encoding: Can you provide more details on how text position is encoded in the Positional Text Positional (PTP) method?\n- Table 3 Formatting: The best result is not highlighted in bold for the 1st and 3rd columns of Table 3. \n\nTypos and errors:\n- Line 169: \"stop-gradient\" should be corrected to \"stop-gradient\".\n- Line 189: \"Table 7\" should probably be \"Table 1\".\n- Line 243: RTPP is introduced but never explained.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper provides a detailed description of model sizes, trainable parameters and pre-training data size. This information is critical for real-world application of these models where size and efficiency are critical factors.\n- The authors carry out a very detailed ablation study. For some aspects of the study, they ensured that the number of parameters remained constant, minimising bias and allowing a clearer understanding of the impact of each component.\n- The paper introduces a novel hierarchical visual feature aggregation module and a new instruction tuning task.", "weaknesses": "- Complexity vs. gain: The use of reconstruction loss increases complexity for what appears to be a small performance gain, as shown in Table 5 (lambda=0).\n- RTP Usage: While the Random Text Positional (RTP) approach is interesting for handling complex documents, it is only used during training. There is potential to optimise and use RTP during inference to further improve performance. See the questions.\n- Lack of open source availability: The models and code are not open source.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's acknowledgment of our introduction of novel components, thorough ablation studies, and detailed model information. Below are our responses to the main concerns.\n\n**[W1] Effectiveness of reconstruction loss**\n\nTable A. Effectiveness of reconstruction loss on both models. The bold-faced numbers indicate the best performance in each column for each backbone model.\n\n|  Backbone | Recon |   DocVQA  |  InfoVQA  |  DeepForm |    KLC    |    WTQ    |  TabFact  |  ChartQA  |  VisualMRC |  TextVQA  |  TextCaps  |\n|:---------:|:-----:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:----------:|:---------:|:----------:|\n| BLIP2-OPT |       |   50.46   |   28.48   |   13.18   | **23.82** |   20.06   |   59.83   |   49.48   |   226.65   |   56.04   |   134.06   |\n| BLIP2-OPT |   V   | **51.38** | **29.60** | **14.58** |   23.78   | **21.15** | **59.87** | **50.41** | **228.65** | **57.32** | **135.24** |\n| mPLUG-Owl |       |   71.33   |   45.21   |   52.42   | **36.88** |   34.14   |   68.02   |   62.90   |   224.34   |   58.73   |   122.32   |\n| mPLUG-Owl |   V   | **72.68** | **45.90** | **53.02** |   36.73   | **34.51** | **68.19** | **63.28** | **226.43** | **59.22** | **123.11** |\n\nThe reconstruction is performed only during training, so it incurs no additional cost for inference. Although the introduction of reconstruction loss increases training complexity, the extra cost is marginal, particularly compared to other components in our training strategies. More importantly, as shown in Table A from our ablation study, the reconstruction loss consistently improves performance on most tasks with both the moderate and large backbone models, BLIP2-OPT and mPLUG-Owl, respectively. We believe these consistent gains demonstrate the effectiveness of our reconstruction loss, especially given that the inference cost remains unchanged.\n\n\n**[W2 & Q1] Using Relative Text-Position Prediction (RTPP)**\n\nThis comment including the associated question is a bit confusing partly due to the wrong terminology. It would be appreciated if you revise the comments for better answers. Doing the best under the current context, we presume that \"Random Text Positional (RTP)\" means Relative Text-Position Prediction (RTPP) in our approach. Using RTPP during inference is an interesting idea, but it may not be straightforward because it may require additional modules for generating proper instruction tasks or increase inference costs significantly; it can be a direction for the extension of our paper. \n\n**[W3] Lack of open source availability**\n\nWe understand the concern about open-source availability. Due to the proprietary issue, we have to go through bureaucratic procedures to release the full source code. However, we will put our every effort into scientific transparency and reproducibility. We have already provided detailed implementation details in Section D in the Appendix. We will further provide extra details and ensure that our work is independently verified by the research community.\n\n**[Q2] More detail on Predicting Text Positional (PTP) task**\n\nThe PTP task follows the same process as the Reading Partial Text (RTP) task for generating the position pair ($p_{start}$, $p_{end}$) and its corresponding text segment. The key difference lies in how to use this information: \n1) RTP: The position pair is a part of the instruction, and the model predicts the text. \n2) PTP: This is opposite to RTP: the text segment is given as the instruction, and the model predicts the position pair. \n\nIn both cases, the position is represented by ($p_{start}$, $p_{end}$), indicating the start and end positions of the text segment within the document. We hope this clarifies the encoding process. We are happy to provide more details or examples in the revised manuscript if needed.\n\n**[Q3] Table 3 Formatting Issue**\n\nWe highlighted the best results among MLLM-based instruction tuning methods. We will clarify this in the revised version of our paper.\n\n**[Q4] Typos & Errors**\n\nThank you for pointing out our mistakes and we will revise our manuscript for better clarity. FYI, RTPP stands for Relative Text-Position Prediction as specified in line 243, and consists of RPT (Reading Partial Text) and PTP (Predicting Text Position).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V4UYnZvzd8", "reviewer": "Reviewer_CDZQ", "summary": "The paper analyzes the impact of features at different scales for document understanding. It proposes a method to combine multi-scale features without signifficantly increasing computational complexity. In addition the paper also proposes two new instruction tuning tasks that allow the model to better extract text information from the documents. Experiments are reported on a varied set of datasets and tasks showing", "questions": "See above in weaknesses", "limitations": "There is no specific discussion on the limitations of the method.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces multi-scale feature extraction for document understanding. This can help to analyze larger and high resolution documents, without signifficantly increasing the cost.\n- Two new instruction tuning tasks are defined to help the model learn how to read the text inside the document.\n- Experimental results show that combining the proposed approach with existing MLLMs obtain better results than other existing methods. A detailed ablation study shows the contribution of all the components of the framework.", "weaknesses": "- From a technical point of view there is not much novelty. Multi-scale feature integration has been largely been used in many different domains in a similar way.\n- Experimental results lack a better comparison with SoA models. The proposed method is compared with OCR-free models and similar approaches using MLLMs, but ignores other methods of the SoA (maybe using an external OCR) that can obtain better results. For instance, in the leadeboard of DocVQA and InfographicVQA (rrc.cvc.uab.es) there are methods with a better performance. For a fair analysis of the results, I think that these results should be included and discussed comparing and contextualizing with the results obtained by the proposed method. Also, for tasks that mainly work with natural images, such as TextVQA, it would be better to compare with specific methods for scene text VQA. Comparing with methods designed to work with document images like Donut may not be the best option.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for the reviewer's insightful summary of our work, highlighting the key contributions of our multi-scale feature extraction approach, new instruction tuning tasks, and comprehensive experimental results. Below are our responses to the main concerns. \n\n**[W1] Incremental novelty (multi-scale features)**\n\nThe proposed approach is the first practical solution for integrating multi-scale features in high-resolution multi-modal large language models (MLLMs), which was deemed impractical due to resource constraints. Our framework preserves the benefits of multi-scale features without substantial increases in computational costs. Specifically, we developed a hierarchical visual feature aggregation module that substantially reduces computational costs using cross-attentive pooling and feature pyramid hierarchy, making multi-scale integration computationally feasible for MLLMs. Unlike prior works relying on single-scale inputs, our method allows efficient multi-scale processing in MLLMs, significantly improving performance.\n\n**[W2] Justification on comparisons**\n\nWe appreciate the reviewer's comments but respectfully and partly disagree that our comparisons are lacking, due to the following reasons:\n\n Our research focuses on OCR-free multi-task learning with MLLMs for recognizing document images (e.g., DocVQA) and understanding natural images (e.g., TextVQA), which is different from task-specific learning and OCR-based learning. Direct comparisons with task-specific models would not fully capture this key aspect of our work. For instance, ChartPaLI [R1], the state-of-the-art for ChartQA, uses instruction data specifically designed for chart data; it doesn't align with our general-purpose approach.\n\nOn the other hand, the top performers on the referenced leaderboards, such as those for DocVQA and InfographicVQA, utilize significantly larger models, advanced language models with 20-55B parameters such as InternLM2 [R2] and PaLI-X [R3]. These are not directly comparable to our LLaMA-based mPLUG-Owl model due to significant differences in scale and computational costs. Moreover, the size and origin of the massive training data used for these large language models are often not fully disclosed, making direct comparisons potentially unfair. Hence, we compared our model only with OCR-free methods based on comparable MLLMs in terms of model size and dataset scale. \n\nWe included Donut as a baseline, a well-established OCR-free approach, to demonstrate the effectiveness of our approach; Donut has an advantage in document-related tasks because it is pretrained with document-specific data, but our method is still competitive without such a specialized pretaining technique.\n\n[R1] Carbune et al., Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs, arXiv 2024\n\n[R2] Cai et al., InternLM2 Technical Report, arXiv 2024\n\n[R3] Chen et al., PaLI-X: On Scaling up a Multilingual Vision and Language Model, arXiv 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sQipdIs0wF", "reviewer": "Reviewer_rxzX", "summary": "This paper presents hierarchical visual feature aggregation for OCR-free document understanding, leveraging feature pyramid hierarchy with cross-attentive pooling to handle the trade-off between information loss and efficiency. Additionally, a relative text-position prediction task is proposed to address the text truncation issue. Experiment results based on BLIP-2 and mPLUG-Owl demonstrate the effectiveness of the proposed method.", "questions": "1. In line 131, the description \"augmenting each grid by a 2 × 2 scale, effectively enlarging the receptive field size\" is confusing. How does this operation lead to an enlarged receptive field size?\n2. How are the transposition and matrix multiplication operations performed in Equation 2 given the 4-dimensional matrix F_{i+1} and F^’_{i+1}.", "limitations": "The authors did not validate their proposed methods on more advanced models such as mPLUG-DocOwl 1.5 and TextMonkey, so the generalizability of their methods remains open to discussion.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper innovatively utilizes a feature pyramid hierarchy to fuse multi-scale visual features without increasing the computational complexity of large language models. \n2. This paper presents a novel instruction tuning task that is robust to text truncation issues.\n3. Extensive experiments and ablation studies showcase the effectiveness of the proposed method on BLIP-2 and mPLUG-Owl models.", "weaknesses": "1. The comparison in Table 3 is not comprehensive enough; models such as mPLUG-DocOwl 1.5 and TextMonkey should be included.\n2. In the ablation experiments based on BLIP-2, the performance improvements brought by the reconstruction layer (Table 4) and the Relative Text-Position Prediction Task (Table 6, compared to Reading Full Text) are minimal. I wonder whether they can work for more advanced LMMs such as mPLUG-Owl.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank the reviewer for recognizing our key innovations in multi-scale feature fusion, novel instruction tuning, and comprehensive experimental results across different models. Below are our responses to the main concerns. \n\n**[W1] Comparison to the recent works**\n\nTable B: Comparison of different configurations over TextMonkey [R4] and mPLUG-Owl 1.5 [R5].\n\n\n| Method           | Backbone    | # Model Params | # Trainable Params | # Pretraining Data | # Fine-tuning data |\n|------------------|-------------|----------------|--------------------|--------------------|--------------------|\n| TextMonkey [R4]      | Qwen-VL [25]    | 9.7B           | 7.9B               | 1.4B+76.8M         | 2.1M               |\n| mPLUG-DocOwl 1.5 [R5] | mPLUG-Owl 2 [R6] | 8.1B           | 412M               | 400M               | 4M+25K             |\n| Ours             | mPLUG-Owl   | 7.2B           | 96M                | 1.1B               | 650K               |\n\nInitially, we compare the configurations with our framework and two other approaches, TextMonkey and mPLUG-Owl 1.5, represented in Table B. We would like to highlight two key differences between both models and ours. Both models use more advanced Multimodal Large Language Model (MLLM) backbones for initialization - Qwen-VL and mPLUG-Owl2. Our model, in contrast, is based on the mPLUG-Owl. Additionally, both models benefit from substantially larger training datasets and more trainable parameters, which directly impacts their performance.\n\nDespite these advantages, according to the reported results, our method surpasses TextMonkey. Although the mPLUG-DocOwl 1.5 model performs slightly better than ours, this is possible given its use of a more advanced backbone and larger dataset. For a fair comparison, the best option would be to augment both backbones with our framework and train the models with the same data, which is not feasible within a limited rebuttal period. However, we expect improvements in our results by incorporating our framework into the new backbones with more data, since the multi-scale aggregation module facilitates efficient capturing of different levels of detail. \n\n**[W2] Effect of the reconstruction loss and RTPP on mPLUG-Owl**\n\nTable C: Results of ablation studies on the reconstruction loss and RTPP. Note that we compare RPT to RFT for RTPP experiments. The last rows for each backbone model represent our final model.\n\n| Backbone  | MS+HVFA | Recon | RFT | RPT | PTP |   DocVQA  |  InfoVQA  |  DeepForm |    KLC    |    WTQ    |  TabFact  |  ChartQA  |  VisualMRC |  TextVQA  |  TextCaps  |\n|-----------|:-------:|:-----:|:---:|:---:|:---:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:----------:|:---------:|:----------:|\n| BLIP2-OPT |    ✓    |       |     |  ✓  |  ✓  |   50.46   |   28.48   |   13.18   | **23.82** |   20.06   |   59.83   |   49.48   |   226.65   |   56.04   |   134.06   |\n| BLIP2-OPT |    ✓    |   ✓   |  ✓  |     |  ✓  |   50.62   |   29.49   |   12.55   |   22.88   |   20.51   |   59.28   |   48.41   |   226.55   |   57.19   |   134.44   |\n| BLIP2-OPT |    ✓    |   ✓   |     |  ✓  |  ✓  | **51.38** | **29.60** | **14.58** |   23.78   | **21.15** | **59.87** | **50.41** | **228.65** | **57.32** | **135.24** |\n| mPLUG-Owl |    ✓    |       |     |  ✓  |  ✓  |   71.33   |   45.21   |   52.42   | **36.88** |   34.14   |   68.02   |   62.90   |   224.34   |   58.73   |   122.32   |\n| mPLUG-Owl |    ✓    |   ✓   |  ✓  |     |  ✓  |   71.77   | **45.92** |   51.89   |   36.37   |   33.71   |   67.86   |   62.35   |   225.60   |   59.16   |   122.79   |\n| mPLUG-Owl |    ✓    |   ✓   |     |  ✓  |  ✓  | **72.68** |   45.90   | **53.02** |   36.73   | **34.51** | **68.19** | **63.28** | **226.43** | **59.22** | **123.11** |\n\nIn response to concerns about the robustness of reconstruction loss and Relative Text-Position Prediction (RTPP) on the advanced backbone model, we conducted additional ablation studies on the mPLUG-Owl-based model, which are presented in Table C. Note that we compare RPT to RFT for RTPP experiments. The results demonstrate that both reconstruction loss and RTPP consistently improve performance on most tasks with both the moderate and large backbone models, BLIP2-OPT and mPLUG-Owl, respectively. We believe these consistent gains demonstrate the effectiveness of our components. \n\n**[Q1] Clarification on the receptive field**\n\nIn line 131, the description “effectively enlarging the receptive field size” should have been removed. We are sorry for the confusion. To clarify, augmenting each grid by a 2 × 2 scale does not enlarge the receptive field as originally stated. Instead, this operation provides a more detailed view of each grid area by effectively zooming in, allowing our model to capture finer-grained visual features and smaller text fonts within each cell. This increased resolution of information potentially leads to a better understanding of local details. We will revise this description to accurately reflect the purpose and effect of this operation.\n\n**[Q2] Clarification of Matrix multiplication**\n\nWe simply flatten both matrices to the shape of $(H_i \\times W_i \\times Q) \\times C$. We will revise the description to provide a clearer explanation.\n\n[R4] Liu et al., TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document, arXiv 2024\n\n[R5] Hu et al., mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding, arXiv 2024\n\n[R6] Ye et al., mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration, CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hJCLkjDy9e", "reviewer": "Reviewer_Pfd9", "summary": "This paper introduces an approach to enhancing document understanding capabilities by employing a hierarchical visual feature aggregation technique alongside pretrained Multimodal Large Language Models (MLLMs). The method utilizes a feature pyramid hierarchy integrated with cross-attentive pooling, which effectively captures multi-scale visual information. Additionally, the paper presents a new instruction task designed to further boost model performance. The experimental results on standard benchmarks demonstrate the effectiveness and superiority of the proposed method in advancing document understanding.", "questions": "Please see the concerns in Weaknesses.\n\nOther questions:\n\nWhether the \"Ours\" in Table 3 applied the relative text-position prediction task?", "limitations": "No.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper conducts extensive ablation studies to meticulously demonstrate the effectiveness of each proposed module and parameter. These studies provide a comprehensive understanding of the individual contributions and interactions within the framework, thereby validating the design choices and overall robustness of the system.\n\n2. This paper introduces a novel relative text-position prediction task, which significantly enhances document understanding. This task leverages the spatial relationships between text elements to improve the model's ability to interpret and contextualize information within documents.", "weaknesses": "1. The baseline for the ablation studies presented in this paper may be considered less robust, as evidenced by the performance on DocVQA, where it only achieves 35.17% accuracy, significantly below the 72.7% achieved by the proposed model. This discrepancy suggests that the baseline's performance could potentially undermine the persuasiveness of the conclusions drawn from the ablation study. It is possible that certain modules or techniques, which show effectiveness against this baseline, may not demonstrate the same level of impact when compared to a stronger baseline.\n\n2. While the multi-scale feature pyramid hierarchy is identified as the paper's primary contribution, the discussion surrounding this module could be more thorough. For instance, the paper could benefit from an exploration of alternative configurations, such as the implications of utilizing only two local scales versus three or four. A deeper analysis of these options would provide insights into the optimal balance between computational efficiency and performance, thereby enriching the understanding of the module's role in the overall framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's insightful recognition of our extensive ablation studies and the novel relative text-position prediction task, which validate our approach's robustness. Below are our responses to the main concerns. \n\n**[W1] Robustness of the ablation studies**\n\nAs 35.17% represents the performance of the moderate BLIP2-OPT backbone and 72.7% represents the performance of the larger mPLUG-Owl backbone on DocVQA, they are not directly comparable. To provide the proper baseline, we brought the numbers for the mPLUG-Owl baseline in Table D (row 5) from the ablation study of UReader paper [14]. To further address the concern about the robustness of our ablation studies, we have conducted additional experiments on the mPLUG-Owl-based model. Due to the limited time constraints in the rebuttal period, we focused on components that might be considered most marginal: the reconstruction process (row 2 vs row 4, row 6 vs row 8) and Relative Text-Position Prediction (RTPP) (row 3 vs row 4, row 7 vs row 8). Note that we compare RPT to RFT for RTPP experiments. As shown in Table D, the reconstruction loss and RPTT consistently improve performance on most tasks with both the moderate and large backbone models, BLIP2-OPT and mPLUG-Owl, respectively. We believe these consistent gains demonstrate the effectiveness of our components. We will provide the full ablation studies for the large model in the revised version.\n\nTable D: Results of ablation studies on the reconstruction loss and RTPP. Note that we compare RPT to RFT for RTPP experiments. The last row for each backbone model represents our final model.\n\n| Backbone  | MS+HVFA | Recon | RFT | RPT | PTP |   DocVQA  |  InfoVQA  |  DeepForm |    KLC    |    WTQ    |  TabFact  |  ChartQA  |  VisualMRC |  TextVQA  |  TextCaps  |\n|-----------|:-------:|:-----:|:---:|:---:|:---:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:----------:|:---------:|:----------:|\n| BLIP2-OPT |         |       |     |     |     |   35.17   |   20.34   |    4.14   |   18.50   |   15.36   |   53.37   |   31.76   |   201.39   |   43.20   |   126.34   |\n| BLIP2-OPT |    ✓    |       |     |  ✓  |  ✓  |   50.46   |   28.48   |   13.18   | **23.82** |   20.06   |   59.83   |   49.48   |   226.65   |   56.04   |   134.06   |\n| BLIP2-OPT |    ✓    |   ✓   |  ✓  |     |  ✓  |   50.62   |   29.49   |   12.55   |   22.88   |   20.51   |   59.28   |   48.41   |   226.55   |   57.19   |   134.44   |\n| BLIP2-OPT |    ✓    |   ✓   |     |  ✓  |  ✓  | **51.38** | **29.60** | **14.58** |   23.78   | **21.15** | **59.87** | **50.41** | **228.65** | **57.32** | **135.24** |\n| mPLUG-Owl |         |       |     |     |     |    56.7   |    ---    |    ---    |    ---    |    22.9   |    ---    |    56.7   |    205.0   |    54.3   |     ---    |\n| mPLUG-Owl |    ✓    |       |     |  ✓  |  ✓  |   71.33   |   45.21   |   52.42   | **36.88** |   34.14   |   68.02   |   62.90   |   224.34   |   58.73   |   122.32   |\n| mPLUG-Owl |    ✓    |   ✓   |  ✓  |     |  ✓  |   71.77   | **45.92** |   51.89   |   36.37   |   33.71   |   67.86   |   62.35   |   225.60   |   59.16   |   122.79   |\n| mPLUG-Owl |    ✓    |   ✓   |     |  ✓  |  ✓  | **72.68** |   45.90   | **53.02** |   36.73   | **34.51** | **68.19** | **63.28** | **226.43** | **59.22** | **123.11** |\n\n**[W2] Analysis on using more scales**\n\nWe appreciate the reviewer's suggestion to explore alternative configurations of our multi-scale feature pyramid hierarchy. Our method is designed to be flexible and can be extended to use an arbitrary number of scales. For example, a 3-scale configuration with feature hierarchy can be implemented as follows: combining features from scales 3 and 2, then merging the results with features from scale 1. Due to significant time constraints during the rebuttal period, it is challenging to conduct extensive experiments on these alternatives. We are currently running an experiment with the model that integrates 3 scales. We present the validation performance of each task after 2 epochs in Table E. \n\nTable E. Validation performance with varying scales after 2 epochs.\n\n| # of scales | Throughput (img/s) |   DocVQA  |  InfoVQA  |  DeepForm |    KLC    |    WTQ    |  TabFact  |  ChartQA  |  VisualMRC |  TextVQA  |  TextCaps  |\n|:-----------:|:------------------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|:----------:|:---------:|:----------:|\n|      1      |        1.481       |   36.29   |   21.24   |   16.62   |   36.23   |   14.31   |   52.52   |   27.44   |   178.04   |   42.63   |   126.56   |\n|      2      |        1.088       |   42.18   |   24.53   |   21.26   |   38.74   |   16.60   |   53.13   |   32.81   |   195.55   |   45.22   |   129.24   |\n|      3      |        0.494       | **44.86** | **27.57** | **25.61** | **39.48** | **18.77** | **54.17** | **33.05** | **201.49** | **46.92** | **131.82** |\n\nWhen integrating the 3rd scale in addition to the 1st and 2nd scale, the number of visual inputs increases about $(1+g+4g+16g)/(1+g+4g) ≈ 4.2$ times, while the throughput of our model only decreases by a factor of 2. Note that $g$ represents the number of grids in line 124. The validation results demonstrate that incorporating the 3rd scale still helps the document understanding but the improvement gain is diminished. This is expected because, as more scales are added, the incremental benefit decreases since the model has already captured most of the relevant information from the initial scales. We expect this experiment to be finished within the discussion period and will post the performance on the test set.\n\n**[Q1] Clarification on Table 3**\n\nYes. The results labeled as \"Ours\" in Table 3 indeed represent our final model, which includes the Relative Text-Position Prediction Task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript received originally mixed comments. Following the rebuttal and lenghtly discussion between the authors and reviewers, then final assessment overall positive.

The main issue still under discussion, namely the novelty of the use of multi-scale features is clarified, as the contribution clearly lies not in the use of multi-scale features, but in the way of aggregating them in an computationally efficient approach. This is nevertheless something to be clarified in the revised version of the manuscript.

I encourage the authors to properly integrate the clarifications offered during the rebuttal process in the final revision.
Decision: Accept (poster)