Title: A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding
Venue: NeurIPS 2024
Accepted: no
Keywords: LLM

DocAI

Visually Rich Document Understanding

KIE
Abstract: Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce nterleaving Layout and Text in a Large Language Model (LayTextLLM)} for document understanding. In particular, LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in Key Information Extraction (KIE) and Visual Question Answering (VQA). Comprehensive benchmark evaluations reveal significant improvements, with a 27.0\% increase on KIE tasks and 24.1\% on VQA tasks compared to previous state-of-the-art document understanding MLLMs, as well as a 15.5\% improvement over other SOTA OCR-based LLMs on KIE tasks.

1

Abstract

Recently, many studies have demonstrated that exclusively incorporating OCRderived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Lay out and Text in a L arge L anguage M odel (LayTextLLM) for document understanding. In particular, LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in Key Information Extraction (KIE) and Visual Question Answering (VQA). Comprehensive benchmark evaluations reveal significant improvements, with a 27.0% increase on KIE tasks and 24.1% on VQA tasks compared to previous state-of-the-art document understanding MLLMs, as well as a 15.5% improvement over other SOTA OCR-based LLMs on KIE tasks.

1 Introduction

Recent research has increasingly focused on applying Large Language Models (LLMs) [1-17] to document-oriented Visual Question Answering (VQA) and Key Information Extraction (KIE) scenarios. Efforts to build a text-sensitive MultiModal Large Language Models (MLLMs) based on existing LLMs, particularly aimed at enhancing Visually Rich Document Understanding (VRDU), have made significant progress [6, 12, 18]. Although existing MLLMs show promising results in document understanding, they often encounter challenges related to image resolution. When the input image is of low resolution, it is too blurry to extract visual features effectively. Conversely, high-resolution images require additional computational resources to capture detailed textual information [12].

Concurrently, another line of research employs off-the-shelf OCR tools to extract text and spatial layouts, which are then combined with LLMs to address VRDU tasks. These approaches assume that most valuable information for document comprehension can be derived from the text and its spatial layouts, viewing spatial layouts as 'lightweight visual information' [19] . Following this premise, several studies [12, 20-23] have explored various approaches that integrate spatial layouts with text for LLMs, achieving results that are competitive with, or even surpass, those of MLLMs.

The most natural method to incorporate layout information is by treating spatial layouts as tokens, 31

which allows for the seamless interleaving of text and layout into a unified text sequence [20, 22, 23]. 32

For example, Perot et al. [20] employ format such as 'HARRISBURG 78|09' to represent OCR text 33

and corresponding layout, where 'HARRISBURG' is OCR text and '78|09' indicates the mean of 34

the horizontal and vertical coordinates, respectively. Similarly, He et al. [23] use '[x_min, y_min, 35 x_max, y_max]' to represent layout information. These approaches can effectively take advantage of 36 autoregressive characteristics of LLMs and is known as the 'coordinate-as-tokens' scheme [20]. In 37 contrast, DocLLM [19] explores interacting spatial layouts with text through a disentangled spatial 38 attention mechanism that captures cross-alignment between text and layout modalities. 39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

However, we argue that both of the previous approaches have limitations. As shown in Fig. 1, coordinate-as-tokens significantly increases the number of tokens. Additionally, to accurately comprehend coordinates and enhance zero-shot capabilities, this scheme often requires fewshot in-context demonstrations and large-scale language models, such as ChatGPT Davinci003 (175B) [23], which exacerbates issues related to sequence length and GPU resource demands. Meanwhile, although DocLLM does not increase sequence length and integrates spatial layouts through attention, its generalizability is limited. We believe that spatial cross attention and masked span tasks in DocLLM cannot fully utilize the autoregressive traits of LLMs.

To address these problems, this paper explores a simple yet effective approach to enhance the interaction between spatial layouts and text Interleaving Lay out and Text in a L arge L anguage M odel (LayTextLLM) for document understanding. Adhering to the common practice of interleaving any modality with text [15, 24, 25], we specifically apply this principle to spatial layouts. In particular, we maps each bounding box

10 3

Sequence Length (log scale)

20

40

60

80

100

Performance

CORD

FUNSD

SROIE

DocVQA

CORD

FUNSD

SROIE

DocVQA

CORD

FUNSD

SROIE

DocVQA

CORD

SROIE

LayTextLLM-7B(ours)

DocLLM-7B

Coord-as-tokens-7B(Llama2)

Coord-as-tokens-ICL-175B(Davinci-003)

to a single embedding, which is then interleaved with its corresponding text. Then we propose a tailored pre-training task-Layout-aware Next Token Prediction-a completely self-supervised task that enhances the alignment between layout and textual modalities without using synthetic data. Finally, through the proposed Shuffled-OCR Supervised Fine-tuning, LayTextLLM significantly improves performance on downstream document-related VQA and KIE tasks. As shown in Fig. 1, LayTextLLM significantly outperforms the 175B models, while only slightly increasing or even reducing the sequence length compared to DocLLM. Our contributions can be listed as follows:

We propose LayTextLLM for document understanding. To the best of the authors' knowledge, this is the first work to employ a unified embedding approach (Sec. 3.1.1) that interleaves spatial layouts directly with textual data within a LLM. By representing each bounding box with one token, LayTextLLM efficiently addresses sequence length issues brought by coordiante-as-tokens while fully leveraging autoregressive traits for enhanced document understanding.

We propose two tailored training tasks: (1) Layout-aware Next Token Prediction (Sec. 3.2.1), a completely self-supervised training task to enhance the alignment between layout and textual modality; (2) Shuffled-OCR Supervised Fine-tuning task (Sec. 3.2.2) to better elicit the model generalizability in downstream tasks.

Comprehensive experimental results demonstrate quantitatively that LayTextLLM significantly outperforms previous state-of-the-art (SOTA) OCR-free MLLMs by a large margin in zero-shot scenarios, particularly in KIE tasks with an improvement of 27.0%. Additionally, we illustrate that LayTextLLM competes effectively or even surpasses previous SOTA OCRbased methods in both zero-shot and SFT scenarios. Specifically, it surpasses DocLLM by 19.8% on VQA and 15.5% on KIE tasks (Sec. 4).

Extensive ablations demonstrate the utility of the proposed component, with analysis showing that LayTextLLM not only improves performance but also reduces input sequence length compared to current OCR-based models.

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

2 Related Work

2.1 OCR-based LLMs for Document Understanding

Early document understanding methods [26-30] tend to solve the task in a two-stage manner, i.e. , first reading texts from input document images using off-the-shelf OCR engines and then understanding the extracted texts. Considering the advantages of LLMs ( e.g. , high generalizability), some recent methods endeavor to combine LLMs with OCR-derived results to solve document understanding. For example, inspired by the 'coordinate-as-tokens' scheme [20], He et al. [23] propose to use '[x_min, y_min, x_max, y_max]' to introduce the layout information, which can fuse the layout information and texts into a unified text sequence and fully exploit the autoregressive merit of LLMs. To reinforce the layout information while avoiding increasing the number of tokens, DocLLM [19] designs a disentangled spatial attention mechanism to capture cross-alignment between text and layout modalities. Recently, LayoutLLM [21] utilizes the pre-trained layout-aware model [31], to insert the visual information, layout information and text information. However, the aforementioned methods neither suffer from the computational overhead leading by the increasing tokens or hardly take advantage of autoregressive characteristics of LLMs. Thus, it is an urgent problem to address how to better incorporate layout information without significantly increasing the number of tokens.

2.2 OCR-free MLLMs for Document Understanding

Another approach to solve document understanding tasks is the OCR-free method. Benefiting from the end-to-end training framework, it involves processing the text content of documents directly, without relying on OCR engines. Donut [32] first presents an OCR-free method through mapping a text-rich document image into the desired answers. Pix2Struct [33] is trained to parse masked screenshots of web pages into simplified HTML, where variable resolution inputs are supported. While these approaches eliminate the need for OCR tools, they still necessitate task-specific finetuning. With the increasing popularity of LLMs/MLLMs [10-17], various methods are proposed to solve the document understanding task through explicitly training models on visual text understanding datasets and fine-tuning them with instructions to perform a zero-shot prediction. LLaVAR [34] and UniDoc [10] are notable examples that expand upon the document-oriented VQA capabilities of LLaVA [35] by incorporating document-based tasks. These models pioneer the use of MLLMs for predicting texts and coordinates from document images, enabling the development of OCRfree document understanding methods. Additionally, DocPedia [9] operates document images in the frequency domain, allowing for higher input resolution without increasing the input sequence length. Recent advancements in this field, including mPLUG-DocOwl [18], Qwen-VL [6], and TextMonkey [12], leverage publicly available document-related VQA datasets to further enhance the document understanding capability. Although these OCR-free methods have exhibited their advantages, they still struggle with the high-resolution input to reserve more text-related details.

3 Method

In this section, we present our LayTextLLM. First, we introduce a innovative Spatial Layout Projector (Sec. 3.1.1) converts four-dimensional layout coordinates into a single-token embedding. To reduce parameter overhead, we apply Partial Low-Rank Adaptation (Sec. 3.1.2). We also introduce two specific training tasks: Layout-aware Next Token Prediction (Sec. 3.2.1) to align layouts with text during pre-training, and Shuffled-OCR Supervised Fine-tuning (Sec. 3.2.2) to enhance the generalizability of the model. An illustration of our approach is shown in Fig. 2.

3.1 Model Architecture

LayTextLLM is built on the Llama2-7B-base model, which was originally designed to accept only text inputs [36, 37]. To enable the model to interleave spatial layouts with text, we introduce a novel Spatial Layout Projector. This projector converts OCR-derived coordinates into bounding box tokens. We also adopt the Partial Low-Rank Adaptation, a minimally invasive method to incorporate additional modalities while preserving the LLM's inherent knowledge intact.

139

OCR-Engine Pre-processing

Q:"What is the

title of this paper?"

"A Bounding Box...."

"Document Understanding"

A:"A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large

. . .

. . .

. . .

SLP

. . .

Tokenizer

. . .

Tokenizer

SLP

. . .

. . .

. . .

Text Token

Legend

A Bounding Box is Worth One Token:Interleavint

Layout and Text in a Large Language Model for

Document Understanding

ABoundingBoxisWorthOneToken:Interleaving

DocumentUnderstanding

Anonymoes Aathsr(s)

Alllurios

Address

esa11

[∞1,y1,2,y2]

Abstract

dly

Recestly, maty stdies hve dmsrated tat exclsively iscootating OCR.

effetive for doument undentanding tasks, particlary with low-eesolution i-

tinp,LMg

embeding ad iterlewes it with tet, eciety avoiding long seg

Ive traits of LLMs, La/TouLM not oely stem

isaes

a

4

5

Key lfomtion Estraction (KIE) and Visl Qustios Anwering (VQA). Cob

OCR-based LLMs.

i au%

bce in

61

t

ti+1

ti+k

s1Introdoction

bidingupLLaVA ]bLaVAR[d UiDoc ]

applicatio to texi-rich VQA scenarios. For cmplc

sene-stisul qtin erig (VQA)ftFollig ts work,  fts hve

est Undentanding (VRDU) taslks. Models sch as

gh eisting MLLMs show

Whes de ispet in

Concuy anth e f rrch ems f-eshf CR  o et t and stil

patial laoat sig

munle om nglaug, re JA

riee* 2]J. Building o is

wih LLMs, xhi

and att:

lstean.5q, g 9o(gm

3.1.1 Spatial Layout Projector (SLP)

A key innovation in LayTextLLM is the Spatial Layout Projector (SLP), which transforms a spatial 140 layout into a singular bounding box token. This enhancement enables the model to process both 141 spatial layouts and textual inputs simultaneously. To be specifically, each OCR-derived spatial layout 142 is represented by a bounding box defined by four-dimensional coordinates [ x 1 , y 1 , x 2 , y 2 ] , these 143 coordinates represent the normalized minimum and maximum horizontal and vertical extents of the 144 box, respectively. The SLP maps these coordinates into a high-dimensional space that the language 145 model can process as a single token. The process can be computed as z = W · c + b , where c ∈ R 4 146 is the vector of the bounding box coordinates. W ∈ R d × 4 is a weight matrix with d represents 147 the dimension of the embedding, b ∈ R d × 1 is a bias vector, z is the resulting bounding box token 148 represented as an d -dimensional embedding. As illustrated in Fig. 2, the resulting bounding box 149 token z will be interleaved with corresponding textual embeddings to put into LLMs. Note that the 150 SLP is shared by all bounding box tokens so very limited number of parameters are introduced. 151

152

153

154

155

156

Compared to the coordinate-as-tokens scheme, the SLP represents each bounding box with a single token. This approach significantly reduces the number of input tokens and adheres to the practice of interleaving any modality with text, effectively integrating layout and textual information into a unified sequence. This allows the model to process both modalities simultaneously and coherently, fully leveraging the autoregressive traits of LLMs.

157

3.1.2 Layout Partial Low-Rank Adaptation

Pre-trained

Weights

Text Token

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

After using the SLP to generate bounding box tokens and a tokenizer to produce text tokens, these two modalities are then communicated using a Layout Partial Low-Rank Adaptation (P-LoRA) module in LLMs. P-LoRA, introduced in InternLM-XComposer2 [15], is originally used to adapt LLMs to visual modality. It applies plug-in low-rank modules specified to the visual tokens, which adds minimal parameters while preserving the LLMs inherent knowledge.

Formally, as shown in Fig. 3 for a linear layer in the LLM, the original weights W O ∈ R C out × C in and bias B O ∈ R C out are specified for input and output dimensions C in and C out . P-LoRA modifies this setup by incorporating two additional matrices, W A ∈ R C r × C in and W B ∈ R C out × C r . These matrices are lower-rank, with C r being considerably smaller than both C in and C out , and are specifically designed to interact with new modality tokens, which in our case are bounding box tokens. For example, given an input x =

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

LLM

LLM




...




...




...




...

(b) Layout-aware Next Token Prediction

(a) Next Token Prediction

Text Token

w/o Supervision

[ x b , x t ] comprising of bounding box tokens ( x b ) and textual tokens ( x t ) is fed into the system, the 175 forward process is as follows, where ˆ x t , ˆ x b and ˆ x are outputs: 176

3.2 Training Procedure

LayTextLLM is trained with innovative layout-aware training procedure, which consists of two stages: Layout-aware Next Token Prediction pre-training and Shuffled-OCR Supervised Fine-tuning.

3.2.1 Layout-aware Next Token Prediction

Inspired by the next token prediction commonly used in current LLM pre-training [1-7], we propose the Layout-aware Next Token Prediction (LNTP). Fig. 4 presents the contrast of the proposed Layoutaware Next Token Prediction and the conventional next token prediction task. The traditional next token prediction (Fig. 4(a)) relies solely on the textual content, predicting each subsequent token based on the prior sequence of tokens without considering their spatial layouts. Layout-aware next token prediction (Fig. 4(b)), however, interleaves the spatial information encoded by SLP ( i.e. , b i ) with the text tokens ( i.e. , t i ). This integration considers both the content and its layout within the document, leading to a richer, more precise understanding of both the structure and the content.

CASH (MYR)

-20.00

Change

1.30

GST%

Amt(RM)

GST(RM)

Total(RM)

SR

6

17.64

1.06

18.70

GOODS SOLD ARE NON-CASH REFUNDABLE.

EXCHANGE OF GOODS WITHIN 14 DAYS

ACCOMPANIED BY ORIGINALRECEIPT.

the parameters of SLP and P-LoRA.

It is important to note that the proposed Layout-aware Next Token Prediction is a completely selfsupervised pre-training procedure, unlike previous works that require human annotations of document structure data or synthetic data generated by larger LLMs such as GPT-4 [21]. Thus, LNTP facilitates the creation of large-scale, high-fidelity pre-training datasets at minimal cost.

Similarly, primary objective of LNTP is to maximize the likelihood of its predictions for the next token. Thus the loss function is defined as

where P ( t i | t 1 , t 2 , . . . , t i -1 ) represents the probability of i th token t i given the sequence of preceding tokens t 1 , t 2 , . . . , t i -1 , as predicted by the model. Note that we compute the loss only for text tokens, excluding bounding box tokens. During pre-training, our goal is to enhance the alignment between spatial layouts and textual modality, while preserving the LLM's inherent knowledge as much as possible. Thus, we freeze the LLMs and only update

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

3.2.2 Shuffled-OCR Supervised Fine-tuning

OCR engines typically process text from top to bottom and left to right. This order is also adopted as the input sequence for current OCR-based LLMs [19, 21]. However, modern LLMs often exhibit a strong inductive bias toward the positions of input tokens, influenced by designs such as Rotary Position Embeddings (RoPE) [38]. Specifically, tokens that are close together in the input sequence are likely to receive higher attention scores, which is advantageous for processing standard text sequences. Such inductive bias brings cons and pros.

Consider the example illustrated in Fig. 5, where the OCR input text reads: ' ... Change, 1.30, GST%, Amt(RM), GST(RM), Total(RM), SR, 6, 17.64, 1.06, 18.70 ... ' . If the question posed is 'What is the value of the field Change?' (highlighted in a blue box), the model easily identifies '1.30' as it is closely positioned to the word 'Change' in the sequence. However, for a more challenging query like 'What is the value of the field Total(RM)?' (highlighted in a red box), the model struggles to determine the correct answer due to the presence of multiple subsequent numbers closed to 'Total(RM)' . LayTextLLM integrates spatial layouts with textual data, reducing reliance on input sequence order. Thus, we posit that shuffling the OCR input order could enhance the resilience of LayTextLLM in discerning relevant information irrespective of token proximity in the sequence.

Specifically, we propose Shuffled-OCR Supervised Fine-tuning (SSFT) that randomly shuffles the order of OCR-derived text in a certain proportion of examples. The range of exploration for the shuffling ratio can be found in Tab. 7 and 20% shuffled ratio is applied. The training objective is equivalent to predicting the next tokens, but in this scenario, only the tokens of the response are used to compute loss. During SSFT, we unfreeze all parameters including those of LLMs. Experimental results in Section 4.6 demonstrate that utilizing SSFT can further enhance model performance, making it more robust to disruptions in input token order.

4 Experiments

4.1 Datasets

Pre-training data In our training process, we exclusively use open-source data to facilitate replication. We collect data from two datasets for pre-training: (1) IIT-CDIP Test Collection 1.0 [39] and (2) DocBank [40]. The IIT-CDIP Test Collection 1.0 comprises an extensive repository of more than 16 million document pages. DocBank consists of 500K documents, each presenting distinct layouts with a single page per document. For training efficiency, we choose to utilize the entire DocBank dataset and only subsample 5 million pages from the IIT-CDIP collection 1.0.

SFT data For document-oriented VQA, we select Document Dense Description (DDD) and Layout-aware SFT data used in Luo et al. [21], which are two synthetic datasets generated by GPT-4. Besides, DocVQA [41], InfoVQA [42], ChartQA [43], VisualMRC [44] is included following [12]. For KIE task, we select SROIE [45], CORD [46], FUNSD [47], POIE [48] datasets following [12, 19, 21].

4.2 Implementation Detail

The LLM component of LayTextLLM is initialized from the Llama2-7B-base [36], which is a widelyused backbone. Other parameters including SLP and P-LoRA are randomly initialized. During pre-training, the LLM is frozen, and the parameters of SLP and P-LoRA modules are updated. During SFT, all parameters are fine-tuned. Other detailed setup can be found in Appendix B.

We have configured the model with three versions of LayTextLLM for a side-by-side comparison under different settings. Aligned with Luo et al. [21], the first version, LayTextLLM zero , is trained exclusively with DDD and Layout-aware SFT data. Building upon this, and in alignment with the setting of Liu et al. [12], we introduce the DocVQA, InfoVQA, and ChartQA training sets to the dataset pool for our second version, termed LayTextLLM vqa . Finally, we incorporate a comprehensive suite of KIE datasets-FUNSD, CORD, POIE, SROIE, and VisualMRC-as described by Wang et al. [19], creating our most extensive version, LayTextLLM all . Note that all versions are based on the same pre-trained LayTextLLM weight.

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

4.3 Baselines

OCR-free baselines In the category of OCR-free MLLMs, we have chosen the following SOTA models as our strong baselines due to their superior performance in both document-oriented VQA and KIE tasks. These include UniDoc [10], DocPedia [9], Monkey [49], InternVL [50], InternLMXComposer2 [15], TextMonkey , and TextMonkey + [12].

OCR-based baselines For OCR-based baseline models, we implemented a basic approach using only OCR-derived text as input. This was done using two versions: Llama2-7B-base and Llama27B-chat . We also adapted the coordinate-as-tokens scheme from He et al. [23] for these models, resulting in two new variants: Llama2-7B-base coor and Llama2-7B-chat coor . It's important to note that we did not employ the ICL strategy with these models, as it would significantly exceed their maximum sequence length constraints. Additionally, we included results from a stronger baseline using the ChatGPT Davinci-003 (175B) model [23], termed Davinci-003-175B coor . One other recent SOTA OCR-based approach, DocLLM [19] is also considered in our analysis. Finally, LayoutLLM and LayoutLLM CoT [21], which integrates visual cues, text and layout is also included.

4.4 Evaluation Metrics

To ensure a fair comparison with OCR-free methods, we adopted the accuracy metric, where a 269 response from the model is considered correct if it fully captures the ground truth. This approach 270 aligns with the evaluation criteria described by [9, 10, 12]. To further enhance the comparability with 271 other OCR-based methods, we conducted additional evaluations using original metrics specific to 272 certain datasets, such as F1 score [19, 23], ANLS [19, 21, 51] and CIDEr [19, 52]. 273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

4.5 Quantitative Results

Comparison with SOTA OCR-free Methods The experimental results shown in Tab. 1 demonstrate the outstanding performance of the LayTextLLM series across various tasks. Note that the results for ChartQA are reported in Appendix E due to concerns about fairness in comparison, as the dataset does not include OCR-derived results and we used in-house OCR tools instead. Firstly, LayTextLLM zero significantly outperforms previous SOTA OCR-free methods, such as TextMonkey [12], in zero-shot capabilities, even when these methods use the training set of the dataset. For example, in the DocVQA and InfoVQA datasets, LayTextLLM zero achieves accuracies of 71.8% and 33.8%, respectively, which are markedly higher than existing OCR-free methods such as TextMonkey and InternLM-XComposer2. When fine-tuned with corresponding datasets, LayTextLLM shows even greater performance improvements, particularly in document-oriented VQA datasets. Specifically, its accuracies on DocVQA and InfoVQA increase to 77.4% and 66.1%, respectively, demonstrating the model's strong ability to leverage task-specific data. Additionally, LayTextLLM zero excels in KIE datasets, particularly on the SROIE and POIE datasets, achieving accuracies of 86.7% and 66.1%, respectively. These results significantly surpass those of previous SOTA OCR-free model ( i.e. , TextMonkey + ) by margins of 40.5% and 34.1%, respectively. This significant performance gain is likely due to these datasets containing low-resolution images that are too blurred for current MLLMs to extract visual features, whereas LayTextLLM shows robustness in such challenging scenarios.

Comparison with SOTA OCR-based Methods For comprehensive comparison, we have also conducted correspinding experiments to align with OCR-based methods [19, 21]. The experimental results presented in Tab. 2 showcase significant performance improvements achieved by LayTextLLM models compared to pure OCR-based SOTA methods such as DocLLM [19]. Specifically, when comparing with DocLLM, LayTextLLM zero demonstrates notably superior performance, with even its zero-shot capabilities being competitive with supervised SFT approaches. We believe that the subpar performance of DocLLM is likely due to its use of cross-attention and the masked span pre-training tasks [53], which fail to leverage the autoregressive features of LLMs effectively. Similarly, when contrasting with coordinate-as-tokens employed in Llama2-7B, LayTextLLM zero again outperforms significantly. This disparity in performance can be attributed to the following three reasons: (1) The coordinate-as-tokens approach tends to introduce an excessive number of tokens, often exceeding the pre-defined maximum length of Llama2-7B ( i.e. , 4096). Consequently, this leads to a lack of crucial OCR information, resulting in hallucination and subpar performance. (2) When re-implementing the coordinate-as-tokens method with Llama2-7B, we did not introduce the ICL strategy, as it would contribute additional length to the input sequence. (3) The coordinate-as-tokens approach necessitates a considerably larger-sized LLM to comprehend the numerical tokens effectively.

In comparison to LayoutLLM [21], our approach exhibits discrepant performance in different tasks, as shown in Tab. 3. In zero-shot scenarios, we outperform LayoutLLM in most KIE datasets, validating our capability to leverage OCR-based results effectively. However, we fall short on documentoriented VQA tasks since answering some questions that are strongly related to vision information may challenge our approach. Two main reasons may well explain this performance discrepancy:

(1) The visual encoder in LayoutLLM provides additional visual information. (2) LayoutLLM 313 incorporates the Chain-of-Thought (CoT) mechanism to model contextual information while it is 314 not used in our approach. However, when fine-tuned with tailored data, LayTextLLM significantly 315 outperforms LayoutLLM, showcasing its strong ability to utilize task-specific data. More qualitative 316 example demonstrates can be found in Appendix A. 317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

4.6 Analysis

Ablations To better assess the utility of Layout-aware Next Token Prediction and Shuffled-OCR Supervised Fine-tuning in LayTextLLM, an ablation study was performed (see Tab. 4). Details on the training setup for all variants are provided in Appendix B. It is evident that both LNTP and SSFT significantly enhance the utility of LayTextLLM. Specifically, disabling LNTP results in an 8% decrease in performance on VQA tasks and a 5.5% decrease on KIE tasks. Disabling SSFT leads to a decrease in average accuracy by 6.7% and 2.6% for VQA and KIE tasks, respectively.

Sequence Length Tab. 5 presents statistics on the average input sequence length across different datasets. Intriguingly, despite interleaving bounding box tokens, LayTextLLM consistently exhibits the shortest sequence length in three out of four datasets, even surpassing DocLLM, which is counterintuitive. We attribute this to the tokenizer mechanism. For example, using tokenizer.encode(), a single word from the OCR engine, like 'International' is encoded into a single ID [4623] . Conversely, when the entire OCR output is processed as one sequence, such as '... CPC,International,Inc... ' , the word 'International' is split into two IDs [17579 , 1288] , corresponding to 'Intern' and 'ational' respectively. This type of case occurs frequently, more discussion in Appendix C.

5 Limitation

Although LayTextLLM has shown significant capabilities in text-rich VQA and KIE tasks, this alone does not suffice for all real-world applications. There are some instances, particularly in chart analysis, where reasoning must be based solely on visual cues ( e.g. size, color)-a challenge that remains unmet. Questions such as 'What is the difference between the highest and the lowest green bar?' illustrate this gap. The ChartQA results, detailed in Appendix E, also underscore these limitations. Addressing these challenges highlights the urgent need for future enhancements that integrate visual cue within the capabilities of LayTextLLM.

6 Conclusion

We propose LayTextLLM for various VRDU tasks, in which spatial layouts and textual data are seamlessly interleaved to make more accurate prediction by introducing a innovative Spatial Layout Projector. Two tailored training tasks - Layout-aware Next Token Prediction and Shuffled-OCR Supervised Fine-tuning - are designed to improve the comprehension of document layouts. Extensive experiments confirm the effectiveness of LayTextLLM.

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

References

OpenAI:Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, FlorenciaLeoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, HyungWon Chung, Dave Cummings, and Jeremiah Currier. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , Dec 2023.

Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of lmms: Preliminary explorations with gpt-4v(ision). arXiv preprint arXiv:2309.17421 , Sep 2023.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card , 2024.

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jeanbaptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 , 2024.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525 , 2024.

Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652 , 2024.

Hao Feng, Qi Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. arXiv preprint arXiv:2311.11810 , 2023.

Hao Feng, Zijian Wang, Jingqun Tang, Jinghui Lu, Wengang Zhou, Houqiang Li, and Can Huang. Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding. arXiv preprint arXiv:2308.11592 , 2023.

Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mPLUG-DocOwl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403.12895 , 2024.

Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473 , 2024.

Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Hao Feng, Yang Li, Siqi Wang, Lei Liao, et al. Textsquare: Scaling up text-centric visual instruction tuning. arXiv preprint arXiv:2404.12803 , 2024.

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821 , 2024.

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, 396 Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering 397 free-form text-image composition and comprehension in vision-language large model. arXiv 398 preprint arXiv:2401.16420 , 2024. 399

400

401

402

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814 , 2024.

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https: //llava-vl.github.io/blog/2024-01-30-llava-next/ .

Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mPLUG-DocOwl: Modularized multimodal large language model for document understanding. arXiv:2307.02499 , 2023.

Dongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma, Petr Babkin, Simerjot Kaur, Yulong Pei, Armineh Nourbakhsh, and Xiaomo Liu. Docllm: A layout-aware generative language model for multimodal document understanding. arXiv preprint arXiv:2401.00908 , 2023.

Vincent Perot, Kai Kang, Florian Luisier, Guolong Su, Xiaoyu Sun, Ramya Sree Boppana, Zilong Wang, Jiaqi Mu, Hao Zhang, and Nan Hua. Lmdx: Language model-based document information extraction and localization. arXiv preprint arXiv:2309.10952 , 2023.

Chuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. Layoutllm: Layout instruction tuning with large language models for document understanding. CVPR 2024 , 2024.

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195 , 2023.

Jiabang He, Lei Wang, Yi Hu, Ning Liu, Hui Liu, Xing Xu, and Heng Tao Shen. Icl-d3ie: In-context learning with diverse demonstrations updating for document information extraction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1948519494, 2023.

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv:2302.14045 , 2023.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv:2306.14824 , 2023.

Wonseok Hwang, Jinyeong Yim, Seung-Hyun Park, Sohee Yang, and Minjoon Seo. Spatial dependency parsing for semi-structured document information extraction. Cornell University arXiv,Cornell University - arXiv , May 2020.

Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , Aug 2020. doi: 10.1145/3394486.3403172. URL http://dx.doi.org/10.1145/3394486.3403172 .

Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. Layoutlmv2: Multimodal pre-training for visually-rich document understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , Jan 2021. doi: 10.18653/ v1/2021.acl-long.201. URL http://dx.doi.org/10.18653/v1/2021.acl-long.201 .

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

Teakgyu Hong, DongHyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents. Proceedings of the AAAI Conference on Artificial Intelligence , page 10767-10775, Jul 2022. doi: 10.1609/aaai.v36i10.21322. URL http://dx.doi.org/ 10.1609/aaai.v36i10.21322 .

Zineng Tang, Zhenfeng Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Zhu C, Michael Zeng, Zhang Cha, and Mohit Bansal. Unifying vision, text, and layout for universal document processing. Cornell University - arXiv,Cornell University - arXiv , Dec 2022.

Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia , pages 4083-4091, 2022.

Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision , pages 498-517, 2022.

Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning , pages 18893-18912. PMLR, 2023.

Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107 , 2023.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv:2304.15010 , 2023.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024.

D. Lewis, G. Agam, S. Argamon, O. Frieder, D. Grossman, and J. Heard. Building a test collection for complex document information processing. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval , Aug 2006. doi: 10.1145/1148170.1148307. URL http://dx.doi.org/10.1145/1148170. 1148307 .

Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. Docbank: A benchmark dataset for document layout analysis. In Proceedings of the 28th International Conference on Computational Linguistics , Jan 2020. doi: 10.18653/v1/2020. coling-main.82. URL http://dx.doi.org/10.18653/v1/2020.coling-main.82 .

Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 2200-2209, 2021.

Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 1697-1706, 2022.

Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

for Computational Linguistics: ACL 2022 , pages 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.177. URL https://aclanthology.org/2022.findings-acl.177 .

Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmrc: Machine reading comprehension on document images. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 13878-13888, 2021.

Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 competition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR) , pages 1516-1520. IEEE, 2019.

Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. Cord: a consolidated receipt dataset for post-ocr parsing. In Workshop on Document Intelligence at NeurIPS 2019 , 2019.

Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW) , volume 2, pages 1-6. IEEE, 2019.

Jianfeng Kuang, Wei Hua, Dingkang Liang, Mingkun Yang, Deqiang Jiang, Bo Ren, and Xiang Bai. Visual information extraction in the wild: practical dataset and end-to-end solution. In International Conference on Document Analysis and Recognition , pages 36-53. Springer, 2023.

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607 , 2023.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238 , 2023.

Liangcai Gao, Yilun Huang, Herve Dejean, Jean-Luc Meunier, Qinqin Yan, Yu Fang, Florian Kleber, and Eva Lang. Icdar 2019 competition on table detection and recognition (ctdar). In International Conference on Document Analysis and Recognition , 2019.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4566-4575, 2015.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1162. URL https://aclanthology.org/P16-1162 .

Appendix

A Qualitative Examples

Qualitative examples of document-oriented VQA (upper row) and KIE (bottom row) are shown in Fig. 6. The results indicate that LayTextLLM is highly effective in utilizing spatial layout information to make more accurate predictions for these challenging examples. For example, in the upper right figure, many numeric texts in the receipt act as noise for the baseline method. In contrast, LayTextLLM integrates layout information to accurately predict the total price, as demonstrated by the other examples, underscoring the utility of LayTextLLM.

What is the date in this receipt?

2003

Anwsered  by LayTextLLM (Ours)

29 JUN 18

✓

✖

NO.19&21

JALANTEMENGGUNG23/9

BANDAR MAHKOTA CHERAS

4320O SELANGOR DARUL EHSAN,MALAYSIA

TAX INVOICE

GSTID:000164270080

2COUNTER1

2003 saiful

Chk 276703

Guest1

29Jun1806:01:34PM

TBL35/ 1

1

N22 SPRG NDL+SU+CrE

9.34S

1

R02N'LMK+CC+StE

20.66 S

1

D27MIL0（I)E

5.19 S

7.45S

CASH

56.90

what is the 10th service provided

under the 'services provided by

child welfare staff'?

Anwsered  by LayTextLLM (Ours)

✖

10th service provided under the 'services

provided by child welfare staff' is 'Licenses

children's institutions, agencies and day

✓

licenses independent full time and day

care homes

SERVICES PROVIDED BY CHILD WELFARE STAFF

1. Protective casework scrvices to neglected and/or abused children and their familles.

2. Forter family homc placement when the children are comesited to the legal custody of the Division

of Welfare.

3. Recruits, studies, evaluates, and approves or disappeoves foster family, and adoptive bomes

Supenvises childen legplly coemied to the agmcy in foster family and adoptive homes.

5. Countels unmarried parents

6. Places children, legpally free for adoption, In adoptive hceses

Counsehs, and sug

snity resouroes to parents of children with emotioeal problcms, or

8. Protects dependent and neglected children in intcrstate placement

9.  Places children in family day care homes or in group day -care.

10. Licenses independent full time and day care hcenes

11. Licenses children’s institutions, agencies and day care facilities

the cost of maintaining one or more County Child Welfare W'orkers giving these services.

In seventy per cent of the counties, Counry Coerts share with dhe Suse and Federal gover

What is the total price in this receipt?

100.00

Anwsered  by LayTextLLM (Ours)

79.50

✓

✖

LAN HiNG STATIONERY SDN BHD

(162761-M)

NO.32 & 33, JALAN SR 1.9 SEKSY'EN 9.

TAMAN SERDANG RAYA

43300 SERI KEMBANGAN SELANGOR

DARUL EHSAN

GST ID : 002139201536

Tax Invoice

27.032018

Cty Tax

RM

Durafle H399 (110 × 95men)

100SR

58 30

Name Badge (H) (p 0.5500

809 Metal Name Badge Cip

1

SR

2120

- 100’s (@ 20 0000

056

Rounding Adjustment

Total Ant Payable

79.50

Paid Amount

100.00

Change:

20.50

Total Qty Tender

101

GST Summary

Amount

(RM)

(RM)

Tax

SR(A

75.00.

4.50

Total

75.00

4.50

THANK YOU

Which is the last financial year

(FY) listed under sub-heading

"Funding"?

1977

Anwsered  by LayTextLLM (Ours)

FY 1978

✓

✖

3

Selected

ProgramStatis

Funding

Authorizations

FY 1974

$20mlion

FY 1975

$100 milllon

FY 1976

$250 m1lllon

TQ

$62,5 million

FY 1977

$250 mllion

FY 1978

$250 mllion

B Implementation Detail 540

All training and inference procedures are conducted on eight NVIDIA A100 GPUs. 541

Training LayTextLLM is initialized with Llama2-7B-Base model, the pre-training, SFT, and other 542 model hyper-parameters can be seen in Tab. 6. Please note that all variants of LayTextLLM, including 543 those utilized in ablation studies, are trained in accordance with the SFT settings. All baseline results 544 are sourced from their respective original papers, with the exception of the Llama2-7B series and the 545 Llama2-7Bcoor series. These were re-implemented and can be referenced in [21, 23]. 546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

Inference For the document-oriented VQA test set, we use the original question-answer pairs as the prompt and ground truth, respectively. For Key Information Extraction (KIE) tasks, we reformat the key-value pairs into a question-answer format, as described in [12, 19, 21]. Additionally, for the FUNSD dataset, we focus our testing on the entity linking annotations as described in [21].

To eliminate the impact of randomness on evaluation, no sampling methods are employed during testing for any of the models. Instead, beam search with a beam size of 1 is used for generation across all models. Additionally, the maximum number of new tokens is set to 512, while the maximum number of input tokens is set to 4096.

C Discussion of Input Sequence Length

As mentioned in Section 4.6, it is intriguing that LayTextLLM has fewer input sequences than DocLLM, which is counterintuitive given that LayTextLLM interleaves bounding box tokens, typically resulting in longer sequence lengths. We attribute this to the Byte Pair Encoding (BPE) tokenizers [54] prevalently used in modern LLMs such as Llama2.

BPE operates by building a vocabulary of commonly occurring subwords (or token pieces) derived from the training data. Initially, it tokenizes the text at the character level and then progressively merges the most frequent adjacent pairs of characters or sequences. The objective is to strike a balance between minimizing vocabulary size and maximizing encoding efficiency.

Thus, when tokenizing a single word like 'International' on its own, the tokenizer might identify it as a common sequence in the training data and encode it as a single token. This is especially likely if 'International' frequently appears as a standalone word in the training contexts. However, when the word 'International' is part of a larger sequence of words such as including in a long sequence of OCR-derived texts like '...335 CPC,International,Inc... ' , the context changes. The tokenizer might split 'International' into sub-tokens like 'Intern' and 'ational' because, in various contexts within the training data, these subwords might appear more frequently in different combinations or are more useful for the model to understand variations in meaning or syntax.

When using LayTextLLM, we input word-level OCR results into the tokenizer, typically resulting in the former situation, where words are encoded as single tokens. Conversely, with DocLLM, the entire OCR output is processed as one large sequence, leading to the latter situation and a longer sequence length than in LayTextLLM. This difference underscores the utility of LayTextLLM in achieving both accuracy and inference efficiency due to its shorter sequence length.

D Shuffle Ratio Exploration

Tab. 7 presents the results of exploring training and testing shuffling ratios on the FUNSD dataset using two different models: Llama2-7B-base and LayTextLLM. The table shows the performance of these models at various shuffling ratios (100%, 50%, 20%, and 0%).

LayTextLLM consistently outperforms Llama2-7B-base across all levels of shuffling, which further underscores the significance of interleaving spatial layouts with text. Particularly at the 100% shuffle level, Llama2-7B-base demonstrates limited accuracy at only 20.3, while LayTextLLM maintains a relatively higher performance. It is also interesting to note that Llama2-7B-base generally improves as the shuffling percentage decreases, whereas LayTextLLM performs best when 20% of the examples with OCR-derived text are shuffled. This observation suggests that LayTextLLM effectively utilizes spatial layouts and is less dependent on the sequence of input tokens. Therefore, a certain proportion of shuffled examples can serve as adversarial examples to enhance the model's robustness, addressing

situations such as errors in the text order from the OCR engine, which are caused by subtle differences 589 in horizontal or vertical coordinates. 590

E Results of ChartQA

591

As shown in Fig. 7, the question-answer pairs in ChartQA [43] tend to involve the visual cues for 592 reasoning. However, with only text and layout information as input, the proposed LayTextLLM 593 inevitably have difficulties in reasoning visual-related information. Thus, on the ChartQA dataset, 594 LayTextLLM can hardly achieve better performance than previous methods that include visual inputs. 595 Although the visual information is not used in LayTextLLM, it can still exhibit better zero-shot ability 596 than UniDoc [10]. After incorporating the training set of ChartQA, the performance of LayTextLLM 597 can be boosted to 35.7%. Considering the importance of visual cues in ChartQA-like tasks, we will 598 try to involve the visual information into LayTextLLM in future work. 599

Question: What is the

difference

between

the highest and the lowest

green bar

?

GroundTruth:

6

Our Prediction:

40

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have detailed the contributions accurately in the abstract and introduction. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: As discussed in Section. 5, we have listed some limitations of our work and shown corresponding failure cases in the supplementary material.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This work does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Section 4.1, 4.2 and Appendix B, we have described the details of implementing and training the proposed model to ensure the reproducibility of our work.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We would release our code after this paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have detailed the experimental setting and implementation details in Section 4.1, 4.2 and Appendix B of the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our deep learning model is designed for a complex task (requiring huge computing resources) where traditional error bars are less informative due to the high variability in model training and initialization. We ensured the robustness of our model by fix the random seed during inference. In addition, comparative analysis with baseline models demonstrated improvements in key performance areas, underscoring the practical effectiveness of our approach. We acknowledge the limitation of not using traditional statistical tests and suggest that future work could explore statistical significance in more controlled settings.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have reported the needed computer resources in Section B of the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research in this paper conforms with the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: All the datasets used in this paper are publicly available and they contain no unsafe images.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: For the used datasets and pre-trained models, we have cited their corresponding works.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

912

913

914

915

916

917

918

919

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the LayTextLLM method for document understanding, which encodes text positional information in the embedding space of an LLM and trains for effective understanding of document data as interleaved OCR-detected text and bounding box information. The results show improved performance compared to prior works on the KIE tasks, as well as on VQA in many cases.\n\nSTRENGTHS:\nThe treatment of layout information as a modality interleaved with text is logical, and the use of a projection into the LLM’s embedding space to represent bounding boxes is clever and appears to be novel. The tasks approached are important and overall the proposed method does appear to improve document understanding (though this will be more convincing if the caveats listed below are addressed).\n\nI also appreciate the focus on open-source models and data for the method and its evaluation, making the results reproducible.\n\nWEAKNESSES:\nThere are some issues regarding the comparisons to existing models, making it unclear how much of the observed improvement is really due to the novel method proposed.\n\nLayTexLLM is implemented with Llama-2-7b, but it seems that many models compared to (e.g. the strong-performing LayoutLLM) may use other LLM backbones, making it unclear whether the superior performance of LayTexLLM in many settings is due to the proposed novel method or the LLM backbone. The results will be more convincing with a comparison of different methods with the same LLM backbone (or at least an analysis of the number of parameters in each model).\n\nIt is not clear what OCR engine is used, raising the concern that different OCR engines could explain some of the gaps in performance between models being compared.\n\nThere are also issues with how the training is presented that make it difficult to interpret results. Some places (L131, L179, etc.) mention pre-training and SSFT, implying that pre-training means the LNTP training task. However, Sec 4.1 mentions “pre-training” and “SFT”, implying that pre-training refers to SSFT+LNTP and that it is followed by SFT (Supervised Fine Tuning) for particular tasks (VQA and KIE). The results also mention zero-shot and supervised results (e.g. L297), but it is unclear from the text and results tables which results are obtained zero-shot or from SFT, making it hard to understand if the comparisons are fair.\n\nThe statements about large improvements over SOTA MLMMs (L13-14, L83-84) seem slightly misleading since LayTextLLM uses OCR detections and thus is more comparable to other OCR-based methods.\n\nLNTP (Sec. 3.2.1) is presented as a novelty but seems to just be the regular language modeling objective. If I understand correctly, this could be toned down to simply say that the added SLP and P-LoRA parameters are updated with a language modeling loss.\n\nQUESTIONS:\nI don’t fully follow the claim of L54-55 about autoregressive models vs. DocLLM-style models. Why would autoregressive modeling a priori be expected to outperform spatial cross attention for document understanding?\n\nWhile the justification of SSFT (Sec 3.2.2) makes sense, it seems that the issue stems from the use of positional encodings which encode left-to-right order of tokens. Have you considered using positional encodings that directly encode (x, y)-positions of text to avoid this artifact or to give the model an inductive bias towards the layout’s 2D positioning?\n\nWhy was only Llama-2-7b used? Would the proposed method work for other LLM backbones?\n\nWhat is the motivation for using P-LoRA (Sec. 3.1.2)? Is it applied to every layer?\n\nL195 states that LNTP loss is calculated for text tokens only. Does this mean that bounding box tokens are still used as inputs but just not as targets? Why is this done, and is it tested?\n\nThere are a number of minor grammatical errors throughout the paper that need revision, including missing articles (e.g. L163 “to (the) visual modality”, among others) and some awkward wording (e.g. L142 “specific”, L160 “communicated”, L211 “cons and pros” => “advantages and disadvantages”, L262 “it’s” => “it is”, L332 “(and we provide further) discussion”, among others). The acronym SFT used throughout should be defined somewhere. \n\nTables 1-3: The term “polys” and the exact meaning of the asterisk * are unclear.\n\nLIMITATIONS:\nLimitations are clearly discussed in Section 5 (which should have the title “Limitations” in plural). Additionally, does the limitation of lacking visual cues apply to text formatting such as bolding or italics? This would connect well to the examples in Figure 6 where bold text is prominent.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-19"}

comments: SUMMARY:
This work presents an innovative method for integrating layout information into LLMs to enhance document understanding tasks. Instead of treating bounding box coordinates as input text tokens, the bounding box information is embedded into a single token and interleaved with text tokens. This approach addresses the challenge of long sequences while leveraging the autoregressive nature of LLMs. Experimental results demonstrate the effectiveness of the proposed method, achieving state-of-the-art performance and resulting in shorter input sequence lengths.

STRENGTHS:
1.Interleaving layout information and text is novel.

2.The proposed Shuffled-OCR Supervised Fine-tuning is interesting and may benefit other OCR-based approaches.

3.The approach achieves state-of-the-art performance on most text-rich VQA and KIE tasks, validating the effectiveness of interleaving layout and text and significantly reducing input length.

4.The paper is well-written, providing sufficient experimental details, ablations and discussions to comprehend each component of the model.

WEAKNESSES:
1.In layout-aware pretraining tasks, whether it is beneficial to predict both the bounding box and the text, rather than just the text. 

2.LaytextLLM achieves satisfying performance in various tasks, but it will be better to incorporate the visual modality for more application scenarios.

QUESTIONS:
Is the input length shorter than the original input when not using a bounding box, only in the Llama tokenizer, or is this also observed in tokenizers of other LLMs?

LIMITATIONS:
Please refer to weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach, named LayTextLLM, for document understanding tasks, which efficiently integrates spatial layouts and textual data within LLM. It employs a Spatial Layout Projector and introduces two innovative training tasks: Layout-aware Next Token Prediction and Shuffled-OCR Supervised Fine-tuning. Extensive experiments demonstrate significant improvements over previous state-of-the-art models in KIE and VQA. This paper demonstrates the importance of layout information in document understanding tasks.\n\nSTRENGTHS:\n1. The paper introduces a novel approach by integrating SLP and P-LoRA to effectively encode and process layout information. This method significantly improves the interaction between spatial layouts and textual data within LLM, providing a new direction for future research.\n2. The paper proposes the LNTP task and SSFT task to enable the LLM to layout information, thereby enhancing its document understanding capabilities and improving performance on document-related tasks.\n\nWEAKNESSES:\n1. Due to miss the crucial visual information necessary for document understanding, this LayoutTextLLM heavily relies on OCR-derived text and spatial layouts. Other works such as LayoutLLM, layoutLMv3, introduces visual information to enhance the document understanding performance.\n2. The exploration of the shuffling ratio was conducted only on Key Information Extraction (KIE) tasks. It should also be validated on Visual Question Answering (VQA) datasets to determine if the 20% shuffling ratio is optimal across different types of tasks.\n3. The effectiveness of LNTP and SSFT methods should be substantiated with more ablation studies. It is recommended to fine-tune Llama2-7B directly using the existing data for a more comparisons.\n4、Although LayTextLLM shows higher performance on DocVQA compared to LayoutLLM, this comparison is not entirely fair as LayoutLLM was evaluated in a zero-shot setting. Moreover, the zero-shot performance of LayoutLMv3 on DocVQA surpasses that of LayTextLLM.\n\nQUESTIONS:\nCompared with layoutlm series，what are the advantages of the encoding bounding box method proposed in this paper.\n\nLIMITATIONS:\nThe author has already mentioned in the limitation section of the paper that the proposed model is difficult to handle scenarios where inference relies on visual cues.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents LayTextLLM, a novel approach to document understanding that effectively integrates spatial layout information and text into a large language model. Existing methods that integrate spatial layout with text often produce excessively long text sequences. LayTextLLM addresses these problems by projecting each bounding box into a single embedding and interleaving it with text.  The method is evaluated on Key Information Extraction (KIE) and Visual Question Answering (VQA) tasks.\n\nSTRENGTHS:\n- Effective sequence reduction: The proposed method reduces the length of text sequences, addressing a common problem in document understanding.\n- Performance improvement: LayTextLLM demonstrates improvements in KIE and VQA tasks, showing performance gains over alll state-of-the-art models.\n- Evaluation: The paper provides detailed benchmark evaluations on 2 tasks and 7 datasets\n\nWEAKNESSES:\n- Incomplete related work: The paper omits several relevant OCR-based models, such as UDOC, LayoutMask, BROS, LAMBERT, DocFormer and LiLt.\n- Insufficient explanation: The repeated claim that DocLLM cannot fully exploit autoregressive features is not adequately explained.\n- Limited comparisons: There is no comparison with alternative methods that embed coordinates, such as co-as-token approaches (Lmdx, Shikra, ICL-D3IE).\n- Marginal token reduction: The reduction in the number of tokens appears to be limited, and the paper does not clarify whether words or lines are encoded, which could have a significant impact on token reduction.\n\nQUESTIONS:\n- Why use an embedding to encode coordinates that are already 4D vectors? What is the gain, considering there is no additional information (e.g., font, style, zone type)?\n- How can SLP be trained if there is no loss computed on bounding box tokens?\n- How do you explain the good performance of LayTextLLM_zero?\n\nLIMITATIONS:\n- Limited comparisons: The paper primarily compares LayTextLLM to DocLLM, which may not provide a comprehensive assessment of its performance.\n- Impact of token reduction: The reduction in the number of tokens, while beneficial, appears to be limited and may not provide significant practical benefits in all scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}