A Multi-Modal Multilingual Benchmark for Document Image Classification
Authors: Yoshinari Fujinuma
Siddharth Varia
Nishant Sankaran
Srikar Appalaraju
Bonan Min
Yogarshi Vyas
Venue: EMNLP 2023 Findings (2023)
Keywords: Document AI
layout-aware models
visually-rich document understanding
multilingual document image classification

Abstract: Document image classification is different from plain-text document classification and consists of classifying a document by understanding the content and structure of documents such as forms, emails, and other such documents. We show that the only existing dataset for this task (Lewis et al., 2006) has several limitations and we introduce two newly curated multilingual datasets WIKI-DOC and MULTIEURLEX-DOC that overcome these limitations. We further undertake a comprehensive study of popular visually-rich document understanding or Document AI models in previously untested setting in document image classification such as 1) multi-label classification, and 2) zero-shot cross-lingual transfer setup. Experimental results show limitations of multilingual Document AI models on cross-lingual transfer across typologically distant languages. Our datasets and findings open the door for future research into improving Document AI models.

A Multi-Modal Multilingual Benchmark for Document Image Classification
Yoshinari Fujinuma"
Siddharth Varia" Nishant Sankaran
Bonan Min
Srikar Appalaraju
Yogarshi Vyas
AWS AI Labs
(fujinuy,siddhvar nishank, bonanmin, bomaa srikara,
yogarshi yowgarshijammzzon.ccoom jeamazon.com
Abstract
answering (Mathew al., 2020). Models for docu- ment understanding tasks have also benefited from large-scale unsupervised pretraining that infuses data from the various modalities (Appalaraju et al., 2021; Huang et al., 2022, inter alia).
Document image classification is different from plain-text document classification and consists classifying a document by understanding the content and structure of documents such as forms, emails, and other such documents. We show that the only existing dataset for this task (Lewis et al., 2006) has several limitations and we introduce two newly curated multilin- gual datasets (WIKI-DOC and MULTIEURLEX DOc) that overcome these limitations. We fur- ther undertake comprehensive study of pop- ular visually-rich document understanding or Document AI models in previously untested setting in document image classification such 1) multi-label classification, and 2) zero-shot cross-lingual transfer setup. Experimental re- sults show limitations of multilingual Docu- ment AI models cross- longull transfer across typologically distant languages. Our datasets and findings open the door for future research into improving Document Al models.!
this work, we focus on the task of document classification on visually-rich documents, which aims to classify a given input document (usually PDF or an image) into one or more classes. A popular benchmark dataset for this task is the RVL- CDIP collection (Lewis et al., 2006) which con- sists of 16 different classes. However, this dataset is not designed to evaluate models the docu- ment elassification task with deeper understanding (Larson et al., 2023)-it consists of documents English only, the documents are relevant to a sin- gle domain, each document belongs to only one class (i.c., multi-class), and some class labels (c.g.. "email"" "resume") do not require rich semantic understanding of the contents of the document.
Given these limitations, we argue that it is paramount to expand the evaluation for document classification to gain further insights into existing approaches, well identify their limitations. We propose two new datasets, MULTIEURLEX- Doc and WIKI-DoG that complement RVI.-CDIP in different ways. MUL.IEURILEX-DO consists of EU laws in 23 different languages and a multi- label dataset where each document is assigned to one or more of different labels. Additionally, WIKI- Doc is multi-class dataset comprises of rendered Wikipedia articles that covers non-European lan- guages. These datasets are derived from prior work by Chalkidis et al. (2021) and Sinha et al. (2018) respectively. Both datasets require deeper under- standing of the text and contents of the documents to arrive at the correct label(s).
1 Introduction
Visual document understanding aims to extract use- ful information from variety of documents (c.g.. forms, tables) beyond merely Optical Character Recognition (OCR). Unlike texts in traditional NLP tasks, this task is challenging since the content is laid out in 2D structure where each word contains an (x. y) location in the document. Prior work has shown that this task requires a model to process in- formation occurring in multiple modalities includ- ing cues in images and text well as spatial cues (Xu et al., 2020). Visual document understanding consists of several sub-tasks including, but not lim- ited to, document image classification (Lewis et al., 2006), entity extraction (Guillaume Jaume, 2019) and labeling (Park et al., 2019), and visual question
We use these new datasets study the behavior of pre-trained visually-rich document understand- ing (or Document AI) models with a focus on an- swering three specific questions document image
"Equal contribution.
Cooee and datusct linked at https://huggingface.uu co/d ataset ts baasets/Amaazee www.zonnceenne/uullliiinniiooiii si fication

---

Dataset
Domain
Multilingual
Multi Class
Multi-Label
Layout
Class Type
#Classes
RVI--DIIP
Tobacco Ind.
Diverse
Doc. Type
16
MULTIEURLEN EELTTIURREE-DDO DOC
EU Law Wikipedia
Static
Content
567
WIKI-DoC
Static
Content
111
Table 1: A comparison of the datasets introduced in this work with RVL-CDIP, the most commonly used dataset for document image classification. The newly curated datasets complement RVL-CDDIP since they are multilingua, multi-label, and have classes based on document contents rather than document types (Doc. Type).
classification. First, focusing only on the English portions of the datasets, we examine if different pre- trained models perform consistently across datasets, Second, we ask whether multi-lingual document understanding models exhibit similar performances across different languages. Finally, we focus on the cross-lingual generalization ability of these mod- cls, and ask whether muuli--ingual document under- standing models can be used to classify documents without having to be trained on that specific lan- guage. We use a variety of pre-trained models that consume different inputs text-only (Chi al., 2021), text+l (Xu al., 2020, 2021b; Wang al., 2022), multi-modal (Appalaraju al., 2021; Xu al., 2021a, 2022; Huang al., 2022) that fuse input from text, layout, and visual features. as well as image-only models (Kim et al., 2022). Em- pirical results on the two new datasets reveal that image-only models have large improvement oppor- tunities unlike what is reported on RVL-CDIP (Kim et al., 2022), and multi-modal models have lim- ited cross-lingual generalization ability. Our main contributions are follows:
overview in this section and leave the details to the survey paper by Cui (2021). Technically, any pre- trained text models (e.g.. BERT or GPT-3) can be applied to handle document images after being pro- cessed by OCR tools. However, initial work by Xu et al. (2020) opened up the exploration of including additional modalities such as layouts and images for handling documents. Since then, many layout- aware models and pretraining tasks have been pro- posed in the literature (Li et al., 2021; Hong et al.. 2022; Wang et al., 2022; Li et al., 2022a; Hao et al., 2022), followed by using images as additional in- put for better models that exploit broader set of signals from the input (Appalaraju et al., 2021: Xu et al., 2021a; Huang et al., 2022; Biten et al., 2021; Appalaraju et al., 2023).
Recently, Document AI models using only im- ages inputs (or OCR-free models) have been attracting attention from researchers and practition- ers (Rust et al., 2022; Kim al., 2022; Lee et al., 2022). As a results of not being dependent on OCR, such models potentially avoid the propagation of OCR errors. Furthermore, these models do not have fixed vocabulary and can technically be ap- plied to any language without out-of-vocabulayy concerns. Nevertheless, most models support only English, and only limited number of multilingual models have been explored in the literature. Ad- ditionally, such models are not thoroughly bench- marked on multilingual datasets due to lack of ap- propriate datasets. We address this by curating new multilingual datasets for document classification.
We introduce two new document image classi- fication datasets which complement the do- mains and languages covered in the com- monly used document image classification dataset (Lewis et al., 2006).
We evaluate Document AI models on the newly created datasets and address limitations of such models on both multi-label and multi- class document classification tasks where un- derstanding of document contents crucial.
Multilingual Evaluation Datasets for Document AI Models
We evaluate both the multilingual and cross-
lingual generalization ability of Document AI models showing challenges in transferring across syntactically distant languages.
A common benchmark dataset for evaluating Doc- ument AI models on document classification tasks is RVL-CDIP (Lewis et al., 2006). However, we gue that using single benchmark dataset naturally inhibits our understanding of the task as well as the solutions built for it. Specifically, we argue that RVL-CDIP has the following limitations:
2
Related Work
Models designed for document image understand- ing tasks, often referred to as Document AI models, appear in various different forms. We give brief

---

Aligned Text Blocks
Right-to- -oo--o---LLof script
Tables
Images
Column
Format
(a) MULTIBURLEX- DDC Examples
(b) WIKI-DOL Examples
Figure| I: Example documents from the newly curated datasets. MULTIEURLEX-DEE contains documents in different languages with highly-aligned layouts across European languages in multi-column format. WIKI-DoC contains documents with rich non-textual contents (e.g., tables and images) and documents in a broader set of languages, including Arabic which follows right-to-left writing.
1. It only includes English documents, thus lim- iting our understanding of multilingual docu- ment understanding models (Xu al., 2021b) and their cross-lingual generalization ability. 2. The labels assigned to documents focus on the type of the document (e.g.. "emails", "in- voices". "tax form"), which does not neces- sarily require deeper understanding of the contents of the documents.
understanding the document such as headers, foot- ers, and tables. Further, since the same law ex- ists in multiple languages, layouts are also aligned across languages. Figure la shows an example of documents in this dataset. With this dataset, we aim to marry the recent progress in legal NLP (Hendrycks et al., 2021; Kementchedjhiiva and Chalkidis, 2023; Chalkidis et al., 2022, inter alia) with the progress made in visually-rich docu- ment understanding.
Larson et al. (2023) provide an in-depth analysis on these limitations of RVL-CDIP and recommend that new datasets for document image classifica- tion should be multi-label to handle the naturally occurring overlap across labels, large-scale with 100+ classes, and multilingual to test the ability of models to transfer across languages (besides being accurately labeled). We introduce newly curated datasets covering these desired characteristics to complement the limitations of RVL-CDIP.
Labels in MULTIEURLEX-DDE from the EUROVOC taxonomy and are hierarchi- cally organized into three increasingly specific lev- els.? For example, for the label 'Agri-foodstuffs' at level 1, the corresponding labels at level 2 are *Plant Product' and 'Animal Product' where 'Plant Product' is further divided into 'Fruit', 'Vegetable' and 'Cereals' at level 3. We focus evaluating models at level-3 (total of 567 classes) to make our results comparable to those reported in Chalkidis et al. (2021).
are derived
3.1 Newly Curated Datasets
We put together MULTIÉURLEX-DOO follow- ing multi-step process: 1) We download the PDFs for all languages from EurLex website by using the CELEX ID3 for each PDF obtained from ' released by Chalkidis et al. (2021) where it does not include PDFs. 2) We convert each page of the PDF into an image (currently, we only use image of first page of each document for modelling). 3) We apply OCR to extract words and bounding boxes for each image from the previous step. 4) Finally,
We now introduce our two newly curated datasets and the summarized comparsion between RVL- CDIP and the two newly curated datasets are in Table I and we give a summarized overview of the datasets in Figure 1.
Multi-EurLex PDFs (MULTTEURLEX-DOC
Our first dataset is MULTTEUULLEX EXCDDD, multi- lingual and multi-label dataset consisting of Eu- ropean Union laws covering 23 European lan- guages in their original PDF format. Documents in MUL.TUURLEX- consist of PDFs with lay- outs such as single column or multiple columns, and include many structural elements essential to
Phttp: /tttto//eeurooc..oo europa.
"The English example in Figure ia at https://eur-1 ex. europa. aelea.eu/leee www.egallcoonene wonnten/////X///////rrr 32005D8213
"https://huggingfaatt http://wgginngace..coo//sssss//s hot.e/tssttttt/tttieeer..ee.

---

we add the labels for each document by using the dataset by Chalkidis al. (2021). In other words, the texts and bounding boxes come from Step above and the images come from Step 2 above. We only reuse the CELEX ID and labels from the dataset by Chalkidis et al. (2021). Since we ap- plied OCR and directly extracted texts from the original source PDF documents and converted PDF documents into images, any structured elements like tables and figures are retained in our curated dataset unlike the dataset created by Chalkidis et al. (2021) where the HTML mark-up texts are used instead of PDFs.
Model
L
Langs
Pretrain Doc. Images
InfoXLM
100
LiLT
100 IIT-CDIP
LayoutXLM
7
53
IIT-CDIP CC
Docformer
V
IIT-CDIP
Donut
IIT-CDIP Syn. Wiki
Table 2: Overview of the Document models used this work with their corresponding input modalities (T: text, L: layout, I: image). Most models are pretrained on English-only IIT-CDIP dataset (Lewis et al., 2006). LayoutXLM is also pretrained documents from Com- mon Crawl (CC) and Donut on synthetic Wikipedia documents (Syn. Wiki). InfoXLM not pretrained on document images.
Rendered Wikipedia Articles (WIKI-DOC) We additionally curate the WIKI-Doc dataset which complements MULTIEURLEX-DDG in the follow- ing ways: (1) It contains documents other than legal domain, and (2) It contains documents in non-Furopean languages written in scripts other than Latin. Most Document Al models are OCR- dependent, and hence they suffer from error propa- gation due to incorrect OCR. This is especially an issue for languages not written in Latin or Cyrillic scripts since such languages are reported to have fewer OCR errors (Ignat et al., 2022). An exam- ple is Arabic, language with right-to-left writing, which results in higher OCR error rate (Heggham- mer, 2021). To complement the language and do- main coverage in MULTIEURLEX- Doc. we use the documents of the dataset created by Sinha et al. (2018) by scraping and rendering Wikipedia arti- cles. See Appendix B.1 for the full details on the creation steps.
setup, where we train and evaluate on the same lan- guage, and 2) cross- lessslligual setup, where we train and evaluate on different languages.
In the intralingual setting, we are interested in the accuracy difference between multi-modal mod- els and uni-modal (i.e., text- image-only) models. Specifically, are interested in knowing whether multi-modal models achieve higher accuracy in content classification datasets. Kim al. (2022) report that the image-only model is more accu- rate than a multi-modal model on RVL.-DDIP, but it is unclear if this will hold true for the datasets introduced in this work, as they require deeper understanding of the text.
In the cross-lingual setting, in the cross- cross- linssll generalization ability of Docu- ment Al models. We experiment on the (zero-shot) cross-lingual transfer setting, where we only have English training and validation documents. This is a practical setting since labeled document images are often more scarce than plain texts, especially in -on-EEnlishh languages. Both uni-modal and multi- modal have potential strengths in the cross- poss--lnggual transfer setting, On one hand, multi-modal models consume various input signals that are expected transfer across certain languages (e.g., the layout information as reported by Wang et al. (2022)). On the other hand, image-only models can be applied to languages even unseen during pretraining as they are not dependent on the vocabulary of the model.
interested
4 Experiments
We now empirically explore both intra- and cross- lingual generalization ability Document Al mod- els for document classification for multiple lan- guages. We ask the following research questions:
RQI: Do Document AI models, specifically those that are multilingual in nature, perform equally accurately across languages?
RQ2: Can Document AI models classify docu- ments in a cross-lingual transfer setting where we train on English and evaluate on other languages?
4.1 Experimental Setup
4.1.1 Models
We conduct experiments under two different set-
We now describe the Document Al models we experiment with in this paper. Our focus is on experimenting with diverse models that consume different input modalities (text, and/or document
tings answer these questions: 1) intralingual
httpp:: haap::://huggingaacc. http://gggggaaaee..ootatttss//// www.aaaatttt//mlttt........ eurlex #source-das data

---

layout, and/or images) and models that support multiple languages (Table 2). We choose represen- tative model candidates for different settings. In all cases, we use the base model unless specified.
requires document images as inputs; this removes the dependency on OCR extract text and layout information. Hence, Donut can be even applied to languages unseen during pretraining stage as it no longer requires a tokenizer. Except for Donut, all other models considered in the paper are encoder- only models. We experiment an encoder-only version of Donut, where we remove the decoder layers from the original model and replace them with a single linear classification layer. Donut is pretrained with an OCR-like task where the input the document image and the previously decoded text and the output is the OCR output.
InfoXLM (Chi et al., 2021) (Text-only) In- foXLM is a text-only RoBERTa-lik model which uses the same architecture as XLM-R (Conneau et al., 2020). We select InfoXLM instead of other text-only models following the prior work by Wang et al. (2022).
LiLT + InfoXLM (Wang et al., 2022) (Text + Layout) LiLT uses two different Transformers, one dedicated to text (initialized with multilin- gual Transformer model checkpoint) and another for layout, allowing for plug-and- sang-anddplyy of arbitrary text-only models with the same architecture while keeping the layout Transformers part of the model. As result if initialized with a multi-lingual text model (such as XLM-R), LiLT can be pre-trained with only English documents but fine-tuned on any language. We follow Wang et al. (2022) and use InfoXLM as the underlying text model.
4.1.2 Hyperparameters
The hyperparameters of each model are tuned using English and one other non-English language, using the hyperparameters from the original papers as a recommendation. We use class-balance training for all experiments in order to handle class imbal- ances. The chosen hyperparameters for each model are in Appendix C. All experiments are conducted on AWS p3. 6xxaare instances with 8 V100 16GB memory GPUs.
DocFormer (Appalaraju et al., 2021) (Text + layout + image) DocFormer is encoder-only Transformer model with a CNN for visual feature extraction. uses multi-modal self-attention to fuse visual and layout features at every layer to enforce an information residual connection to learn better modal feature representations. We fol- low Appalaraju et al. (2021) and use the model with an attached linear classification layer.
4.1.3 Dataset Preprocessing
For both MULTIEURLEX-DOC and WIKI-Doc, we use pdf2image to convert a PDF into cor- responding image using dpi of 300. We then use Tesseract 5.1 (Smith, 2007) to run OCR and extract words and bounding boxes. We use Transformers library (Wolf et al., 2020) for implementation. Fi- nally, only consider the first page of each PDF document, as similar truncation approaches have been shown be strong baselines for text classifi- cation (Park et al., 2022). We leave long document classification for future work.
LayoutXLM (Xu et al., 2022) (Text + layout + image) LayoutXLM uses the same architecture Lay (Xu et al., 2021a), which is mul- timodal Transformer model which extends Lay- outLM (Xu et al., 2020) by adding the document image an input to the model in addition to text and layout inputs. LayoutXLM is pretrained on PDFs from 53 languages extracted from Common Crawl thus enabling it to process documents in multiple languages. Following multilingual pre- training convention, the data follows exponential sampling to handle the imbalance across languages.
Preprocessing Since MULTIEURLEX Doc is a multi- dataset, we convert the list of labels into multi-hot vectors. Refer to Appendix B (Table 11) for language specific data splits.
MULTIEURLEX-DOC
WIKI-DOC Preprocessing We make WIKI- Doc challenging for existing Document Al models by performing a series of preprocessing steps on
Donut (Kim et al., 2022) (Image-only) Donut is an encoder-decoder model where the encoder is Swin Transformer (Liu al., 2021) and the de- coder is mBART (Liu et al., 2020). Donut only
"Our imitial experiments on the on English WIKI-Doc in 10-shot setting scored macro of 9.272.92 222..2 which is significantly lower than the encoder-only version which scored 26.633.06. Therefore, we focus on the encoder-only version. " pettps://pyp...ot wwcpypp..rgg/aeeo wwwpo////pppppp/////r
version
"https://comoncraat

---

Models
Eurlex
Wiki
what is reported by Kim al. (2022) where Donut reported be better than the multi-modas model (i.e., LayoutLM v2) on the RVL-CDIP dataset. To further analyze these results, we compare Donut and LayoutXI.M under the few-shot setting de- scribed in Section 1.3. We observe that the macro F] gap between the two models is significantly smaller in the full-shot setting (90.13 vs. 94.40) than the few-shot setting (26.63 vs. 82.18, full sults at Appendix E.1). Thus, having large num- ber of finetuning examples is crucial for obtaining high accuracy using Donut.
InfoXLM
6
91.04..17
LiLT
61.562.6
11
Lay
65.67o.5
94.400.
Donut
45.27s.3
10.130.58
DucFormer
63.46o.6
94.77%.110
Table 3: English results (en->en) on MULTIEURLEXX Doc (Eurlex) and WiKI-DoC (Wiki).
the curated dataset: 1) Few-shot Setting: We sub- sample each class in the training split to be 10 train- ing examples per class and created 5 different splits with different random seeds. This setting aims to test how much models learn from few examples in contrast to the RVL-CDIP dataset which includes 25K examples per class. 2) We further merge the subset of 219 classes curated by Sinha et al. (2018), which scored higher class-wise Fl scores than threshold", into a single "Other" class. The Wikipedia language links are used retrieve and curate the corresponding article in other languages. We further filter and keep Wikipedia articles which licenses. The resulting dataset statistics after filtering and preprocessing are shown in Appendix B (Table 10).
Given the results on these two datasets, we fo- cus the four models pretrained on multiple lan- guages in the remaining experiments study the cross-lingual transferability.
4.3 Non-English Intralingual Results
4.3.1 MULTIEURLEX-DOS Results
Next, we evaluate the models on other languages, starting with MULTIEURLEX Doc. The average mRP scores averaged across languages in Table 4 show that LayoutXLM is the most accurate. Both InfoXLM and LiLT perform poorly on certain lan- guages either due to low score or high variance whereas we find LayoutXLM to be relatively con- sistent in its performance across languages. On the other hand, the encoder only Donut results are significantly lower (36.64), which can likely be at- tributed to the fact that the multi-label classification task requires identifying certain spans or distribu- tion of words that indicate specific attribute/label of the document. An image-only model like Donut is expected to struggle in capturing such nuances from the visual document structure and correlate them a group of labels describing the document contents rather than document types, without the knowledge of the words comprising it.
only use images with
Evaluation Metrics For multi-label classifica- tion, we use mean R-Precision (mRP) as the evalu- ation metric following Chalkidis et al. (2021). To compute mRP, we rank the predicted labels in de- creasing order of the confidence scores and com- pute Precision @ where k is the number of gold labels for the given document. For each model and language pair, we report the average mRP and stan- dard deviation across different runs. For multi- class classification, we report the average macro Fl scores and standard deviations.
4.2 English Results
We can also break down the results in Table 4 by language groups: Germanie (da, de, nl, sv), Ro- mance (ro, es, fr, it, pt), Slavic (pl, bg, es), and Uralic (hu, fi, et). InfoXLM and LayoutXLM have similar mRP on Germanic and Romance languages except Romanian (ro). However, LayoutXLM is scoring higher accuracy on average than InfoXLM on Slavic (hu: 63.87 vs. 58.84, fi: 63.52 vs. 63.46. et: 63.43 vs. 60.37) and Uralic (pl: 63.26 vs. 61.12, bg: 63.67 vs. 14.23, es: 63.60 vs. 40.99) languages.
We first experiment on the English portion of MULTIEURLEX-DoG and WIKI-DoC to confirm whether multilingual models are competitive with the English model scoring high accuracy RVL- CDIP i.e., DocFormer (Appalaraju et al., 2021).
Results in Table confirm that InfoXLM and LayoutXLM yield very similar results on English compared to DocFormer, and in fact are slightly more accurate on MULTIEURLEX- -DC. On the other hand, the accuracy of the OCR-free Donut model in English is relatively low. This is unlike
We do not see a strong correlation between the amount of training data in a given language and
"set to 0.8 hased on the development set

---

Models
da
de
al
Sv
10
es
pt
InfoXLM
63.161.2
63.890.9
62.82 62.82a.s
64.081.1
28.3124.7
63.21.
12..
64.741.4
64.011.5
LiLT
42.5728.9
61.481.3
59.142.9
63.78a.s
1.01o.3
63.11.
42.04s0.6
62.84o.7
58.10
LayoutXLM
65.17
65.09o.5
65.07p.2
61.761.0
64.151.1
65.25o.3
65.360.7
65.22n.3
64.266..2
Donut
39.22a.9
40.37s.s
40.481.6
35.534.7
26. 10..6
34.99s.a
41.835.4
41.325.4
40.187.7
Models
pl
bg
hu
fi
el
Avg
InfoXLM
61.12
14.28a.
40.9934.9
58.840.9
63.46 461.0
63.95o.7
60.37o.a
56.89
LiLT
58.850.5
1.55g.1
37 37.60st
39.27
61.750.6
60.45
59.266..
49.08
LayoutXLM
63.26 260.7
63.67o.s
63 63.6n.3
63.87o.s
12.
62. 190.3
63. 23.4
64.32
Donut
33. 23.262.7
27.851.7
3
34.03s.7
31.921.2
43.560.1
34.830.5
36.64
Table 4: Intralingual results (X X) MULTIEURLEX- Doc level 3. We report the average and standard
deviation of mRP scores across different random seeds. "Avg" indicates average across all languages.
Models
es
fr
it
de
pt
zh
ja
Fhot Setting
InfoXLM
77.970.91
77.33g.22
78.281.4u
78. 78.100.a9
76.841.49
72.93p.83
74.974.43
76.111.94
LiLT
77.211.12
76.24a.48
76.171.s7
76.840.97
75.90a.45
70. 692.69
75, 75.41o.73
75.052.47
LayoutXLM
71.807.29
77.82
64.956.70
75.852.18
76.63g.06
65.314.79
70.163.06
60.494.09
Donut
28.851.88
32.77s.76
36.952.49
21.813.02
27 27.503.12
24.071.43
28.732.25
38.713.82
Full-shot Setting
InfoXLM
A11.73
A11.97
A9.25
A11.66
A11.54
A12.37
A12.88
A11.03
LiLT
A12.28
A11.89
A11.97
A11.85
110.53
A15.89
A11.98
A11.47
LayouIXIM
A16.52
A10.52
A23.52
A13.16
A8.98
A19.94
A15.98
A25.05
Donut A.2.52
A31.45
2.2..2
A34.32
222.98
2.38.55
3.30.9
2.2.94
Table 5: Few- and full-shot intralingual results (X - X) on WIKI-Doc. We report the average and standard deviation of macro Fl scores across 5 different random shots. For the full-shot selting, report the score gap between the few-shot setting. The largest score gap between full- and few-shot setting is Arabic (ar) for LayoutXLM.
LayoutXLM's (model with best intra-lingual re- sults) intra-lingual accuracy. For Greek (el), there are approximately 55K training examples (see Ap- pendix B Table 11) but we see lowest mRP score 62.19. On the contrary, there are languages such as Romanian and Bulgarian with approximately 16K training examples for which the mRP score is 64.15 and 63.67 respectively. For InfoXLM, we do see some correlation between the amount of training data and intra-lingull performance across languages. We conjecture that it is tied to the fact that LayoutXLM has been pre-trained with layout information so it is able to perform well even with less amount of data whereas InfoXLM being a text only model needs more data for the task.
higher OCR error rate than languages that use Latin scripts (Hegghammer, 2021). Second, the Arabic pretraining data is relatively smaller than the other eight languages when pretraining LayoutXLM (Xu et al., 2022). Lastly, the layout position of Arabic texts are reversed unlike the other eight languages, making it harder for LayoutXLM to learn 2D po- sition embeddings during pretraining. Figure 2 further shows that semantically close classes are of- misclassified by LayoutXLM such "Fish" "Amphibian" and "Mollusca" vs. "Crustacean".
In contrast, Donut scores are significantly lower than other models (Table 5). This is likely due to Donut being pretrained only English with real PDF documents (i.e., IIT-CDIP) and on synthetic Chinese, Japanese, and Korean documents (Kim et al., 2022). Also, models consuming images (i.e., Donut and LayoutXLMM are not the most accurate, especially when the input text contains strong sig- nals for classifying documents."" 10
4.3.2 WIKI-DOC Results
We now turn to experimenting on WIKI-Doc which includes non-European languages such as Chinese, Japanese, and Arabic (Table 5). Focusing on the few-shot LayoutLLM results, the largest accuracy gap with other models is on Arabic (< 15 Fl point between InfoXLM). We hypothesize that this is due to multiple factors. First, Arabic has a
"We didn't 't include other image-only models like Docu- ment Image Transformer (Li et al., 2022b, DiT) since Donut was seen to show much stronger performance than DIT on RVL-CDIP dataset (95.30 for Donut (Kim et al., 2022) vs

---

Models
da
SV
10
es
p
InfoXLM
51.281..
53.271.3
46.471.4
47.912.
48.732.
52.632.4
52. 22.22
47.7b2.3
47.981.0
LiLT
43.946.1
44.30s.s
38.75
42.117.3
43.32s.7
47.414.6
43.96a.3
45.43a.4
42.996.5
LayoutXLM
11.29
16.261.5
19.1
17.751.
50.152.1
52.351.1
52.500.7
1..331.6
48.461.7
Donut
16.972.4
14.086.7
16.68o.9
18.
21.412.7
18.551.
19.022.4
18.361.1
20.002.1
Models
pl
bg
hu
el
Avg
InfoXLM
41.62o.6
45.782.3
46.352.2
45.74
42.865.4
34.872.4
41.78s.7
46.70
LiLT
35.4
40.776.9
37.286.o
39.03e.6
34.02y.0
27.17
34.17.1
40.02
L
41.282.7
47.311.3
42.322.2
39.36p.9
31.851.
27. 27.151.4
38.371.
44,51
Donut
11.452.6
6.580.8
12.942.8
7.530.5
9.392.3
5.560.9
15.162.6
14.49
Table 6: Cross-linguss results (en X) MULTIÉURLEX- at level We report the average and standard
deviation of mRP scores across different random seeds. "Avg" indicates average across all languages.
Models
fr
it
de
pt
zh
ja
Feo- Setting
InfoXLM
59.361.07
60.37a.94
50.171.75
59.171.11
58.961.u1
44.26u.
39.051.0p
39.301
60.161.03
19.19..00
49.731.72
19.141.10
67.82a.s0
44. 59..5..
39.571.a1
38.231.96
LayoutXLM
49.734.38
48.315.96
42.075.83
47.345.48
16.895.14
29.214.20
27.655.82
24.045.98
Donut
4.701.09
3.45
4.650.84
1.91
3.751.18
2.260.26
2.50p.75
2.37p.83
Full-shot Setting
InfoXLM
A13.60
A10.77
A18.44
A11.87
A11.30
A8.12
A6.28
A7.79
LiLT
A13.90
A11.98
A16.82
A10.63
A13.04
49.86
A7.14
A5.95
LayoutXIM
A9.26
A10.23
A11.75
A7.26
A9.17
A0003
55.39
r3.04
Donut
A3.20
A3.40
23.94
A3.01
2..24
255.18
2.0.52
00.05
Table Cross-linguns transfer Macro F1 results + X) on WIKI-DoG with full- and few-shot (10-shot) setup. We report the accuracy difference between few-shot and full-shot setting. The scores either stays par decrease in Chinese, Japanese, and Arabic for LayoutXLM and Donut when comparing the full and few-shot (10-shot) setup.
4.4 Cross Cross-Lingual Experiments
We now explore the cross-lingual generalization ability Document AI models by evaluating the models in cross-lingual transfer setting, i.c.. fine- tuning only on English and directly evaluating on non-English target languages (en -> X).
Ac Academittame do
Ampricane rolo Fraper
4.4.1 MULTIEURLEX-DRR Results
In Table 6, we observe that even though Lay- outXLM performs the best on individual languages in intralingual setting, it does not generalize as well as InfoXLM in cross-lingual setting. This result is bit surprising because the parallel documents across languages in the dataset have the same lay- out information thus the layout and image features should be easily transferable across languages.
gnerget
juize
Weeath
0.0
0.2
0.4
0.8
0.8
1.0
Across language groups, we can see that both InfoXLM and LayoutXLM perform similarly on Germanic languages (da, nl, sv) except German (de) where InfoXLM does much better. On the other hand, LayoutXLM performs slightly better (than InfoXLM) on Romance languages (ro, es, fr, it, pt). However most of the accuracy gap between the two models is introduced due to Uralic lan-
Figure The confusion matrix of the LayoutXLM prediction results on the first 35 classes in Arabic WIKI- Doc under intralingual setting. Semantically close classes (e.g., "Mollusca" "Crustacean") are often misclassified, challenging LayoutXLM on this dataset.
92.69 for DiT-Large (Li al., 2022b)) and publicly available DiT checkpoints pretrained in multiple languages.

---

guages (hu, fi, et) where InfoXLM yields far better results.
Data
Model
SYN
PHON
INV
P
P
S
S
InfoXLM
.44
.65
-336
-27
.23
.21
4.4.2 WIKI-DOC Results
Eurlex
LayoutXLM
-56
.68
-60
-52
.07
.06
From Table 7, we also see that Donut cannot gen- cralize actoss languages. These and other results from this section indicate that there large areas of improvement on either the model the pre- training strategy for Donut to generalize across langauges. Finally, Table 7 also shows the limited cross-lingual transferability due to the accuracy drop for LayoutXLM in Japanese 5.39) and Ara- bic 3.04) when increasing the number of English training examples from few-shot (10) to full-shot setup. We further analyze the correlation typo- logical features in the next section.
InfoXLM
.91
.96
.32
.4
.71
.52
Wiki
LayoutXLM
.88
.88
.27
.34
.75
.57
Table 8: Correlation analysis the cross-lingual trams- fer gap and typological distances using syntactic (SYN). phonological (PHON), and phonetic inventory (INV) features. Spearman (S) and Pearson (P) correlations are used. The highest correlations are in bold.
guages in WIKI-DoC. Similar trends are observed on MULTIEURLEX-DO in Table 6 when com- paring LayoutXLM and InfoXLM for the most syntactically-diite languages (es: .66, hu: .60) among the 16 languages.
4.5 Correlation with Typological Features
To further understand the cross- cross-lingual transfer re- sults in Tables 6 and 7, we look at the correlation between these results and the typological features (syntactic, phonological, and phonetic inventory features) of the languages involved. A higher corre- lation implies that the cross- cross-lingual gap is harder to bridge via those features. Inspired from Lauscher et al. (2020), we analyze the correlation of cross- lingual transfer gap with the typological distance between the source and target languages. Because the test sets of the newly curated datasets are not completely parallel across all languages, mea- sure the accuracy gap between models trained on source and target languages, and evaluate those on the same target language test set. Specifically, the correlation calculated among the two set of numbers. The first set is the accuracy gap between a model trained on English and evaluated on lan- guage X (en + X) and model trained on language X and evaluated on language X (en + X). The sec- ond set is the typological distance between English and the target languages where we the pre- computed typological distance between languages from LANG2VEC (Littell et al., 2017).
Conclusion
In this paper, we curated two new multi- lingual document image classification datasets, MULTIEURLEX-DD DOC and WIKI-DOC, to evalu- ate both the multilingual and cross-lingual general- ization ability of Document AI models. Through benchmarking the two newly curated datasets, show strong intralingual results across lan- guages of the multimodel model but also show the limited cross- cross-lingual generalization ability of the multimodal model. Furthermore, the OCR-free or image-only model showed the largest gap between the best performing models, showing large areas of improvement in datasets which require deeper content understanding from texts.
Future work in this space should investigate im- provement strategies of molli- and OCR-free Document Al models to enable them to achieve a deeper understanding of the text from document images. Finally, the curated datasets still cover only small subset of languages spoken in the world. Future work should expand the datasets and experi- ments more diverse set of language families.
The correlation analysis results (Table 8) show that the transfer gap is highly correlated with the syntactic cosine distance between the source and the target language. This further explains the gap between few-shot and full-shot cross-lingual trans- fer results in Table 7 where increasing the train- ing examples in the source language (i.e., English) hurts the accuracy of LayoutXLM in Japanese and Arabic, which have the highest syntactic distance from English (ja: .66, ar: .57) among the 8 lan-
A Acknowledgements
We sincerely thank the anonymous reviewers and the colleagues at AWS AI Labs for giving construc- tive feedback on the earlier versions of this paper. We also thank Philipp Schmid for the detailed and educative tutorial and the associated codes to ex- periment with Document Al models.!!
"https://ww/r phi schmii.. de/ tee/fine-turi tuning donut and https::/wwww wwwpschhiidpccmcm d. de/ de/fine-tee de/fine-tuning- lilt

---

Limitations
work (e.g., (Chen al.. 2023b)) and encourage further rescarch on this topic by the community.
Low-Resource Language Coverage The lan- guage covered by the two newly curated datasets is limited in terms of the coverage of language families (e.g.. Afro-Asiatic family is not covered) especially on low-resource languages. We intro- duce WIKI-DoC to extend the language coverage beyond European languages which are covered by MUI.TEEULLEX-DDR but the dataset size becomes too small due to frequent missing laneralan links in Wikipedia and we have not covered low- resource languages in the newly curated dataset.
References
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha, 2021. Docformer: End-to-end transformer for document understanding. 2021 IEEE/CVF International Conference Com- puter Vision (ICCV), pages 973-983
Srikar Appalaraju, Peng Tang. Qi Dong, Nishant Sankaran, Yichu Zhou, and R. Manmatha. 2023. Docformerv2: Local features for document under- standing.
English as the Source Language The cross- lingual experiments conducted in the paper uses English as the source language to train the model. While it's true that choice of source lan- guage changes the downstream task results signifi- cantly (e.g., Lin al. (2019)), we choose English the source language from practical perspective and leave exploration of the choice of source lan- guages in Document Al models as future work.
Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju. and R. Manmatha. 2021. Latr: Layout- aware transformer for scene-text vqa. 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 16527-16537.
Maas Chalkidis, Manos Fergadiotis, and Ion Androut- sopoulos. 2021. MultiURLEEX - a multi- multi-lingual lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6974-6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Diserepancy in Pretraining Data among Models We use the pretrained Document Al models out-of- the-box without any additional pretraining. As a result, there are slight discrepancies in the corpus used for pre-training each model (Table 2).
Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310-4330, Dublin, Ireland. Association for Computational Linguistics.
Task Coverage Our curated dataset is specifi- cally designed for multi-class and multi-label doc- ument classification in multiple languages. To the best of our knowledge, XFUND (Xu et al., 2022) and EPHOIE (Wang et al., 2021) are the only publicly available non-English datasets to evalu- ate Document AI models. We therefore encour- age the research community to build diverse set of document image datasets to cover various tasks in multiple languages.
Catherine Chen, Zejiang Shen, Dan Klein, Gabriel Stanovsky, Doug Downey, and Kyle Lo. 2023a. Are layout-infused language models robust layout dis- tribution shifts? a case study with scientific docu- ments. In Findings of the Association for Computa- tional Linguistics: ACI. 2023, pages 13345-13360, Toronto, Canada. Association for Computational Lin- guistics.
Diverse Document Layouts The document lay- outs in our newly curated dataset are mostly static except for MULTIEURLEX- -Doc being multi- column and some layout variation based on Wikipedia templates in WIKI-DoC. Layout-aware models tend to have the issue of layout distribution shifts (Chen et al., 2023a) and such issues may not be captured in the newly curated datasets.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo. Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Yi Tay. Siamak Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic. Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi. Bo Pang. Ceslee Montgomery, Paulina Pietrzyk. Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic. Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lu- cas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua
Larger Models The size of all models bench- marked in this paper are < 400M (Appendix A) and relatively smaller compared to models explored in the recent literature. Though not focused on doc- ument image classification, we leave to other

---

Zhai, Neil Houlsby, and Radu Soricut. 2023b. PaLI- X: On scaling up multilingual vision and language model.
ACL 2022, pages 1164-1174, Dublin, Ireland. As- sociation for Computational Linguistics.
Yova Kementchedjhieva and Ilias Chalkidis. 2023, An exploration of encoder-decoder approaches to multi- label classification for legal and biomedical text. In Findings of the Association for Computational Linguisties: ACI. 2023, pages 5828-5843, Toronto, Canada. Association for Computational Linguistics.
Zewen Chi, Li Dong, Furu Wei, Nan Yang. Saksham Singhal. Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information theoretic framework for cross loossll language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 3576-3588, On-
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeong Yeon Nam, Jinyoung Park, Jinyeong Yim, Won- scok Hwang, Sangdoo Yun, Dongyoon Han, and Scunghyun Park. 2022. Ocr-free document under- standing transformer. In European Conference on Computer Vision (ECCV).
line. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Stefan Larson, Gordon Lim, and Kevin Leach. 2023. On evaluation of document classifiers using RVL-CDIP. In Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 2665-2678. Dubrovnik, Croatia. Association for Computational Linguistics.
Lei Cui. 2021. Document Al: Benchmarks, models and applications (presentation icdar 2021). DII. workshop in ICDAR 2021.
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4483-4499, On- line. Association for Computational Linguistics.
Jean-Philippe Thiran Guillaume Jaume, Hazim Ke- mal Ekenel. 2019. Funsd: A dataset for form under- standing in noisy scanned documents. In Accepted to ICDAR-OST.
Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Boyang Li, and Mu Li. 2022. Mixgen: new modal data augmen- tation. 2023 IEEF/CVF Winter Conference on Ap- plications of Computer Vision Workshops (WACVW), pages 379-389.
Kenton Lee, Mandar Joshi, Tulia Ture, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandel- wal. Peter Shaw, Ming-Was Wei Chang, and Kristina Toutanova. 2022. Pix2struct: Screenshot parsing as pretraining for visual language understanding.
Thomas Hegghammer. 2021. Ocr with tesseract, ama- zon textract, and google document ai: a benchmark- ing experiment. Journal of Computational Social Science, 5:861-882.
D, Lewis, G. Agam, S. Argamon, O. Frieder, D. Gross- man, and J. Heard. 2006. Building a test collection for complex document information processing. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '06, page 665-666, New York, NY, USA. Association for Computing Machinery.
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nip dataset for legal contract review. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume Curran.
Chenge Li, István Fehérvári, Xiaonan Zhao, Ives Macèdo, and Srikar Appalaraju. 2022a. Sectek: Very large-scale open-set logo recognition with text-aware metric learning. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), pages 587-596.
Teakgyu Hong, DongHyun Kim, Mingi Ji, Wonseok Hwang. Dachyun Nam, and Sungrae Park. 2022. BROS: A pre-trained language model for understand- ing texts document.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Lay:utlmv3; Pre-training for doc- ument ai with unified text and image masking. In Proceedings of the 30th ACM International Confer- ence on Multimedia.
Chenliang Li. Bin Bi, Ming Yan. Wei Wang, Songfang Huang, Fei Huang, and Luo Si. 2021. StructuralLMM: Structural pre-training for form understanding. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 6309- 6318, Online. Association for Computational Lin- guistics.
Oana Ignat, Jean Maillard, Vishrav Chaudhary, and Francisco Guzmán. 2022. OCR improves machine translation for low-resource languages. Findings of the Association for Computational Linguistics:

---

Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. 2022b. DiT: Self-supervised pre-training for document image transformer. In ACM Multimedia 2022.
Processing, pages 817-823, Brussels. Belgium. As- sociation for Computational Linguistics.
R. Smith. 2007. An overview of the tesseract oct engine. In Ninth International Conference on Document Anal- ysis and Recognition (ICDAR 2007), volume 2, pages 629-633.
Yu-Hsiang Lin, Chian-Yu Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junx- ian He, Zhisong Zhang, Xuezhe Ma, Antonios Anas- tasopoulos. Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for gross-linguse learn- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125-3135, Florence, Italy, Association for Compu- tational Linguistics.
Jiapeng Wang, Lianwen Jin, and Kai Ding. 2022. LiLT: A simple yet effective Language-indepeeen transformer for structured document understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7747-7757, Dublin, Ireland. Association for Computational Linguistics.
layout
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and .ori Levin. 2017. URIEL and lang2vee: Representing languages as typological, geographical, and phylogenetic vectors. In Proceed- ings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 2, Short Pupers, pages 8-14, Valencia, Spain. Association for Computational I Linguistics.
Jiapeng Wang, Chongyu Liu, Lianwen Jin, Guozhi Tang, Jiaxin Zhang. Shuaitao Zhang, Qianying Wang, Yaqiang Wu, and Mingxiang Cai. 2021. Towards robust visual information extraction in real world: New dataset and novel solution. In Proceedings of the AAAI Conference on Artificial Intelligence.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis. and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Ze Liu, Yutong Lin, Yue Cao, Han HU, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. 2021. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2021a. Lay: Multi-modd pre-training for visually-rich document understanding, In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 2379,2591, Online. Association for Computational Linguisties
Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V. Jawahar. 2020. Docvqa: A dataset for vqa on document images. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1199-2208.
Hyunji Park, Yogarshi Vyas, and Kashif Shah. 2022. Efficient classification of long documents using trans- formers. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers). pages 702-709, Dublin, Ireland. Association for Computational Linguistics.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020. Layoutim: Pre-training of text and layout for document image understanding.
Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jacheung Surh, Minjoon Sco, and Hwalsuk Lee. 2019, Cord: A consolidated receipt dataset for post-ocr parsing.
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang. Yi- juan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. 2021b. Layoutxlm: Multimodal pre- pre-training for mul- tilingual visually-rich document understanding.
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Eliz- abeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2022. Language modelling with pixels. arXiv preprint.
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yi- juan Lu, Dinei Florencio. Cha Zhang, and Furu Wei. 2022. XFUND: A benchmark dataset for multilin- gual visually rich form understanding. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3214-3224, Dublin, Ireland. Association for Computational Linguistics.
Koustuv Sinha, Yue Dong, Jackie Chi Kit Cheung, and Derek Ruths. 2018 A hierarchical neural attention- based text classifier. Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language

---

A Model Details
Number of Documents (train/dev/test)
Lang.
Table shows the licenses the publicly available checkpoints of the models used in this paper. Since LL follows
English (en)
54808 4997 4988
German (de)
54804 4997 4988
Share 4.0 International (CC BY-NC-SA 4.0) license, we solely used it for conducting experi- ments in this paper.
French (fr)
54804 4997 4988
Italian (it)
54805 4997 4987
Spanish (es)
52621 4997 4988
Polish (pl)
23063 4997 4988
Model
#Params
License
Romanian (ro)
15914 4997 4988
Dutch (nl)
54803 4997 4988
InfoXLM
278M
MIT
Greek (el)
54828 4997 4988
LiLT
284M
MIT
Hungarian (hu)
22542 4997 4988
LayoutXLM
369M
CC-BY-NC-SA 4.0
Portuguese (pt)
52205 4997 4988
Donut
74M
MIT
Czech (cs)
23056 4997 4988
Table 9: The number of parameters and the license of the model explored in this paper.
Swedish (sv)
42356 4997 4988
Bulgarian (bg)
15979 4997 4988
Danish (da)
54806 4997 4988
Finnish (fi)
42362 4997 4988
B Dataset Details
Slovak (sk)
22858 4997 4988
Tables 10 and 11 show the dataset statistics af- ter all the preprocessing steps for WIKI-Doc and MULTIEURLEX-DoO respectively. For MULTIEURLEX -DCc, found that there doc- uments the train/dev/test sets where we ran into errors when converting the pdf page into an im- age for OCR. Thus the number of documents in dev/test sets is slightly lower than 5k.
Lithuanian (It)
23075 4997 4987
Croatian (hr)
7944 2499 4988
Slovene (sl)
23061 4997 4988
Estonian (et)
22986 4997 4988
Latvian (lv)
23045
4997 4988
Maltese (mt)
17390
4996 4988
Table 11: Statistics of MULTIEURLEX-DeN dataset for different languages
B.1 WIKI-DOC Creation Steps
For curating documents for WIKI-DOC we con- ducted following steps:
3. Use Wikipedia inter-language links to retrieve non-English articles.
1. Extract the document titles document labels used in Sinha al. (2018).
4. Since the licenses of each image used in Wikipedia articles differ, we use the Wikime- dia API to obtain the license information of each image and filter out the article either the license is not available or is not permis- sible for research purposes.
2. Use the titles of the documents to retrieve En- glish Wikipedia article page (if it gives redi- rection, discard that example).
Train
Val
Test
5. The filtered Wikipedia articles converted from HTML to PDFs using pdfkit.
Lang-
#Cls
Full
Few
English (en)
110
152K
2K
32K
32K
German (de)
103
41K
IK
8K
8K
Table 12 shows the licenses of the datasets we have built upon.
French (fr)
101
33K
1K
7K
7K
Spanish (es)
106
42K
IK
9K
9K
Portuguese (pt)
86
33K
1K
4K
4K
Italian (it)
62
20K
1K
4K
4K
B.2 Preprocessing Details
Chinese (zh)
90
23K
1K
4K
4K
Japanese (ja)
94
23K
IK
4K
4K
Languages with Latin script are processed with the language option of Tesseract set to the language of interest. For languages that use non-Latin scripts
Arabie (ar)
60
8K
IK
IK
IK
Table 10: Statistics of WIKI-Dor on the number of
classes (#CIs) examples before (Full) and after (Few) subsampling the training split to 10-shots.
1 Phttps::/wwwnediiaee www..wikiiiii....mmmmomm http://ewwwiiiii.oo///kk/////// http//wik//AP::::/ t http://ppp...oopppjppp/ccc/////e

---

Dataset
License
Model
Batch
LR
Warm-up
Multi-Euricx
cC BY-SA 4.0
InfoXLM
64
3e-5
10%
DBpedia
CC-BY-SA 3.0
LiLT
64
3e-5
10%
Wikipedia texts
CC-BY-SA 3.0
LayoutXLM
32
2e-5
10%
Wikipedia images
Varies
Donut
32
1e-4
1000 steps
Table 12:
The licenses of the datasets we have cu- rated from and built upon. Most datasets follow Cre- ative Commons AtrrbutioonShaarAllke (CC-BY-SA)
Table 14:
Selected hyperparameters for
MULTIÉURLEX-DOG dataset. Learning rate (LR). Batch size is per_device_ www.ttraaatnaanaate__ batch_sszee the number of GPUs used during training.
License.
such as Japanese, Chinese, and Arabic, we fur- ther include English as the sub-language model to conduct OCR on Latin texts appearing in non- Latin languages. For WIKI-DOC, it is split into train:validationntess to 7:1.5:1.5 following stratified sampling.
E
Full Results
E.1
Few- and Full-Shot Results on English WIKI-DOC
Table 15 shows the results comparing few- and full-shot results English WikI-Doc.
Models
Few
Full
C Hyperparameters
InfoXLM
81.30o.s5
94.04a.17
LiLT
81.44..0
94.150.11
Table 13 shows the hyperparameter used for eval- uating WIkI-Doc. We conduct manual search starting from the original hyperparameters reported in the original paper with the range of [5e 6, le 4] for the learning rate and [0, 100, 200] for the warm-up steps.
LayoutXLM
82. 82.181.62
94.40u.18
Donut
26.633.
90.130.58
Table 15: English results (en-> en) few- and full-shot setup on WIKI-DOC.
Similarly, table 14 covers the batch size, learning rate and warmup ratio used to train different models for MULTIBURLEX- DoC dataset.
E.2 Initial All Page Results on MULTIEURLEX-DOC
In Table 17, we include BERT and XLM-R results all pages. Additionally
also include XLM- R results on first page of the PDF document. All Page results are borrowed from (Chalkidis et al., 2021). We can see that as we go from all pages just first page, there is a drop in the score across all languages indicating that text from other pages is important because the classes at level are very fine- mne-grained.
Model
Batch
LR
Warm-up
InfoXLM
64
le-5
200 steps
LiLT
64
le-5
200 steps
LayoutXLM
32
2c-5
200 steps
Donut
32
le-4
200 steps
Table 13: Selected hyperparameters for WIKI- Doc dataset. Learning rate (LR). Batch size is per per_device_ devicce_rran__btter train_batch_size x the number of GPUs used during training.
E.3 Correlation Analysis on Other Typological Features
Table 18 shows the correlation analysis results all typological features using the pre-computed ty- pological distances from Littell et al. (2017).
D List of WIKI-DOC Classes
Table 16 shows the list of classes in the WikI-Doc dataset after conducting preprocessing steps plained in Section 3.1. Some classes are specific to some countries (CanadianFoothal/Tean, Furo- visionSongConttssEEnt))) potentially encouraging the models to bias on such countries.

---

Academic/oorral
BuroviiionSonnContssEEng
Poem
AdultActor
Fern
Poet
Album
FilmFestival
Pope
BoxteurBoxer
Fish
President
Ambassador
FootballMaach
PrimeMinister
FottblanFootballlllayrr
Glacier
TublicTransitSystemn
Amphibian
GolfTournament
Racecourse
BhimangaCharacte
Governor
RadioHost
Anime
Gymnast
RadioStation
Arachnid
Historian
Religious
Baronet
lceHockeyLeeauee
Reptile
BasketballTeam
Insect
Restaurant
BeautyQueen
Journalist
Road
BroadcastNetwork
Judge
RoadTunnel
BusCompany
Lighthouse
RollerCoaster
BusinessPerson
Magazine
RugbyClub
CanadianFootbal/TTaa
Mayor
RugbyLeaggus
Canal
Medician
Saint
Cardinal
MemberOtParliament
School
Cave
MilitaryPerson
ScrcenWriter
ChristianBishop
Model
Senator
ClassicalMusicArriis
Mollusca
ShoppingMall
Clasicca/MusicCoompsitioo
Monarch
Skater
CollegeCoach
Moss
SoccerLeague
Comedian
Mountain
SoccerManager
ComicsCreator
MountainPass
SoccerPlayer
Congressman
Mountain
PoccerTourrnment
Conifer
MusicTeettival
Sports Member
Convention
Musical
SumoWrestler
Cricketer
MythologicalFiguree
TelevisionStation
Crustacean
Newspaper
TennisToumament
Cultivated Variety
Noble
TradeUnion
Cycad
OfficeHolder
University
Dam
Other
Village
Economist
Philosopher
VoiceActor
Engineer
Photographer
Volcano
Entomologist
PlayboyPlay Playmate
WrestlingEvent
Table 16: Classes in the WIKI-DOC dataset.

---

Models en
da
Sv
cs
All Pages
NATIVE-BERT
67.7
65.5
68.4
66.7
68.5
68.5
67.6
67.4
67.9
XLM-R
67.4
66.7
67.5
67.3
66.5
66.4
67.8
67.2
67.4
First Page only
XLM-R
64.631.9
64.656.9
65.7lo.2
64.71.1
64.531.4 36
56.512.3
65.02p.7
65.2o.7
64.95g.
Models
pt
pl
bg
hu
Avg
All Pages
NATIVE-BERT
67.4
67.2
66.7
67.7
67.8
67.8
66
67.2
XLM-R
67
65
66.1
66.7
65.5
66.5
65.8
65.7
66.61
First Page only
XLM-R
65.9u.4
61.561.0
59.156.2
61.54e.2
61.51v.7
64.151.7
63.941.2
61.281.1
63.23
Table 17: Intralingual results (X - X) of BERT and XLM-R on the MULTIEURLEX-DD DOC dataset at level 3. "Avg" in the table indicates average across all languages.
Data
Model
SYN
PHON
INV
GEO
GEN
FEA
P
S
S
InfoXLM
.44
.65
-36
-,7
23
.21
30
.29
30
330
.02
.16
Eurlex
LayoutXLM
.56
.68
-60
-.22
.07
.06
.42
.49
29
26
-52
-04
InfoXLM
.91
.96
.32
.4
771
.22
.88
.73
511
.82
.85
.22
Wiki
LayoutXLM
.88
.88
.27
34
75
-57
.90
81
46
.72
.22
.72
Table 18: Full correlation analysis results the cross-lingual transfer gap and typological distances using syntactic (SYN), phonological (PHON), phonetic inventory (INV), genetic (GEN), and featural (FEA) features. Spearman (S)
and Pearson (P) correlations are used.


Reviews:
[{"review_id": "UevlgkBxhN", "reviewer": "Reviewer_choU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments on our work.\n\n### Rebuttal to Reasons To Reject\n\n> “The writing, especially the result analysis is confusing”\n\nOur results section (Section 4) is structured such that we first present results on English language for both datasets (Section 4.2). Next we present intralingual results for non-English languages (Section 4.3) and finally we present cross-lingual results (section 4.4). We appreciate more detailed feedback on which specific parts are confusing to further improve the clarity of the paper.\n\n\n> “The numbers of different language training samples [...] significantly different”\n\n\nWe did not normalize for the amount of training data to make our results comparable to prior work (i.e Chalkidis et al., 2021). Additionally, our primary goal with intralingual experiments was to compare different models’ performance on the same language or language group (Page 6 right column in Lines 429-435). Furthermore the difference in training data size is only a concern for intralingual experiments and not cross-lingual experiments due to using the model trainined on the single English training set. We agree that the influence of the number of training examples for the intralingual experiments are not mentioned and clarify it in the next version of the paper. \n\n\n> “Document images in these datasets are [...] real applications”\n\n\nThe PDFs in both these datasets do have layout variations. The primary focus of this work is not layout understanding but rather understanding the content of the documents where layout also plays an important role. For layout understanding, there is RVL-CDIP dataset where the accuracies of state of the art models have already reached 95-96% thus we choose not to focus on layout understanding since layout understanding is a relatively easy task compared to document content understanding. For example, one of the models (Docformer) achieves ~96% accuracy on RVL-CDIP however it only achieves mRP of 63.46 on MULTIEURLEX-DOC (Table 3). \n\n\n### Rebuttal to Typos Grammar Style And Presentation Improvements\n> “Are these datasets proposed for Document AI, or document classification task?”\n\n\nWe will improve the naming convention in final version of the paper to make it consistent. As mentioned earlier, our focus is more on understanding content of the semi-structured documents rather than the high level layout of these document since there is already an existing dataset for layout understanding (RVL-CDIP).\n\n\n> “Line 439 - 440. This sentence is inaccurate and confusing”\n\n\nThese lines talk about results on WIKI-DOC specifically on Germanic languages like German (de) and romance languages like Spanish (es), French (fr), Italian (it) and Portuguese (pt) based on results in Table 5 so it is not inaccurate.\n\n\n> “The expressions in Sec 4.4.1 are confusing”\n\nWe kindly request Reviewer choU for adding further details on which expressions in Section 4.4.1 are confusing so that we can rephrase and clarify for the future readers of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed datasets are novel in the document understanding task in terms of the language coverage, and category coverage. \n2. The paper performs comprehensive experiments on the datasets and analyzes multiple models' performance on the proposed datasets.\n3. The paper reports and analyzes challenges and limitations posed by these datasets which would be inspiring for future research.", "reasons_to_reject": "1. The writing, especially the result analysis is confusing. \n2. The numbers of different language training samples in MULTIEURLEX-DOC dataset are significantly different. However, its influence is not well discussed in the paper.\n3. Document images in these datasets are all created by converting pdfs or wiki-pages, which lask coverage of multiple text style, and document style in real applications.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper reported two new multilingual document image datasets for visual document understanding tasks. This paper also evaluates multiple document understanding models on the proposed datasets and analyzes the limitations and challenges proposed by these datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Are these datasets proposed for Document AI, or document classification task? The terms should be consistent, since document classification task is one of tasks in Document AI, as argued in the paper.\n2. Line 439 - 440. This sentence is inaccurate and confusing.\n3. The expressions in Sec 4.4.1 are confusing. Please refine and reorganize the statements on these findings."}}, {"review_id": "yHBqABVPYH", "reviewer": "Reviewer_BLSd", "summary": "", "questions": "In addition to the issues mentioned in the reasons to reject section, here are a few more questions:\n\nLn. 270: What do you mean here? It is used because it was previously used? I was confused.  This is not a self-reference is it?\n\nLn 337.  There are doubts about the methodology here. Creating pdf’s from text (in MultiEURLEX) and then converting them to images with pdf2image is going to affect the real life performance, since the OCR output you get after putting these images through this pipeline will probably create much less noise. Have you considered this?\n\nLn 400: Where do we observe the result mentioned here? Is it referred to in the tables for few shot learning?\n\nLn 429: I did not find a table I can refer to for this information.\nTable 8. Can you explain how the correlation is calculated more clearly please?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing detailed feedback on our work.\n\n### Rebuttal to Reasons To Reject\n\nWe would like to first correct a critical misunderstanding. \n> \"text files were converted to PDF files in the MULTIEURLEX case\"\n\n> \"the authors first convert texts into pdf format\"\n\n**This is not correct; the original documents used to create MultiEurlex dataset by Chalkidis et al. (2021) [1] comes with PDF files, which we scraped from the EurLex website to build MULTIEURLEX-PDF dataset** (e.g., The example PDF document used in Figure 1 for MULTIEURLEX-DOC is available at https://eur-lex.europa.eu/legal-content/en/TXT/PDF/?uri=CELEX:32006D0213). What we borrowe from the dataset built by [1] are 1) CELEX document IDs (that are used to obtain the corresponding PDF documents from the EurLex website), and 2) the document labels (one or more labels per document due to being multi-label dataset)\n\n> \"MultiEURLEX reportedly only has text documents\". \n\nSimilar to the above point, this is not the case for the original PDF EurLex documents used to create MultiEurlex by Chalkidis et al. (2021) [1]. For example, the document at https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32006D0213 includes tables and figures which are discarded in the MultiEurlex dataset created by Chalkidis et al. (2021) [1] (which is directly accessible at https://huggingface.co/datasets/multi_eurlex/viewer/en/train?row=0). The structured elements of the documents are not retained in the textual version of the dataset, but this make it very relevant to the problem we are studying.\n\n> \"Xu et al (XFUND:A Benchmark Dataset for Multilingual Visually Rich Form Understanding, ACL 2022) reports results of experiments with multi-label classification, zero-shot cross-lingual transfer\". \n\nWe emphasize here that the tasks explored in Xu et al. 2022 and this paper are different. Xu et al. 2022 report zero-shot cross-lingual transfer results on the semantic entity recognition task and the relation extraction task using the XFUND and FUNSD datasets, but not on the document image classification task. The tasks explored in our work are multi-label and multi-class document image classification (not semantic entity recognition nor relation extraction) and zero-shot cross-lingual transfer on document image classification. Relatedly, we agree with the reviewer that the sentence in Lines 11-14 \"Document AI models in previously untested setting such as 1) multi-label classification, and 2) zero-shot cross-lingual transfer setup\" is not correctly scoped when read in isolation. We will further clarify the scope of this sentence by scoping only to document image classification task by rephrasing it as \"untested for multi-label document image classification and zero-shot cross-lingual transfer on document image classification\".\n\n> “D)  [...] the authors claim the datasets need to be hierarchical, multi-label etc”. \n\nWe would like to remind that the paper does not claim that datasets need to be hierarchical or multi-label. The goal of the paper is to curate new datasets “to complement the limitations of RVL-CDIP dataset” (Lines 173-175) or specifically, complement the aspect of document image classification task not covered by RVL-CDIP, such as 1) being multi-label in nature (i.e., one or more annotated labels for each document) and 2) covering multiple languages (Lines 150-175.)\n\nWe will make all the points listed above clear in the next version of the paper to avoid similar misunderstanding by the future readers of the paper. \n\n\n\n### Rebuttal to Questions For The Authors\n\n> Ln. 270: What do you mean here? It is used because it was previously used? I was confused. This is not a self-reference is it?\n\nIn Line 270, we simply meant to state that we selected InfoXLM as the text-only model following the prior work by Wang et al. 2022.\n\n\n> Ln 400: Where do we observe the result mentioned here? Is it referred to in the tables for few shot learning?\n\nLine 400 from the paper refers to “F1 gap between the two models is significantly smaller in the full-shot setting (90.13 vs. 94.40) than the few-shot setting (26.63 vs. 82.18)“. The full-shot setting is referred from Table 3 (Donut: 90.13, LayoutXLM: 94.40), but as Reviewer BLSd correctly pointed out, the few-shot results in the English WIKI-DOC dataset is not referred in the tables of the paper because the focus of the paper is on non-English experiments, and therefore, we excluded few-shot results on the WIKI-DOC dataset in English. \n\nHere, we show the results we are referring to (i.e., Donut: 26.63, LayoutXLM 82.18) along with other models:\n\n    Models       | Accuracy on WIKI-DOC (with standard deviation across 5 random seeds\n    ------------------------------\n    InfoXLM      | 81.30 (0.85)\n    LiLT         | 81.44 (0.90)\n    LayoutXLM    | 82.18 (1.62)\n    Donut        | 26.63 (3.06)\n\nThank you for pointing this out and we will add these results in the appendix of the next version of the paper for the sake of completeness.\n\n\n\n> Ln 429: I did not find a table I can refer to for this information.\n\nLine 429 is referring to Table 4 and analyzing the results with respect to language families. Here are the excerpts of Table 4 on Uralic and Slavic language families on mean R-Precision scores (omitting the standard deviations):\n\nUralic languages:\n\n       | InfoXLM | LayoutXLM\n    ------------------------------\n    hu | 58.84   | 63.87\n    fi | 63.46   | 63.52\n    et | 60.37   | 63.43\n\nSlavic languages:\n\n       | InfoXLM | LayoutXLM\n    ------------------------------\n    pl |  61.12  |  63.26 \n    bg |  14.23  |  63.67\n    et |  40.99  |  63.6 \n \nFrom the above excerpts from Table 4, we observe that “LayoutXLM is much better than InfoXLM on Slavic and Uralic languages. ” (Lines 434-435). We will add specific numbers we are referring into in Lines 434-435 for the next version of the paper. \n\n\n\n> Table 8. Can you explain how the correlation is calculated more clearly please?\n\nThank you for pointing this out. This is explained in Page 8, Footnote 7 and Lines 507-511. To elaborate, the correlation in Table 8 is calculated among the two arrays of numbers. The first array is the accuracy gap between a model trained on English and evaluated on language X and model trained on language X and evaluated on language X. For example, hu→hu scored 58.84 (Table 4), en→hu scored 45.74 (Table 6) and the accuracy gap is 58.84 - 45.74 = 13.10. Focusing now on the syntactic typological feature, the syntactic distance between English and Hungarian is 0.60 (Line 525). We apply similar calculation to the rest of non-Hungarian languages and then calculate the correlation between the array of accuracy gap ([13.10, ...]) and the syntactic distance between English and the target languages ([0.60, ...]) to obtain correlation values in Table 8. The same calculation is done for all typological distances between source and target languages to obtain all correlation values in Table 8. We will add this explanation into the appendix of the next version of the paper.\n\n[1] MultiEURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer https://aclanthology.org/2021.emnlp-main.559/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors present two multilingual datasets for the task in hand. They are sizeable datasets and, since they were created from existing datasets, they are likely to be cleaner than a brand new one.\n\nThere are a substantial number of multilingual experiments on document image classification.  They do intra-lingual and cross-lingual experiments on the datasets. The intra-lingual ones aim to test multi-modal vs image-only model performance comparison in any given languages by testing the pretrained out of the box models. The cross-lingual experiments aim to test few-shot learning on the cross-lingual transfer setting for a pair of languages where the model is trained on English and tested on a different language. They test uni-model and multi-modal models in this setting.", "reasons_to_reject": "There are several issues about methodology and the data used in the paper:\n\nA) It is not entirely clear how the two datasets are \"curated\" from existing datasets. The only mention of the modification I could find was that some files were removed, and text files were converted to PDF files in the MULTIEURLEX case. The original label sets were kept. Is there any other major alteration that I might have missed? If not, I would suggest not to call them “novel datasets”.\n\nB) Regarding the methodology used, the authors first convert texts into pdf format, which was later converted to images for some models. Doesn’t this automatically cause some disadvantage for layout- and image-based models and an advantage for text-only models since that is the source of primary information for those models? I think the way the datasets were created and were tested on models that were  trained on radically different kinds of examples is the cause of many unexplained results the authors observed in the experiments such as InfoXML beating other models in some settings and Donut being very considerably worse than all other models in almost all settings.\n\nC) Regarding the motivation for creating/curating these particular datasets, the authors claim the datasets complement the existing ones in some respects. In addition, these two datasets seem to be completely different in content/characteristics from the one the authors are comparing their work to for the task of Document AI. For instance, the category set in RVL-CDIP has only 16 categories whereas the law document dataset has hundreds, MultiEURLEX reportedly only has text documents.  I am wondering whether these datasets were chosen only because they were multilingual rather than whether they were appropriate for document image classification tasks or not.", "questions_to_authors": "In addition to the issues mentioned in the reasons to reject section, here are a few more questions:\n\nLn. 270: What do you mean here? It is used because it was previously used? I was confused.  This is not a self-reference is it?\n\nLn 337.  There are doubts about the methodology here. Creating pdf’s from text (in MultiEURLEX) and then converting them to images with pdf2image is going to affect the real life performance, since the OCR output you get after putting these images through this pipeline will probably create much less noise. Have you considered this?\n\nLn 400: Where do we observe the result mentioned here? Is it referred to in the tables for few shot learning?\n\nLn 429: I did not find a table I can refer to for this information.\nTable 8. Can you explain how the correlation is calculated more clearly please?", "paper_topic_and_main_contributions": "The paper describes a document image classification benchmark. The authors introduce two multilingual datasets and perform experiments on these with out-of-the-box Document AI models, namely LayoutXLM, InfoLXM, Donut and LiLT. They also use DocFormer, which does not feature in all experiments. \n\nThe two multilingual datasets that are 'curated' are 1.MultiEurLex-Doc and 2.Wiki-Doc. These are both derived from previous work. The former has documents about EU laws in 23 different languages and the latter is created from Wikipedia articles by a previous study, and has non-latin script languages such as Arabic to complement MultiEurLex-Doc.\n\nThe paper aims to find answers to 3 research questions: 1.Whether pre-trained models for document classification work on their datasets. 2.To test how the multilingual aspect of the pre-trained models perform on different languages. 3.Cross-lingual aspect of these models, meaning how well they perform in zero-shot transfer environments where the training language is English and the model is tested on other languages.\n\nThe authors perform experiments where they test these pre-trained models that use different sets of modalities of 1.Document Layout, 2.Images, 3.Text. They report results for each language in the two datasets in the first round of results. Then, they report other results for cross-lingual transfer where the source language is English in all cases. They also report \"full-shot\" and \"few-shot\" results for all languages. Furthermore, they report challenges in cross-lingual transfer in syntactically distant languages with English, which seems logical. They also report that image-only models are not as successful as the multi-modal or text-based ones in these settings.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "XFUND is mentioned in \"Task coverage\" section.  However, it is not clear why there are no comparisons or any discussion in the text about this multilingual dataset. It would be nice to see a comparison of especially the zero/few shot experiments with the ones in the LayoutXLM paper. Since the reference is already included because of the \"Task coverage\" section I am not including it again."}}, {"review_id": "D3MDmPi4yJ", "reviewer": "Reviewer_VmrD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing detailed feedback on our work.\n\n### Rebuttal to Reasons To Reject\n\n> “Insufficient details and analysis of proposed datasets, including label semantics, distribution, and discussion of the influence of domain”\n\n\nWe would like to remind that the dataset distribution details provided in Appendix B, Tables 10 and 11 of the paper.  Regarding the influence of domain, we agree that different models use different pre-training corpus (Table 2) and therefore would exhibit slight biases towards different data domains (e.g., less domain gap between InfoXLM on the WIKI-DOC dataset) and thus we have stated it as one of the limitations of the work. We will add additional insights while comparing the models performances based on these pre-training variations.\n\n\n> “could benefit from including discussions and insights about vision transformer-based models, such as DiT,  and comparing them with image-only vision encoders”\n\n\nWe have included Donut as part of the image-based models for our benchmarking, however, we didn’t include other image-only models like Document Image Transformer (DiT) since Donut was seen to show much stronger performance than DiT on RVL-CDIP dataset (95.30 for Donut [1] vs 92.69 of the DiT-Large model [2]) and hence we considered Donut as a suitable representation for image-based models. Moreover, image-only models have typically under-performed compared to image+text models for document understanding tasks (Table 4 in [3]) and therefore we limited our analysis to one model i.e., Donut. Finally, the DiT models from [2] are pretrained only on English document images (i.e., IIT-CDIP dataset [3]), and therefore, we excluded from experimenting in our submission due to focus on experimenting in non-English languages.\n\n[1] OCR-free Document Understanding Transformer https://arxiv.org/abs/2111.15664\n\n[2] DiT: Self-supervised Pre-training for Document Image Transformer https://arxiv.org/abs/2203.02378\n\n[3] DocFormerv2: Local Features for Document Understanding https://arxiv.org/pdf/2306.01733.pdf\n\n[4] Building a test collection for complex document information processing  https://dl.acm.org/doi/10.1145/1148170.1148307", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introduces two valuable new multilingual datasets for document image classification, filling an important gap and enabling more comprehensive evaluation of models.\n\n2. Provides extensive empirical analysis and benchmarking of state-of-the-art document AI models, shedding light on their capabilities and limitations especially for multilingual scenarios.\n\n3. Compares uni-modal and multi-modal models in multilingual settings, finding image-only models struggle in certain scenarios unlike prior English-only results.\n\n4. Open sources code and datasets to facilitate reproducible research.", "reasons_to_reject": "1. Insufficient details and analysis of proposed datasets: More in-depth information about the proposed datasets, including label semantics, distribution, and discussion of the influence of domain for MultiEurLex-Doc and Wiki-Doc, is necessary. This additional context would clarify the dataset's strengths and limitations. For example, InfoXLM performs well on Wiki-Doc possibly due to its pre-training on Wikipedia.\n\n2. The paper could benefit from including discussions and insights about vision transformer-based models, such as DiT, and comparing them with image-only vision encoders and end-to-end encoder-decoder models like Donut.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces two new datasets, MULTIEURLEX-DOC and WIKI-DOC, to advance multilingual document image classification. The key contributions are:\n\n1. MULTIEURLEX-DOC curates a multi-label legal document dataset in 23 European languages derived from MULTIEURLEX. \n\n2. WIKI-DOC curates a multi-class Wikipedia document dataset in 9 languages, including non-European languages like Arabic, Chinese, and Japanese.\n\n3. Several multilingual Document AI models are evaluated on the new datasets to analyze their multilingual and cross-lingual abilities. Limitations in cross-lingual transfer are revealed, especially between distant languages.\n\n4. Significant performance gaps exist between image-only models like Donut and the best models, indicating opportunities for improvement in image-only approaches.\n\n5. The new datasets enable more comprehensive analysis of multilingual Document AI models, complementing existing datasets. \n\n6. Multilingual model limitations are revealed, particularly in cross-lingual transfer and for image-only models.\n\nIn summary, the key contributions are curating two new challenging multilingual document datasets and leveraging them to analyze and reveal limitations of current multilingual Document AI models. The datasets advance the field's ability to evaluate and improve multilingual document classification.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Pros: 
- This paper introduces two new multilingual document image classification datasets: MULTIEURLEX-DOC and WIKI-DOC
- "The paper reports and analyzes challenges and limitations posed by these datasets which would be inspiring for future research."

Cons:
- The details on the proposed dataset and experimental setup are highlighted to be confusing and can be made clearer.
Decision: Accept-Findings