RoBERTa Ensemble Technique for Document
Information Localization and Extraction
Notebook for the DocILE Lab at CLEF 2023
Bao Gia Tran1 , Duy-Ngo Minh Bao1 , Khanh Gia Bui1 , Huy Viet Duong1 ,Dang Hai Nguyen1 and Hieu Minh Nguyen1
1

University of Information Technology - VNUHCM, Ho Chi Minh City, Vietnam

Abstract

Document Information Localization and Extraction (DocILE) is attracting a large amount of attention
from the research community due to its potential to significantly reduce manual work. With theexplosive growth of technology as they are today, we want to experiment with a method that leverages
the advantages of language models in information extraction since it requires an understanding of
the contextual information of the text, which large language models are currently successful on. The
experiments include using a new combination of published baseline with our model 𝑅𝑜𝐵𝐸𝑅𝑇 𝑎, alongwith a post-processing step, which helped us achieve a Top 3 position in the competition ranking board.

Keywords

DocILE, RoBERTa, Ensemble, Pseudo-Labeling

1. Introduction
Extracting information from documents is an indispensable part of human activities in themodern era. However, manual information extraction is time-consuming and labor-intensive.
Therefore, automating the process of extracting information has gained much attention fromthe research community as it has high applicability in reducing workload for workers in manualtasks and creating opportunities for them to focus more on strategic work.
The information extraction process is challenged since it requires an understanding of thesemantics, layout, and context of content in the documents. In Machine Learning (ML), the scopeof addressing this issue is called Document Information Extraction (IE), a part of Document
Understanding.
DocILE 2023 [1][2] is a competition for extracting information from business documents. The
participating teams will receive a dataset of invoice-like documents such as tax invoices, orders,purchase orders, receipts, sales orders, proforma invoices, credit notes, utility bills, and debit
notes [1]. In Track 1, also known as KILE (Key Information Localization and Extraction), participants were challenged to develop algorithms that can locate and extract specific information
CLEF 2023: Conference and Labs of the Evaluation Forum, September 18–21, 2023, Thessaloniki, Greece$ 22520121@gm.uit.edu.vn (B. G. Tran); 22520320@gm.uit.edu.vn (D. M. Bao); 22520630@gm.uit.edu.vn(K. G. Bui); 22520540@gm.uit.edu.vn (H. V. Duong); 22520189@gm.uit.edu.vn (D. H. Nguyen);
22520440@gm.uit.edu.vn (H. M. Nguyen) https://github.com/xbaotg (B. G. Tran)© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR

Workshop
Proceedingshttp://ceur-ws.org
ISSN 1613-0073

CEUR Workshop Proceedings (CEUR-WS.org)

such as names, dates, addresses, .. or any other key data from a given document.
The pipeline to localize Key Information and extract them is built upon the provided baselines
for the DocILE competition, and we acknowledge their contributions [1]. At first, the input
data is a set of PDF pages containing invoices processed using DocTR [3], from which thebounding boxes and content of the information are obtained. Afterward, they classify thosecontent using Token Classification models. Finally, they merge content based on their field type.
The flowchart is shown in Figure 1.

Figure 1: Pipeline of localizing Key Information and extracting them

Our focus lies in optimizing their pipeline. Specifically, we leverage different versions of𝑅𝑜𝐵𝐸𝑅𝑇 𝑎 - a large language model that is used to achieve state-of-the-art results on GLUE,RACE, and SQuAD in Natural Language Processing [4], i.e. two provided baseline RoBERTa [5]and one RoBERTa trained by us - together with a post-processing step. After that, we use itto generate pseudo-label datasets from provided unlabeled dataset and re-train our models on
that, which showed a relatively good result. Our pipeline is shown in Figure 2. We will discusseach component in detail in the following sections.

2. Proposed method
2.1. Ensemble
After evaluating several models provided, we see that one model only performs well in certain
categories while the opposite thing happens for the other models. This leads us to the idea ofusing Ensemble [6].
We first implement Ensemble using Average and Max Voting [6] since they are two of the mostcommon methods. However, the results acquired show a relatively high score precision while
the recall is not significant, which means that the models provide fairly accurate predictions
but the proportion of positive samples missed in the dataset is quite large.
From the concept of affirmative ensemble presented in prior research [7], we decided toincorporate this concept into our problem as a method to address the aforementioned issues.

Figure 2: Pipeline of our approach to this competition.

Specifically, if any of the models predict that certain content belongs to a particular field type,
we consider that content to actually belong to that field type.

2.2. Pseudo-labelingIn this task, we are provided an abnormally huge amount of unlabeled data compared to thesmall amount of labeled data [1]. We believe that utilizing this unlabeled data will improve theperformance of the models. This leads us to the idea of using semi-supervised learning methods.
Pseudo-Labeling [8] is a more effective method compared to other methods such as [9], [10],and [11]. We propose a different way to implement this technique for our models:
1. Models will be trained on the Train (annotated) dataset.
2. We use the Ensemble technique with Post-processing to predict labels for the unlabeleddataset, which is then called as pseudo-labeled dataset.
3. Train the model on the pseudo-labeled dataset for some epochs.
4. Fine-tune the model on the Train dataset.
Here, we train the model on the pseudo-labeled dataset instead of mixing it with the labeleddataset, because it is a dataset that we have little control over, and it may contain cases thatare completely different from the training dataset. Training the model on the pseudo-labeleddataset for some epochs helps the model approach more types of data, thereby learning generalfeatures. Then, we fine-tune the model on the training dataset to learn the correct features foreach specific problem, helping the model improve its effectiveness on that problem.

2.3. Post processing
The models we use struggle in distinguishing information that have the same field type but
is a bit far apart. This leads to the problem that even though the information belongs to thesame field type and the same bounding box, the model predicts it as multiple different boundingboxes. Moreover, after observing the prediction results compared to the ground truth, andexperimenting on various documents, we found that it is rare for information of the same field
type to be close to each other on the same document.
We first find the distance between the centers of each pair of bounding boxes belonging to
the same field type predicted by the model. Then, we experiment with grouping these boundingboxes on different thresholds. For each pair of bounding boxes of information belonging to thesame field type, if their Euclidean distance is below or equal to the threshold, we will merge
those two bounding boxes into a new one, its coordinate is calculated by using formula (1).
How the Post-processing work is shown in Figure 3.
′′
𝑓 𝑖𝑛𝑎𝑙 = (min(𝑥left ; 𝑥′left ), min(𝑦top ; 𝑦top), max(𝑥right ; 𝑥′right ), max(𝑦bottom ; 𝑦bottom))(1)

By doing this, we can reduce the number of false bounding boxes predicted by the model andimprove the accuracy of the predictions.

3. Experiment3.1. Dataset
We maintain the same dataset partitioning as provided by the organizer, with the informationof each dataset used for the Training, Validating, and Testing processes as described in Table 1.
Most experiments below were conducted on an environment consisting of 4 RTX 2080 Ti
12GB GPUs, along with the following selected parameters and hyperparameters:Figure 3: Post-Processing merges two bounding boxes that have the same field type (same border color)
and their distance is below a threshold into one bounding box.
Table 1
Number of documents and annotations in each dataset
DatasetDocument Number

Annotation Number

Train
Validation
TestSynthetic
Unlabeled

5,180
500
1,000
100,000932,000

65,651
5,862
1,117,000-• Train batch size = 4• Test batch size = 4• Gradient Accumulation Steps = 4• Weight decay = 0.01
• Data Loader workers = 32
• Training Epoch = 500• Learning rate = 1e-5
At the same time, we use the validation set to evaluate the model and compare the performance
between different models based on the results.

3.2. Model
We use 3 𝑅𝑜𝐵𝐸𝑅𝑇 𝑎 models, 2 published baseline models [5] trained on the Synthetic + Train
dataset, and the remaining model is taken directly from HuggingFace [12]. We will refer tothese models with different names for easy distinction as follows:• R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝐵𝐴𝑆𝐸: baseline 𝑟𝑜𝑏𝑒𝑟𝑡𝑎_𝑏𝑎𝑠𝑒_𝑤𝑖𝑡ℎ_𝑠𝑦𝑛𝑡ℎ𝑒𝑡𝑖𝑐_𝑝𝑟𝑒𝑡𝑟𝑎𝑖𝑛𝑖𝑛𝑔• R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝑂𝑈 𝑅𝑆: baseline 𝑟𝑜𝑏𝑒𝑟𝑡𝑎_𝑜𝑢𝑟𝑠_𝑤𝑖𝑡ℎ_𝑠𝑦𝑛𝑡ℎ𝑒𝑡𝑖𝑐_𝑝𝑟𝑒𝑡𝑟𝑎𝑖𝑛𝑖𝑛𝑔• R𝑜𝐵𝐸𝑅𝑇 𝑎_𝑂𝑈 𝑅𝑆: the model was not trained on any DocILE dataset.

3.3. Fast Gradient Method
For R𝑜𝐵𝐸𝑅𝑇 𝑎_𝑂𝑈 𝑅𝑆, we fine-tune it from RoBERTa [12] using the Fast Gradient Method(FGM) [13] technique on the Synthetic dataset with 30 epochs and on the Train dataset with500 epochs. The result obtained as shown in Table 2, which is similar to the baseline but thistechnique helps the model to be more generalized [13], so we still keep and apply it with othermethods.
Table 2
Result obtained from training R𝑜𝐵𝐸𝑅𝑇 𝑎_𝑂𝑈 𝑅𝑆 with Fast Gradient Method
Model

AP

R𝑜𝐵𝐸𝑅𝑇 𝑎_𝑂𝑈 𝑅𝑆R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝑂𝑈 𝑅𝑆R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝐵𝐴𝑆𝐸

0.562
0.557
0.566

3.4. Lion Optimizer
For all 3 models, we replaced the default optimizer, from AdamW to Lion Optimizer[14] and trained for an additional 300 epochs on the Train dataset. However, only
R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝑂𝑈 𝑅𝑆 showed significant improvement, while the other models aremostly unchanged. Nevertheless, we will still use Lion Optimizer for the methods below because
it seems to converge much faster than AdamW. Table 3 shows the results obtained.
Table 3Result obtained from training 3 𝑅𝑜𝐵𝐸𝑅𝑇 𝑎 with Lion Optimizer
Model

AP

R𝑜𝐵𝐸𝑅𝑇 𝑎_𝑂𝑈 𝑅𝑆R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝑂𝑈 𝑅𝑆R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝐵𝐴𝑆𝐸

0.562
0.566
0.565

3.5. Ensemble
We perform Ensemble using different methods:• Average Ensemble [6]• Max-Voting Ensemble [6]• Affirmative Ensemble [7]For each method, we ensemble the following models:• R𝑜𝐵𝐸𝑅𝑇 𝑎_𝑂𝑈 𝑅𝑆 trained with FGM technique and Lion Optimizer• R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝑂𝑈 𝑅𝑆 trained with Lion Optimizer• R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝐵𝐴𝑆𝐸 without any changes
Table 4 demonstrates that Affirmative Ensemble produces significantly better results compared
to commonly used methods like Max-Voting and Average. Therefore, we will employ theAffirmative Ensemble technique on our three models to predict the output.

Table 4Result obtained from Ensembling 3 𝑅𝑜𝐵𝐸𝑅𝑇 𝑎 models
MethodAP

Average
Max-Voting
Affirmative

0.580
0.576
0.607

3.6. Post-Processing
We performed the Post-Processing method mentioned in Section 2.3, on the predicted output ofeach of the 3 models with different percentage thresholds of the document width. E.g. For adocument with a width is 2000px, the 14.5% threshold means that it will merge two boundingboxes whose distance is below 14.5% of 2000px or 290px.
Table 5Result obtained from changing post-processing threshold, i.e. the distance between the two central
bounding boxes relative to the width of the document
Threshold (%)

R𝑜𝐵𝐸𝑅𝑇 𝑎_𝑂𝑈 𝑅𝑆APR𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝑂𝑈 𝑅𝑆R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝐵𝐴𝑆𝐸

1213
14
14.5
15
16
17

0.606
0.606
0.606
0.607
0.607
0.603
0.598

0.603
0.604
0.604
0.605
0.603
0.599
0.596

0.606
0.607
0.607
0.608
0.607
0.603
0.603

As shown in Table 5, the threshold of 14.5% of the document width gives the highest result
when evaluated on the validation set. From now on, we will use 14.5% as the default threshold. Combining this Post Processing method with the methods we mentioned in Section 2
significantly improves the results.

3.7. Pseudo-Labeling
Due to the large number of documents, we will do this technique on each chunk from provided
chunks dataset. Starting with the chunk0 dataset, we pre-process it as follows:• Remove documents belonging to clusters = -1, i.e., documents whose layouts do notappear in the Train dataset.
• Remove documents too big, i.e. documents with a size larger than 3000 pixels in anydimension.
• Remove rotated documents.

Figure 4: Pipeline of pre-processing unlabeled dataset.

We then ensemble the 3 models trained on the Train dataset, combining with the Post Processing. Afterward, we predict on the unlabeled chunk0 dataset to generate pseudo annotations
for that, which we call pseudo0 . After that, we train 3 models on this dataset with the following
hyperparameters:• Epoch: 30• Learning Rate: 1e-5
Later, we use the Train dataset to train all 3 models more with the following hyperparameters:• Epoch: 300• Learning rate: 5e-6

Figure 5: Pipeline of generating pseudo-labeled dataset from trained models

The addition of Pseudo-Labeling slightly improved our results as shown in Table 6. However,this method was implemented when the competition was in its final days, which only allowed
us to perform it on one chunk of data. Nevertheless, we believe that continuing to use theremaining chunks will continue to improve the final results.

Table 6Result obtained from training models with Pseudo-label
APTrain Data + chunk0

Model
R𝑜𝐵𝐸𝑅𝑇 𝑎_𝑂𝑈 𝑅𝑆R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝑂𝑈 𝑅𝑆R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝐵𝐴𝑆𝐸0.5620.566
0.566

0.568
0.57
0.576

4. Result
Table 7 shows our performance on the validation dataset of each model when combinedwith Fast Gradient Method, Lion Optimizer, and Pseudo-Labeling technique. The value of𝑅𝑜𝐵𝐸𝑅𝑇 𝑎_𝑂𝑈 𝑅𝑆 in the Baseline column indicates the result of training RoBERTa with theFast Gradient Method technique.
Table 7Performance of different models on the DocILE competition task
Model
R𝑜𝐵𝐸𝑅𝑇 𝑎_𝑂𝑈 𝑅𝑆R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝑂𝑈 𝑅𝑆R𝑜𝐵𝐸𝑅𝑇 𝑎_𝐷𝑂𝐶𝐼𝐿𝐸_𝐵𝐴𝑆𝐸Baseline

AP+ Lion Optimizer+ chunk0

0.562
0.557
0.566

0.562
0.566
0.565

0.568
0.57
0.576

Table 8 shows our performance of 3 models with the Ensemble Method, they are trained withand without the Pseudo-Labeling technique. From the predicted result, we do Post-Processing
and evaluate them on the validation dataset.
Table 8Performance of different models on the DocILE competition task
ModelMethod

3 𝑅𝑜𝐵𝐸𝑅𝑇 𝑎+ Pseudo–Labeling+ Ensemble

AP+ Post–Processing

0.608
0.612

0.644
0.648

Overall, our results increased significantly compared to the baseline, +0.082 on the Valset
and +0.073 on the Testset, which is shown in Table 9. However, we believe there are still manythings we can do to further improve the results:• Use more unlabeled data. Currently, only a very small fraction (10k out of almost 1M) ofthe unlabeled data was used.
• Use models incorporating layout features such as LayoutLMv3, LiLT, etc.

Table 9
Performance of different models on the DocILE competition task

ValsetTestsetBaseline

Ours

Change

0.566
0.539

0.648
0.612

+0.082+0.073

5. Conclusion
In this paper, we have presented a solution for the tasks required in Track 1 KILE of DocILE
2023. Our improvements to the baseline [5] have demonstrated their effectiveness. This result issignificantly higher than the initial performance and demonstrates the potential of our methodin addressing issues related to information extraction from business documents. We hope that
our solution will contribute to the development of the field of information extraction frombusiness documents, and we look forward to further researching and improving our method inthe future.

References[1] Š. Šimsa, M. Uřičář, M. Šulc, Y. Patel, A. Hamdi, M. Kocián, M. Skalický, J. Matas, A. Doucet,
M. Coustaty, D. Karatzas, Overview of DocILE 2023: Document Information Localizationand Extraction, in: A. Arampatzis, E. Kanoulas, T. Tsikrika, S. Vrochidis, A. Giachanou,
D. Li, M. Aliannejadi, M. Vlachos, G. Faggioli, N. Ferro (Eds.), Proceedings of the FourteenthInternational Conference of the CLEF Association (CLEF 2023), LNCS Experimental IR
Meets Multilinguality, Multimodality, and Interaction., 2023.
[2] Š. Šimsa, M. Šulc, M. Uřičář, Y. Patel, A. Hamdi, M. Kocián, M. Skalický, J. Matas, A. Doucet,
M. Coustaty, D. Karatzas, DocILE Benchmark for Document Information Localization andExtraction, in: 17th International Conference on Document Analysis and Recognition,
ICDAR 2021, San José, California, USA, August 21–26, 2023, Lecture Notes in Computer
Science, Springer, 2023.
[3] Mindee, doctr: Document text recognition, https://github.com/mindee/doctr, 2021.
[4] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer,
V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint
arXiv:1907.11692 (2019).
[5] Š. Šimsa, M. Šulc, M. Uřičář, Y. Patel, A. Hamdi, M. Kocián, M. Skalickỳ, J. Matas, A. Doucet,M. Coustaty, D. Karatzas, Docile baselines, https://github.com/rossumai/docile/tree/main/baselines, 2023.
[6] P. Sanagapati, Ensemble learning techniques tutorial, https://www.kaggle.com/code/pavansanagapati/ensemble-learning-techniques-tutorial, 2021.
[7] A. Casado-García, J. Heras, Ensemble methods for object detection, 2019. https://github.
com/ancasag/ensembleObjectDetection.
[8] D.-H. Lee, et al., Pseudo-label: The simple and efficient semi-supervised learning method

for deep neural networks, in: Workshop on challenges in representation learning, ICML,
volume 3, 2013, p. 896.
[9] A. Tarvainen, H. Valpola, Mean teachers are better role models: Weight-averaged
consistency targets improve semi-supervised deep learning results, in: I. Guyon,
U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett(Eds.), Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017. URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/68053af2923e00204c3ca7c6a3150cf7-Paper.pdf.
[10] T. Miyato, S.-I. Maeda, M. Koyama, S. Ishii, Virtual adversarial training: A regularization
method for supervised and semi-supervised learning, IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (2019) 1979–1993. doi:10.1109/TPAMI.2018.2858821.
[11] D. P. Kingma, S. Mohamed, D. Jimenez Rezende, M. Welling, Semi-supervised learningwith deep generative models, in: Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence,
K. Weinberger (Eds.), Advances in Neural Information Processing Systems, volume 27,
Curran Associates, Inc., 2014. URL: https://proceedings.neurips.cc/paper_files/paper/2014/file/d523773c6b194f37b938d340d5d02232-Paper.pdf.
[12] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán,
E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Xlm-roberta-base, https://huggingface.
co/xlm-roberta-base, 2020.
[13] T. Miyato, A. M. Dai, I. Goodfellow, Adversarial training methods for semi-supervised textclassification, arXiv preprint arXiv:1605.07725 (2016).
[14] X. Chen, C. Liang, D. Huang, E. Real, K. Wang, Y. Liu, H. Pham, X. Dong, T. Luong,
C.-J. Hsieh, et al., Symbolic discovery of optimization algorithms, arXiv preprint
arXiv:2302.06675 (2023).