RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Related Work
Contextual embeddings. Peters et al. ( 2018 ) first proposed to pre-train a bidirectionnal LSTM on large corpora to learn context-sensitive word embeddings that can be used to improve results on downstream tasks.
The Bert model (Devlin et al., 2019 ) instead uses a self-attentive network (i.e. the encoder part of a transformer) to take full advantage of GPU architectures.
However, Bert cannot be trained using the standard language modeling objective as it is not an autoregressive model.
Instead, the authors proposed a masked language modeling objective where the loss aims to increase a reconstruction term on a hidden part of the input.

Document analysis. For document processing, one must take into account spatial information together with textual content. LayoutLM Xu et al. ( 2020 ) extends Bert ’s positional embeddings with spatial positions.
In other words, Bert uses as input embeddings representing the position in the sequence, 3 3 3 Note that some models uses positional encoding without relying on an embedding table, see for examples (Vaswani et al., 2017 , Section 3.5) whereas LayoutLM also includes 6-tuples of embeddings describing (discretized) positions and sizes of the boxes containing one or several words.
This allows the self-attentive network to capture spatial information, which is especially useful for documents containing tables and/or processed with optical character recognition. 4 4 4 OCR’s outputs are composed of boxes containing part of the document text.

LayoutLMv2 Xu et al. ( 2021 ) and v3 Huang et al. ( 2022 ) incorporate more visual information, both as input and in auxiliary training losses.
Moreover, the architecture is modified to integrate relative positional information. Li et al. ( 2021 ) introduced richer positional information,
whereas Wang et al. ( 2022 ) focused on language adaptation during the fine-tuning phase.
Contrary to these works, we focus on the original LayoutLM model as we aim for computational efficiency.

Efficient encoders. The self-attention mechanism has a quadratic-time complexity with respect to the input, which can be slow for long documents.
Several works in document analysis (Nguyen et al., 2021 ; Douzon et al., 2023 , inter alia ) have addressed these drawbacks by integrating more computationally efficient types of attention that are better motivated for document processing.
In this work, we instead explore the impact of the number of layers on downstream results.
Payslips Dataset
We build a novel dataset containing financial pay statements from the insurance domain which we call Payslips .
This dataset consists of a training set of 485 pages and a test set of 126 pages.

The documents originate from data of disability insurance.
In the event of a work-related accident, this insurance product compensates the insured person during their recovery period.
To determine the indemnity amount, the insurer verifies salary information from each insured person’s payslip.
To speed up information processing, it is essential to build tools capable of automatically extracting the useful financial information.
To this end, we worked with insurance professionals and identified nine specific fields, as detailed in Table 1 .
The task is therefore reduced to a standard NER problem, similar to what is done in the FUNSD dataset Jaume et al. ( 2019 ) .
Unlike datasets such as FUNSD or CORD Park et al. ( 2019 ) , Payslips is notably sparse, with a predominant O class, which poses a challenge for the information extraction task. We explain in more details the particularities and challenges of the PAYSLIPS dataset and other usual datasets for NER in documents in Appendix C.

Payslips was annotated in-house by people familiar with the documents.
Then, samples have been validated by insurance specialists to ensure annotation quality.
More details about the annotation process are given in Appendix D .

For privacy reasons, unnecessary or potentially identifying information was altered or deleted.
Moreover, images are not shared as they are not used by our LayoutLM model, but could give visual cues about the entity emitting the files.
Neural Architecture
We use the LayoutLM model Xu et al. ( 2020 ) , which is based on the same neural architecture as Bert Devlin et al. ( 2019 ) , but where inputs are tailored to represent texts in 2D documents.

Given a document where the content has been divided into text blocks, each individual block is encoded as follows: (1) words are tokenized;
(2) each token is represented by an embedding;
(3) 2D positional embeddings are added to word embeddings.
The 2D positional embeddings are 6-tuples representing the coordinates of the block in the page’s image, and its height and width, discretized and normalized between 0 and 1000.

The original LayoutLM could also incorporate an image embedding derived from a vision model.
We do not include this input as it slows down the model without significant impact on downstream task results — sometimes the impact is even negative, see (Xu et al., 2020 , Table 4) .

Then, for the self-attentive part, we use the Base model, which consists of 12 self-attentive layers. Each layer contains 12 heads of dimension 768, as originally defined by Devlin et al. ( 2019 ) .
Finally, during pre-training, the output contextual embeddings are projected into the vocabulary space using a linear layer to compute output logits.
Pre-training
Loss. We pre-train the model using a Masked Language Modeling (MLM) loss, where part of the input is replaced by dummy embeddings and a negative log-likelihood loss aims to reconstruct the masked part. Xu et al. ( 2020 ) also experimented with a
Multi-label Document Classification (MDC) loss, a supervised task aiming to classify each page into predefined categories.
Their results show that MDC degrades performances, therefore we do not include this loss during pre-training.

Data. LayoutLM was pre-trained on the IIT-CDIP dataset Schmidt et al. ( 2002 ); Lewis et al. ( 2006 ) , which gathers 11 millions documents from the U.S. state lawsuits against the tobacco industry in the 1990s.
The authors show that pre-training on this data improves results on several downstream tasks, including NER on SROIE Huang et al. ( 2019 ) and FUNSD Jaume et al. ( 2019 ) .
Unfortunately, during preliminary experiments we observed that LayoutLM tends to under-perform on our internal data.
We suspect IIT-CDIP documents are too different in form and content from insurance documents (see Figure 2 ).
Moreover, adapting information retrieval systems to the insurance domain poses significant challenges due to the sensitivity of the data involved, i.e. we cannot train and distribute models based on internal data due to private data protection laws.

We found no existing datasets of pay statements.
However, some relevant invoice datasets are available. Limam et al. ( 2023 ) provides a dataset of generated invoices, and RVL-CDIP Harley et al. ( 2015 ) includes a subset of invoices from the IIT-CDIP collection.
A more recent and larger dataset, DOCILE Šimsa et al. ( 2023 ) , offers a better match in terms of layout and semantics with our downstream task dataset, Payslips , see Figure 2 .
It contains approximately 900k unlabeled invoices sourced from two public repositories. 5 5 5 https://www.industrydocuments.ucsf.edu/ 6 6 6 https://publicfiles.fcc.gov Although it is more than 10 times smaller than IIT-CDIP , our experimental results shows that it is big enough for pre-training LayoutLM .

Technical details. We pre-train LayoutLM from scratch with the MLM loss on the DOCILE dataset, with similar settings to Xu et al. ( 2020 ) .
We use a minibatch size of 80 , and ran the training for 5 epochs with a learning rate of 5\times 10^{-5} .
We use a cosine scheduler with warmup on 5\% of the updates.
Pre-training is done on 8 NVIDIA Tesla V100 16GB GPUs.
Experiments
We tackle the NER problem using the standard BIO -tagging approach Ramshaw and Marcus ( 1995 ) , i.e. each token is tagged with either O (not in a mention), B-Label (beginning of a mention) or I-Label (inside of a mention), where Label is any mention label allowed in the dataset.
We can then trivially rebuild the full predicted mentions from the predicted BIO tags.

We fine-tune all models with a batch size of 16 for 10 epochs, with a fixed learning rate of 5\times 10^{-5} .
Results
We compare the original LayoutLM pre-trained on IIT-CDIP with our LayoutLM pre-trained (from scratch) on DOCILE on two NER datasets:
(1) The subset of the DOCILE dataset which is labeled 7 7 7 As the annotation of the test set are not available online, we performed evaluation on the validation set. — it contains 6759 and 635 document pages for training and testing, respectively;
(2) Our novel Payslips dataset — statistics are reported in Section 3 .
We fine-tune similarly for both datasets.

We report labeled F1-score averaged on 100 fine-tuning runs in Table 2 .
Precision and recall are reported in Appendix B .
The Base model (using the full 12 layers) produces similar results on DOCILE no matter if pre-training on IIT-CDIP or DOCILE .
However, on our internal Payslips datasets, our model pre-trained on DOCILE outperforms the original one.
Moreover, we observe that our pre-trained model exhibits a way lower variance between fine-tuning runs.

To cope with the high and continuous flow of documents, an insurer might require a faster model. Therefore, we also experimented using a smaller number of self-attentive layers, see Table 2 .
Inference times per model are reported in Table 3 .
On Payslips , when pre-training on DOCILE using only 6 layers, we achieve comparable scores to the off-the-shelf LayoutLM model, while dividing the inference time by almost 2.
Statistical Significance
Domain-specific datasets are often of small sizes, so comparing F1-scores may lead to wrong conclusion if they are not statistically significant.
We follow the original Message Understanding Conference (MUC, Chinchor, 1992 ; Chinchor et al., 1993 ) and rely on the approximate randomization method Noreen ( 1989 ) , which does not require assumptions on the data distribution.
For this test, the null hypothesis is
“ The proposed system and the baseline system do not differ in F1 ”.
The difference is computed in term of absolute F1 difference over many random data splits.
Pseudo-code is given in Appendix A .

In our case, we compare the LayoutLM pre-trained on IIT-CDIP to the one pre-trained on DOCILE , both being fine-tuned on Payslips .
As we did 100 fine-tunings, we took two models with a F1-score difference below 1.00 for the test.
The obtained significance value, 0.0019 , is lower than 0.01 and thus considered highly significant, according to Chinchor ( 1992 , Figure 3) .
Conclusion
In this work, we pre-train from scratch a LayoutLM model using the DOCILE dataset.
Importantly, we show that our model obtain better results on a novel domain-specific NER dataset.
This shows that it is still possible to develop fast and state-of-the-art in-house models that allow commercial usage.

We also release our novel Payslips dataset that can be used to challenge document processing models in financial domains.
Limitations
The novel Payslips dataset is of small size compared to many standard benchmarks.
Unfortunately, specialized domains like insurance not only induce expensive annotation costs,
but it is also difficult to obtain authorization to publicly release the data.
This issue is also common in other domains like biomedical NLP.
Another issue is that Payslips is highly specialized, so interest may be limited.

Experimental results highlight that NLP models may not be useful for production yet, as the F1 scores are below 65.
Acknowledgement
We thank the anonymous reviewers for their comments and suggestions.
This work was performed using HPC resources from GENCI-IDRIS (Grant 2024-AD011015001).
Figure 1: Sample of the newly introduced Payslips dataset for named-entity recognition in the insurance domain.; Sample of the newly introduced Payslips dataset for named-entity recognition in the insurance domain.
Figure 2: Samples from IIT-CDIP (first column), DOCILE (second column) and Payslips (third column) datasets. Invoices from DOCILE and pay statements from Payslips are closer visually and semantically.; Samples from IIT-CDIP (first column), DOCILE (second column) and Payslips (third column) datasets. Invoices from DOCILE and pay statements from Payslips are closer visually and semantically.
Label distribution in Payslips dataset (word level).
F1 scores for named-entity recognition using different pre-training and fine-tuning datasets. Results are averaged on 100 runs with different seeds.
Inference times per page on the Payslips dataset. Tests were conducted on a machine equipped with a single NVIDIA Tesla V100 32GB GPU.
Precision, Recall, and F1 scores for named-entity recognition using different pre-training and fine-tuning datasets. Results are averaged on 100 runs with different seeds.