Introduction
Modern natural language processing pipelines heavily rely on pre-trained neural networks, primarily language models (Schwenk and Gauvain, 2005 ; Jozefowicz et al., 2016 ; Radford et al., 2019 , inter alia ) and context-sensitive embeddings (Schütze, 1998 ; Peters et al., 2018 ; Devlin et al., 2019 , inter alia ) .
The development of neural architectures based on the attention mechanism Bahdanau et al. ( 2015 ) allows to efficiently pre-train them on GPU using large datasets Vaswani et al. ( 2017 ) : most recent networks can contain several hundreds of billions parameters ( e.g., Chowdhery et al., 2023 ) .

Despite their experimental success, commercial use of pre-trained neural networks can be limited for the following reasons.
Firstly, downstream tasks in information retrieval may require to continuously analyze large amounts of data, which prevents the use of the largest models due to inference time bottleneck.
Secondly, applications in specific fields such as financial, medical or insurance, can forbid the use of API-based models due to privacy concerns.
Thirdly and lastly, authors may at some point decide to not publicly share latest versions of their models, or to change the license to forbid commercial use. 1 1 1 See for example LayoutLMv2 and LayoutLMv3 . As such, it is increasingly important to ensure replicability and robustness to changes in training data (including for domain transfer) not only for scientific reasons, but also to ensure widespread commercial deployment.

In this work, we study LayoutLM Xu et al. ( 2020 ) for named-entity recognition (NER) on financial documents from the insurance domain.
Our aim is to understand how such a model can be used in a constrained setting:
Can performance in downstream tasks be improved by pre-training on domain-specific documents, even when the amount of available data is limited?
Can inference time be improved while maintaining downstream performance?
To address this, we pre-train several models from scratch using a smaller, but more relevant, set of publicly available documents.

To evaluate these models, we build a novel dataset, Payslips , that contains anonymized insurance pay statements with annotated financial information for NER, detailed in Table 1 .
Although these documents are private, we have manually anonymized them.
Our experiments show that pre-training on documents that are semantically and structurally similar to those in the downstream task leads to improved performance, even with less training data.
Moreover, if speed of inference is crucial, we show that comparable results can be obtained by using only half the number of layers compared to the original LayoutLM model.

Our contributions can be summarized as follows:

We build and release Payslips , a novel NER dataset of 611 labeled pages of anonymized payslips from the insurance domain;

We pre-train a LayoutLM network using a smaller set of documents ( DOCILE , Šimsa et al., 2023 ) ;

We evaluate our model on Payslips and show that not only does it achieve better F1 scores, but it also has a lower variance;

We show that a smaller model with half the number of layers maintains performances while improving computational efficiency.

Our code and data are publicly available. 2 2 2 https://github.com/buthaya/payslips