LayoutXLM

## [](#overview)Overview

LayoutXLM was proposed in [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://arxiv.org/abs/2104.08836) by Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei. It’s a multilingual extension of the [LayoutLMv2 model](https://arxiv.org/abs/2012.14740) trained on 53 languages.

The abstract from the paper is the following:

_Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually-rich document understanding tasks recently, which demonstrates the great potential for joint learning across different modalities. In this paper, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for visually-rich document understanding. To accurately evaluate LayoutXLM, we also introduce a multilingual form understanding benchmark dataset named XFUN, which includes form understanding samples in 7 languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese), and key-value pairs are manually labeled for each language. Experiment results show that the LayoutXLM model has significantly outperformed the existing SOTA cross-lingual pre-trained models on the XFUN dataset._

One can directly plug in the weights of LayoutXLM into a LayoutLMv2 model, like so:

```
from transformers import LayoutLMv2Model

model = LayoutLMv2Model.from_pretrained("microsoft/layoutxlm-base")```

Note that LayoutXLM has its own tokenizer, based on [LayoutXLMTokenizer](/docs/transformers/v4.30.0/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizer)/[LayoutXLMTokenizerFast](/docs/transformers/v4.30.0/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizerFast). You can initialize it as follows:

```
from transformers import LayoutXLMTokenizer

tokenizer = LayoutXLMTokenizer.from_pretrained("microsoft/layoutxlm-base")```

Similar to LayoutLMv2, you can use [LayoutXLMProcessor](/docs/transformers/v4.30.0/en/model_doc/layoutxlm#transformers.LayoutXLMProcessor) (which internally applies [LayoutLMv2FeatureExtractor](/docs/transformers/v4.30.0/en/model_doc/layoutlmv2#transformers.LayoutLMv2FeatureExtractor) and [LayoutXLMTokenizer](/docs/transformers/v4.30.0/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizer)/[LayoutXLMTokenizerFast](/docs/transformers/v4.30.0/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizerFast) in sequence) to prepare all data for the model.

As LayoutXLM’s architecture is equivalent to that of LayoutLMv2, one can refer to [LayoutLMv2’s documentation page](layoutlmv2) for all tips, code examples and notebooks.

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/microsoft/unilm).

## [](#transformers.LayoutXLMTokenizer)LayoutXLMTokenizer

### class transformers.LayoutXLMTokenizer

[](#transformers.LayoutXLMTokenizer)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/layoutxlm/tokenization_layoutxlm.py#L149)

( vocab\_filebos\_token = '
'eos\_token = '
'sep\_token = '
'cls\_token = '
'unk\_token = '<unk>'pad\_token = '<pad>'mask\_token = '<mask>'cls\_token\_box = \[0, 0, 0, 0\]sep\_token\_box = \[1000, 1000, 1000, 1000\]pad\_token\_box = \[0, 0, 0, 0\]pad\_token\_label = -100only\_label\_first\_subword = Truesp\_model\_kwargs: typing.Union\[typing.Dict\[str, typing.Any\], NoneType\] = None\*\*kwargs )

Adapted from [RobertaTokenizer](/docs/transformers/v4.30.0/en/model_doc/roberta#transformers.RobertaTokenizer) and [XLNetTokenizer](/docs/transformers/v4.30.0/en/model_doc/xlnet#transformers.XLNetTokenizer). Based on [SentencePiece](https://github.com/google/sentencepiece).

This tokenizer inherits from [PreTrainedTokenizer](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer) which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

#### \_\_call\_\_

[](#transformers.LayoutXLMTokenizer.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/layoutxlm/tokenization_layoutxlm.py#L444)

( text: typing.Union\[str, typing.List\[str\], typing.List\[typing.List\[str\]\]\]text\_pair: typing.Union\[typing.List\[str\], typing.List\[typing.List\[str\]\], NoneType\] = Noneboxes: typing.Union\[typing.List\[typing.List\[int\]\], typing.List\[typing.List\[typing.List\[int\]\]\]\] = Noneword\_labels: typing.Union\[typing.List\[int\], typing.List\[typing.List\[int\]\], NoneType\] = Noneadd\_special\_tokens: bool = Truepadding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsetruncation: typing.Union\[bool, str, transformers.tokenization\_utils\_base.TruncationStrategy\] = Nonemax\_length: typing.Optional\[int\] = Nonestride: int = 0pad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = Nonereturn\_token\_type\_ids: typing.Optional\[bool\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_overflowing\_tokens: bool = Falsereturn\_special\_tokens\_mask: bool = Falsereturn\_offsets\_mapping: bool = Falsereturn\_length: bool = Falseverbose: bool = True\*\*kwargs ) → [BatchEncoding](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.BatchEncoding)

Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences with word-level normalized bounding boxes and optional labels.

#### build\_inputs\_with\_special\_tokens

[](#transformers.LayoutXLMTokenizer.build_inputs_with_special_tokens)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/layoutxlm/tokenization_layoutxlm.py#L316)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.LayoutXLMTokenizer.build_inputs_with_special_tokens.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs to which the special tokens will be added.
-   [](#transformers.LayoutXLMTokenizer.build_inputs_with_special_tokens.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.

List of [input IDs](../glossary#input-ids) with the appropriate special tokens.

Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. An XLM-RoBERTa sequence has the following format:

-   single sequence: `
 X 
`
-   pair of sequences: `
 A 

 B 
`

#### get\_special\_tokens\_mask

[](#transformers.LayoutXLMTokenizer.get_special_tokens_mask)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/layoutxlm/tokenization_layoutxlm.py#L342)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = Nonealready\_has\_special\_tokens: bool = False ) → `List[int]`

Parameters

-   [](#transformers.LayoutXLMTokenizer.get_special_tokens_mask.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs.
-   [](#transformers.LayoutXLMTokenizer.get_special_tokens_mask.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.
-   [](#transformers.LayoutXLMTokenizer.get_special_tokens_mask.already_has_special_tokens)**already\_has\_special\_tokens** (`bool`, _optional_, defaults to `False`) — Whether or not the token list is already formatted with special tokens for the model.

A list of integers in the range \[0, 1\]: 1 for a special token, 0 for a sequence token.

Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method.

#### create\_token\_type\_ids\_from\_sequences

[](#transformers.LayoutXLMTokenizer.create_token_type_ids_from_sequences)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/layoutxlm/tokenization_layoutxlm.py#L370)

( token\_ids\_0: typing.List\[int\]token\_ids\_1: typing.Optional\[typing.List\[int\]\] = None ) → `List[int]`

Parameters

-   [](#transformers.LayoutXLMTokenizer.create_token_type_ids_from_sequences.token_ids_0)**token\_ids\_0** (`List[int]`) — List of IDs.
-   [](#transformers.LayoutXLMTokenizer.create_token_type_ids_from_sequences.token_ids_1)**token\_ids\_1** (`List[int]`, _optional_) — Optional second list of IDs for sequence pairs.

List of zeros.

Create a mask from the two sequences passed to be used in a sequence-pair classification task. XLM-RoBERTa does not make use of token type ids, therefore a list of zeros is returned.

#### save\_vocabulary

[](#transformers.LayoutXLMTokenizer.save_vocabulary)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/layoutxlm/tokenization_layoutxlm.py#L427)

( save\_directory: strfilename\_prefix: typing.Optional\[str\] = None )

## [](#transformers.LayoutXLMTokenizerFast)LayoutXLMTokenizerFast

### class transformers.LayoutXLMTokenizerFast

[](#transformers.LayoutXLMTokenizerFast)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/layoutxlm/tokenization_layoutxlm_fast.py#L152)

( vocab\_file = Nonetokenizer\_file = Nonebos\_token = '
'eos\_token = '
'sep\_token = '
'cls\_token = '
'unk\_token = '<unk>'pad\_token = '<pad>'mask\_token = '<mask>'cls\_token\_box = \[0, 0, 0, 0\]sep\_token\_box = \[1000, 1000, 1000, 1000\]pad\_token\_box = \[0, 0, 0, 0\]pad\_token\_label = -100only\_label\_first\_subword = True\*\*kwargs )

Construct a “fast” LayoutXLM tokenizer (backed by HuggingFace’s _tokenizers_ library). Adapted from [RobertaTokenizer](/docs/transformers/v4.30.0/en/model_doc/roberta#transformers.RobertaTokenizer) and [XLNetTokenizer](/docs/transformers/v4.30.0/en/model_doc/xlnet#transformers.XLNetTokenizer). Based on [BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models).

This tokenizer inherits from [PreTrainedTokenizerFast](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizerFast) which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

#### \_\_call\_\_

[](#transformers.LayoutXLMTokenizerFast.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/layoutxlm/tokenization_layoutxlm_fast.py#L269)

( text: typing.Union\[str, typing.List\[str\], typing.List\[typing.List\[str\]\]\]text\_pair: typing.Union\[typing.List\[str\], typing.List\[typing.List\[str\]\], NoneType\] = Noneboxes: typing.Union\[typing.List\[typing.List\[int\]\], typing.List\[typing.List\[typing.List\[int\]\]\]\] = Noneword\_labels: typing.Union\[typing.List\[int\], typing.List\[typing.List\[int\]\], NoneType\] = Noneadd\_special\_tokens: bool = Truepadding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsetruncation: typing.Union\[bool, str, transformers.tokenization\_utils\_base.TruncationStrategy\] = Nonemax\_length: typing.Optional\[int\] = Nonestride: int = 0pad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = Nonereturn\_token\_type\_ids: typing.Optional\[bool\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_overflowing\_tokens: bool = Falsereturn\_special\_tokens\_mask: bool = Falsereturn\_offsets\_mapping: bool = Falsereturn\_length: bool = Falseverbose: bool = True\*\*kwargs ) → [BatchEncoding](/docs/transformers/v4.30.0/en/main_classes/tokenizer#transformers.BatchEncoding)

Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences with word-level normalized bounding boxes and optional labels.

## [](#transformers.LayoutXLMProcessor)LayoutXLMProcessor

Constructs a LayoutXLM processor which combines a LayoutXLM feature extractor and a LayoutXLM tokenizer into a single processor.

[LayoutXLMProcessor](/docs/transformers/v4.30.0/en/model_doc/layoutxlm#transformers.LayoutXLMProcessor) offers all the functionalities you need to prepare data for the model.

It first uses [LayoutLMv2FeatureExtractor](/docs/transformers/v4.30.0/en/model_doc/layoutlmv2#transformers.LayoutLMv2FeatureExtractor) to resize document images to a fixed size, and optionally applies OCR to get words and normalized bounding boxes. These are then provided to [LayoutXLMTokenizer](/docs/transformers/v4.30.0/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizer) or [LayoutXLMTokenizerFast](/docs/transformers/v4.30.0/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizerFast), which turns the words and bounding boxes into token-level `input_ids`, `attention_mask`, `token_type_ids`, `bbox`. Optionally, one can provide integer `word_labels`, which are turned into token-level `labels` for token classification tasks (such as FUNSD, CORD).

#### \_\_call\_\_

[](#transformers.LayoutXLMProcessor.__call__)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/layoutxlm/processing_layoutxlm.py#L47)

( imagestext: typing.Union\[str, typing.List\[str\], typing.List\[typing.List\[str\]\]\] = Nonetext\_pair: typing.Union\[typing.List\[str\], typing.List\[typing.List\[str\]\], NoneType\] = Noneboxes: typing.Union\[typing.List\[typing.List\[int\]\], typing.List\[typing.List\[typing.List\[int\]\]\]\] = Noneword\_labels: typing.Union\[typing.List\[int\], typing.List\[typing.List\[int\]\], NoneType\] = Noneadd\_special\_tokens: bool = Truepadding: typing.Union\[bool, str, transformers.utils.generic.PaddingStrategy\] = Falsetruncation: typing.Union\[bool, str, transformers.tokenization\_utils\_base.TruncationStrategy\] = Nonemax\_length: typing.Optional\[int\] = Nonestride: int = 0pad\_to\_multiple\_of: typing.Optional\[int\] = Nonereturn\_token\_type\_ids: typing.Optional\[bool\] = Nonereturn\_attention\_mask: typing.Optional\[bool\] = Nonereturn\_overflowing\_tokens: bool = Falsereturn\_special\_tokens\_mask: bool = Falsereturn\_offsets\_mapping: bool = Falsereturn\_length: bool = Falseverbose: bool = Truereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = None\*\*kwargs )

This method first forwards the `images` argument to [**call**()](/docs/transformers/v4.30.0/en/model_doc/layoutlmv2#transformers.LayoutLMv2FeatureExtractor.__call__). In case [LayoutLMv2FeatureExtractor](/docs/transformers/v4.30.0/en/model_doc/layoutlmv2#transformers.LayoutLMv2FeatureExtractor) was initialized with `apply_ocr` set to `True`, it passes the obtained words and bounding boxes along with the additional arguments to [**call**()](/docs/transformers/v4.30.0/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizer.__call__) and returns the output, together with resized `images`. In case [LayoutLMv2FeatureExtractor](/docs/transformers/v4.30.0/en/model_doc/layoutlmv2#transformers.LayoutLMv2FeatureExtractor) was initialized with `apply_ocr` set to `False`, it passes the words (`text`/``text_pair`) and `boxes` specified by the user along with the additional arguments to [__call__()](/docs/transformers/v4.30.0/en/model_doc/layoutxlm#transformers.LayoutXLMTokenizer.__call__) and returns the output, together with resized `images``.

Please refer to the docstring of the above two methods for more information.