How to implement LayoutLM for information extraction
Author: SandyRSK
Pull request: no
Labels: name: wontfix

name: New model

Hi,
I am new to nlp
Can someone please guide me on How to implement the layoutLM using transformers for information extraction (from images like receipt)

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-large-uncased")
model = AutoModel.from_pretrained("microsoft/layoutlm-large-uncased")



Comments:
I think the model's integration is still a work-in-progress @SandyRSK, but will let model author @liminghao1630 chime in if necessary

Similar help required. I want to use the layoutLM model finetuned on DocBank data. As per my understanding, this will be a token classification task, but any example code will be extremely helpful.
Thanks  

@SandyRSK the integration is still on-going. You may refer to https://github.com/microsoft/unilm/tree/master/layoutlm if you want to use the model right now.

Is this still work-in-progress?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi, is it possible to load the docbank pretrained model in your implementation?

It should be possible to load the LayoutLM weights from the official repository in ours. Please feel free to try and upload it on the hub! 

Let us know if you run into trouble by opening a new issue and we'll take a look.

Thanks! I'm just worried about the changes you made (all the classes commented with `Copied from transformers.models.bert.modeling_bert.BertEncoder with Bert->LayoutLM`): https://huggingface.co/transformers/_modules/transformers/models/layoutlm/modeling_layoutlm.html

Right, this is for maintainability purposes. Instead of refactoring models in shared layers, we instead keep the entire forward pass in a single file. Doing it this way makes it easy to edit a single file or to read a paper side by side with the code, whereas refactored code would be harder to navigate in. These comments allow us to ensure that the files do not diverge: we have tests that check that the contents of `LayoutEncoder` is identical to `BertEncoder`, with the only change being utterances of "bert" changed to "layoutlm".

As you can see, the differences between BERT and LayoutLM are especially the embeddings.

Hi, I loaded the pretained DocBank model, which is a LayoutLM from the original unilm repository, using the LayoutLM from HuggingFace. 

I get a warning of the form:

```
Some weights of the model checkpoint at layoutlm_large_500k_epoch_1/ were not used when initializing LayoutLMForTokenClassification: ['bert.embeddings.word_embeddings.weight' [...]
- This IS expected if you are initializing LayoutLMForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LayoutLMForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of LayoutLMForTokenClassification were not initialized from the model checkpoint at layoutlm_large_500k_epoch_1/ and are newly initialized: ['embeddings.word_embeddings.weight' [...]
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

For the full warning see https://paste.ofcode.org/WxFiUf8bYL3TEjc9jLAG4J

However, even after running model.eval(), I get random outputs. 

Edit:
Moreover, the outputs don't seem to make any sense.
Finally, >90% tokens always get the predicted label. But this label changes at each inference pass.