TRAINING CUSTOM MODEL USING LAYOUTLMv2!
Author: karndeepsingh
Pull request: no

 If want to extract the information from the scanned document will LayoutLMV2 work? Just wanted your suggestion before I start annotating for training. Below is the example image, Red marks are the entities I want to extract.
![InkedATG-5747901-04_LI](https://user-images.githubusercontent.com/49562460/131701347-b09a3c69-25b5-4c7a-a80e-195c417871a7.jpg)



Comments:
LayoutLMv2 depends on an OCR engine of choice. If you provide this image to `LayoutLMv2FeatureExtractor`, it will by default use the Tesseract OCR engine to extract a list of words + bounding boxes from the image. You'll then need to create word-level labels for the corresponding words, that indicate which are an entity and which are not.

Next, you can use `LayoutLMv2TokenizerFast` to turn the word-level words, boxes and word_labels into token-level input_ids, bbox, attention_mask, token_type_ids.

You can also use your own OCR engine of choice, as explained in the [docs](https://huggingface.co/transformers/master/model_doc/layoutlmv2.html). 

> LayoutLMv2 depends on an OCR engine of choice. If you provide this image to `LayoutLMv2FeatureExtractor`, it will by default use the Tesseract OCR engine to extract a list of words + bounding boxes from the image. You'll then need to create word-level labels for the corresponding words, that indicate which are an entity and which are not.
> 
> Next, you can use `LayoutLMv2TokenizerFast` to turn the word-level words, boxes and word_labels into token-level input_ids, bbox, attention_mask, token_type_ids.
> 
> You can also use your own OCR engine of choice, as explained in the [docs](https://huggingface.co/transformers/master/model_doc/layoutlmv2.html).

How I can annotate my Dataset. I have similar daatset as I shared it above and Don't want to extract every information from the page. Just the red part as shown in the image. So, Do also need to annotate the words which are not required for extraction??

And please also throw some light on how to do custom training on my dataset.

Thanks @NielsRogge 

Yes so LayoutLMv2 treats information extraction as a sequence labeling (NER) problem. It will label all tokens appearing in the document. So if only the words you indicate are relevant, all other words should be labeled as "not an entity". However, this is cheap to do.

> And please also throw some light on how to do custom training on my dataset.

Check out this notebook, which illustrates how to fine-tune `LayoutLMv2ForTokenClassification` for information extraction: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD.ipynb

> Yes so LayoutLMv2 treats information extraction as a sequence labeling (NER) problem. It will label all tokens appearing in the document. So if only the words you indicate are relevant, all other words should be labeled as "not an entity". However, this is cheap to do.
> 
> > And please also throw some light on how to do custom training on my dataset.
> 
> Check out this notebook, which illustrates how to fine-tune `LayoutLMv2ForTokenClassification` for information extraction: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Fine_tuning_LayoutLMv2ForTokenClassification_on_FUNSD.ipynb

Thanks for answering. Do recommend any tool or way so that I can annotate faster on my dataset. That would be a great help. Thanks. @NielsRogge 

For annotation, there are several tools, including:

* TagTog: https://www.tagtog.net/
* Tagalog: https://www.nlp.town/tagalog/

However, such annotation tools are typically not free. 

Perhaps this one might help (it's free): https://github.com/openvinotoolkit/cvat

> For annotation, there are several tools, including:
> 
> * TagTog: https://www.tagtog.net/
> * Tagalog: https://www.nlp.town/tagalog/
> 
> However, such annotation tools are typically not free.

Just wanted a quick understanding. What are the files needed to prepared for annotation? Data preparation and files would be  similar as FUNSD dataset consist off?

@

> For annotation, there are several tools, including:
> 
> * TagTog: https://www.tagtog.net/
> * Tagalog: https://www.nlp.town/tagalog/
> 
> However, such annotation tools are typically not free.
> 
> Perhaps this one might help (it's free): https://github.com/openvinotoolkit/cvat

@NielsRogge Problem with annotation is need to annotate every word with bounding box + its associated text. Even though I have to specify non relevant words as "Non-Relevant" word, I need to provide bounding box information for those words as well. It would become huge task of annotating every word in an image. 

In that case, I'd use PyTesseract (or another OCR engine) to get a list of words + boxes:

```
from transformers import LayoutLMv2FeatureExtractor
from PIL import Image

feature_extractor = LayoutLMv2FeatureExtractor.from_pretrained("microsoft/layoutlmv2-base-uncased")
image = Image.open("your pdf").convert("RGB")

encoding = feature_extractor(image)
words, boxes = encoding.words, encoding.boxes
```

Next, you can initialize the `word_labels` as:

```
word_labels = ["no entity" for _ in range(len(words))]
```

Then, you can search for the entities in the list of words and label the corresponding indices as "entity".

> In that case, I'd use PyTesseract (or another OCR engine) to get a list of words + boxes:
> 
> ```
> from transformers import LayoutLMv2FeatureExtractor
> from PIL import Image
> 
> feature_extractor = LayoutLMv2FeatureExtractor.from_pretrained("microsoft/layoutlmv2-base-uncased")
> image = Image.open("your pdf").convert("RGB")
> 
> encoding = feature_extractor(image)
> words, boxes = encoding.words, encoding.boxes
> ```
> 
> Next, you can initialize the `word_labels` as:
> 
> ```
> word_labels = ["no entity" for _ in range(len(words))]
> ```
> 
> Then, you can search for the entities in the list of words and label the corresponding indices as "entity".

Thanks @NielsRogge, I will try to annotate with the approach you answered. 

> > In that case, I'd use PyTesseract (or another OCR engine) to get a list of words + boxes:
> > ```
> > from transformers import LayoutLMv2FeatureExtractor
> > from PIL import Image
> > 
> > feature_extractor = LayoutLMv2FeatureExtractor.from_pretrained("microsoft/layoutlmv2-base-uncased")
> > image = Image.open("your pdf").convert("RGB")
> > 
> > encoding = feature_extractor(image)
> > words, boxes = encoding.words, encoding.boxes
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Next, you can initialize the `word_labels` as:
> > ```
> > word_labels = ["no entity" for _ in range(len(words))]
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > Then, you can search for the entities in the list of words and label the corresponding indices as "entity".
> 
> Thanks @NielsRogge, I will try to annotate with the approach you answered. 
Just one more question, How we add relative bounding with every label and convert it into the formate required by LayoutLMv2 finally?


> In that case, I'd use PyTesseract (or another OCR engine) to get a list of words + boxes:
> 
> ```
> from transformers import LayoutLMv2FeatureExtractor
> from PIL import Image
> 
> feature_extractor = LayoutLMv2FeatureExtractor.from_pretrained("microsoft/layoutlmv2-base-uncased")
> image = Image.open("your pdf").convert("RGB")
> 
> encoding = feature_extractor(image)
> words, boxes = encoding.words, encoding.boxes
> ```
> 
> Next, you can initialize the `word_labels` as:
> 
> ```
> word_labels = ["no entity" for _ in range(len(words))]
> ```
> 
> Then, you can search for the entities in the list of words and label the corresponding indices as "entity".

I am getting this error! 
![image](https://user-images.githubusercontent.com/49562460/131834175-af83699c-aeee-4594-96a7-c59d42d626a4.png)


For me, that runs fine. Try restarting the runtime and running again.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@NielsRogge What will be the output format in this case?