LayoutLMv2Processor padding/truncation issues
Author: AmineKheldouni
Pull request: no

Hi there,

I had a troubleshooting concerning LayoutLMv2 during inference. I am using the LayoutLMv2 Processor to encode my inputs, feeding the processor with the inputs (the full image, texts and bounding boxes) in order to encode them and call the LayoutLMv2 model. The issue is that the processor doesn't seem to respond to my padding/truncation strategies and fails to batch encode the inputs as they are not of the same size.
 

## Environment info
 

- `transformers` version: 4.10.0

- Platform: Linux-4.15.0-143-generic-x86_64-with-debian-buster-sid

- Python version: 3.7.11

- PyTorch version (GPU?): 1.8.0 (True)

- Tensorflow version (GPU?): 2.5.0 (True)

- Flax version (CPU?/GPU?/TPU?): not installed (NA)

- Jax version: not installed

- JaxLib version: not installed

- Using GPU in script?: Yes

- Using distributed or parallel set-up in script?: No

 

### Who can help

 

@LysandreJik

 

## Information

 
Model I am using LayoutLMv2:

 

The problem arises when using:

* [x] the official example scripts:

```python
from transformers import LayoutLMv2Processor, AutoModelForTokenClassification, AutoConfig, AutoTokenizer
[...]
self.processor = LayoutLMv2Processor.from_pretrained(model_path, revision="no_ocr")
[...]
encoded_inputs = self.processor(image.convert('RGB'),
                                                      ocrs,
                                                      boxes=bboxes,
                                                      word_labels=self.config["kie_classes"],
                                                      padding="max_length",
                                                      truncation=True,
                                                      return_tensors="pt",
                                                  )
```
The task I am working on is:

* [x] an official GLUE/SQUaD task: Token classification / NER

 

## To reproduce

 

Steps to reproduce the behavior:

 

1. Get a Pillow RGB document image.

2. Given the bounding boxes of this image (during inference), get the OCR transcripts of bboxes.

3. Load the LayoutLMv2 processor and model.

4. Try encoding the inputs (image, ocrs, bboxes) with the processor (using the script provided above.

 

* [x] Stack traces:

```
File "/src/tools/miniconda/envs/myenv/lib/python3.7/site-packages/transformers/models/layoutlmv2/processing_layoutlmv2.py", line 201, in __call__
    **kwargs,
  File "/src/tools/miniconda/envs/myenv/lib/python3.7/site-packages/transformers/models/layoutlmv2/tokenization_layoutlmv2_fast.py", line 313, in __call__
    **kwargs,
  File "/src/tools/miniconda/envs/myenv/lib/python3.7/site-packages/transformers/models/layoutlmv2/tokenization_layoutlmv2_fast.py", line 449, in encode_plus
    **kwargs,
  File "/src/tools/miniconda/envs/myenv/lib/python3.7/site-packages/transformers/models/layoutlmv2/tokenization_layoutlmv2_fast.py", line 658, in _encode_plus
    **kwargs,
  File "/src/tools/miniconda/envs/myenv/lib/python3.7/site-packages/transformers/models/layoutlmv2/tokenization_layoutlmv2_fast.py", line 607, in _batch_encode_plus
    return BatchEncoding(sanitized_tokens, sanitized_encodings, tensor_type=return_tensors)
  File "/src/tools/miniconda/envs/myenv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 210, in __init__
    self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
  File "/src/tools/miniconda/envs/myenv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 722, in convert_to_tensors
    "Unable to create tensor, you should probably activate truncation and/or padding "
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length.
```

## Expected behavior

It is expected from `LayoutLMv2Processor` to provide encoded inputs in order to feed the LayoutLMv2 model. However, even though the `padding` and `truncation` are set to True (or to a suitable `PaddingStrategy` or `TruncationStrategy`) the processor seems to not take that into account.

Many thanks in advance for your help.

Comments:
Hi, 

Can you provide a Colab notebook or code snippet that reproduces your issue? I see that you're only providing a single image to the processor, so that's actually not a batch of examples, but just a single example. 

Also, you need to make sure that the words, boxes and word labels you provide are all of equal length.

Hi,

Thank you for your reply. I agree I am not performing a batch encoding since there is only one image. 

Actually, I followed your guide in `https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LayoutLMv2/FUNSD/Inference_with_LayoutLMv2ForTokenClassification.ipynb` but didn't understand why we needed word labels during inference (perhaps only for evaluating predictions against ground truth ?). 

I debugged my code on Colab, and it was indeed due to the length of the word labels. Thank you very much !

Cheers.