Add LayoutLMv2 OnnxConfig
Author: chainyo
Pull request: yes

# What does this PR do?

Add LayoutLMv2 OnnxConfig to make this model available for conversion.

I took the same config as `LayoutLM` and added the adapted shebang.

## Who can review?

Models: @LysandreJik @lewtun 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16309). All of your documentation changes will be reflected on that endpoint.

There's quite some interest for ONNX support of LayoutLMv2, see #14368 and #14555.

> There's quite some interest for ONNX support of LayoutLMv2, see #14368 and #14555.

Ok thanks for the links, I will look at them when I got some time and start improving the actual PR (it seems that the last PR has been closed due to inactivity, so I will take what I can from this and improve my own PR)

Thanks!

So, with the previous add the dummy inputs generation is working, but it seems that the `forward` function is facing a problem with the position embedding tensors.

There is a problem when `forward` is trying to calculate text embeddings, the `position_embeddings` tensor has not the same shape and can't be added with others

```bash
embeddings = inputs_embeds + position_embeddings + spatial_position_embeddings + token_type_embeddings
torch.Size([1, 14, 768]) torch.Size([1, 63, 768]) torch.Size([1, 14, 768]) torch.Size([1, 14, 768])
```

Here is the error trace :

```bash
Traceback (most recent call last):
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/onnx/__main__.py", line 99, in <module>
    main()
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/onnx/__main__.py", line 81, in main
    onnx_inputs, onnx_outputs = export(
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/onnx/convert.py", line 308, in export
    return export_pytorch(preprocessor, model, config, opset, output, tokenizer=tokenizer)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/onnx/convert.py", line 171, in export_pytorch
    raise err
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/onnx/convert.py", line 148, in export_pytorch
    onnx_export(
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/__init__.py", line 275, in export
    return utils.export(model, args, f, export_params, verbose, training,
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/utils.py", line 88, in export
    _export(model, args, f, export_params, verbose, training, input_names, output_names,
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/utils.py", line 689, in _export
    _model_to_graph(model, args, verbose, input_names,
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/utils.py", line 458, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args,
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/utils.py", line 422, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/utils.py", line 373, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/jit/_trace.py", line 1160, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/jit/_trace.py", line 118, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1039, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 893, in forward
    text_layout_emb = self._calc_text_embeddings(
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 758, in _calc_text_embeddings
    embeddings = inputs_embeds + position_embeddings + spatial_position_embeddings + token_type_embeddings
RuntimeError: The size of tensor a (14) must match the size of tensor b (63) at non-singleton dimension 1
```

I'm investigating to fix the tensors problem.


@ChainYo have you find a solution for tensors problem ? 🤗

> @ChainYo have you find a solution for tensors problem ? hugs

It seems that one tensor size is changing for no reason on my previous tests. I will dig more this week, I had no time last week.

Great,if you want, we can organise a Google meet and looking the issue together. You can send me an email at : hubarjulien@gmail.com

@ChainYo I am getting errors while using your PR during inference time. I am using the below code for token-classification (FUNSD):
processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")
image_path = '../funsd/page1.png'
image = Image.open(image_path).convert("RGB")
words, bboxes = get_words_and_boxes_textract(textract_client, image_path)
encoded_inputs = processor(image, words, boxes=bboxes, padding="max_length", truncation=True, return_tensors="pt")
for k,v in encoded_inputs.items():
      encoded_inputs[k] = v.to(device)
dt = datasets.Dataset.from_dict(encoded_inputs)
outputs = loaded_ort_model.evaluation_loop(dt)

Error:
InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:token_type_ids

Any help is really appreciated. @ChainYo @lewtun @michaelbenayoun 

You can try to add to the processor function token_type_ids like this :
```
  input_ids = batch['input_ids'].to(device)
  bbox = batch['bbox'].to(device)
  image = batch['image'].to(device)
  attention_mask = batch['attention_mask'].to(device)
  token_type_ids = batch['token_type_ids'].to(device)
  labels = batch['labels'].to(device)
  # forward pass
   outputs = model(input_ids=input_ids, bbox=bbox, image=image, attention_mask=attention_mask, 
                            token_type_ids=token_type_ids, labels=labels)
```

> outputs = model(input_ids=input_ids, bbox=bbox, image=image, attention_mask=attention_mask, 
>                             token_type_ids=token_type_ids, labels=labels)

there is a method called evaluation_loop which does inferencing for loaded onnx models. it expects only huggingface dataset as per https://github.com/huggingface/optimum.

Please look at the snippet which I posted earlier.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi! Thanks for working on this @ChainYo 🙏 I am interested in this work. Are you still working on this? If you no longer have the time or resources to do so, would you be able to provide any next steps as you see it for this ONNX export to work? Thank you for your time and effort 🤗 

Hey @ChainYo, regarding your error with the tests - my guess is that the dummy data generation is the culprit. My suggestion would be to:

* First pick an input that works with the `torch` model
* Export the model to ONNX and check the forward pass still works with the same input
* Generalise to the dummy input case

@malcolmgreaves I don't even remember where I was with this issue months ago. But I will try to work on it this week if I can.

@lewtun You are right! It seems that someone solved the things I was trying to achieve by doing it with LayoutLMv3. So I will check that and see what I can apply to v2.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @ChainYo 
Were you able to check this?
Any direction on this would be really helpful.

> @lewtun You are right! It seems that someone solved the things I was trying to achieve by doing it with LayoutLMv3. So I will check that and see what I can apply to v2.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.