LayoutLMv2 forward pass fails during pytorch.jit.trace forward pass
Author: timothyjlaurent
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.0
- Platform: Mac OS
- Python version: 3.8.9
- PyTorch version :  1.8.1 No GPU
- Tensorflow version (GPU?):

### Who can help

@NielsRogge @LysandreJik 

## Information

Model I am using (LayoutLM V2):

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] my own task or dataset: (give details below)

## To reproduce

I am exporting a LayoutLM model to TorchScript using the documentation here https://huggingface.co/transformers/serialization.html#torchscript . 

Steps to reproduce the behavior:

1.  load layoutlmv2 model with `torchscript=true`
2. use `pytorch.jit.trace` to do a forward pass on the model with dummy input
3. see the stacktrace:

```
/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/torch/tensor.py:587: RuntimeWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  warnings.warn('Iterating over a tensor might cause the trace to be incorrect. '
Traceback (most recent call last):
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/timothy.laurent/src/cies-api/apps/layoutlm-ner/scripts/make_torchscript_trace.py", line 42, in main
    traced_model = torch.jit.trace(model, [dummy_input['input_ids'], dummy_input['bbox'], dummy_input['image'], dummy_input['attention_mask'], dummy_input['token_type_ids']])
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/torch/jit/_trace.py", line 733, in trace
    return trace_module(
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/torch/jit/_trace.py", line 934, in trace_module
    module._c._create_method_from_trace(
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/torch/nn/modules/module.py", line 887, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/torch/nn/modules/module.py", line 860, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 1160, in forward
    outputs = self.layoutlmv2(
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/torch/nn/modules/module.py", line 887, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/torch/nn/modules/module.py", line 860, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/Users/timothy.laurent/.pyenv/versions/cies-layoutlm-ner/lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 873, in forward
    position_ids = position_ids.expand(input_shape)
RuntimeError: The expanded size of the tensor (561) must match the existing size (512) at non-singleton dimension 1.  Target sizes: [8, 561].  Tensor sizes: [1, 512]
python-BaseException
```


Further investigation in the debugger revealed that this line: https://github.com/huggingface/transformers/blob/4be082ce398b1d1a52fbf2693f1adbe87710928d/src/transformers/models/layoutlmv2/modeling_layoutlmv2.py#L1020

somehow mutates the `input_shape` with its use of the `+=` operator.


```python
import typer

import torch
from transformers import LayoutLMv2Tokenizer, LayoutLMv2TokenizerFast


MAX_BATCH_SIZE = 8


def get_layoutlm_dummy_input(
    tokenizer: LayoutLMv2Tokenizer,
    batch_size: int = MAX_BATCH_SIZE,
):
    text = ["This", "is", "the", "dummy", "input"]
    bbox = [[1,2,3,4] for _ in range(len(text))]
    encodings = tokenizer(text=[text] * batch_size, boxes=[bbox] * batch_size, padding="max_length", return_tensors='pt')
    images = torch.rand(batch_size, 3, 224, 224)
    return {
        **encodings.data,
        'image': images
    }


def main(
        model_path: str = WANDB_ARTIFACT_PATH,
        max_batch_size: int = 8
):
    model = LayoutLMv2Model.from_pretrained('microsoft/layoutlmv2-base-uncased')
    tokenizer = LayoutLMv2TokenizerFast.from_pretrained("microsorf/layoutlmv2-base-uncased")
    dummy_input = get_layoutlm_dummy_input(
        tokenizer=tokenizer,
        batch_size=max_batch_size
    )
    # This forward pass works fine
    model(dummy_input['input_ids'], dummy_input['bbox'], dummy_input['image'], dummy_input['attention_mask'], dummy_input['token_type_ids'])
    # this one the input_size gets mutated resulting in the error above
    traced_model = torch.jit.trace(model, [dummy_input['input_ids'], dummy_input['bbox'], dummy_input['image'], dummy_input['attention_mask'], dummy_input['token_type_ids']])
  

```

## Expected behavior

The forward pass should be successful with running the `torch.jit.trace`

Comments:
LayoutLMv2 is currently not tested to be exportable to TorchScript, as can be seen [here](https://github.com/huggingface/transformers/blob/f5d3bb1dd20e9d2c3c5bd66b30032bea6c3f45d8/tests/test_modeling_layoutlmv2.py#L262). One can set `test_torchscript` to `True` in the tests, and then update `modeling_layoutlmv2.py` in order to make the corresponding tests pass. Feel free to open a PR for this.

Great thanks for the input @NielsRogge  -- I fixed the initially reported issue, and then realized that there are other obstacles with making the torchscript trace so will serve the model with torchserve eager mode.