`VisionEncoderDecoderModel.generate()` rejects argument `interpolate_pos_encoding=True`
Author: jungomi
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.0-69-generic-x86_64-with-glibc2.35
- Python version: 3.9.7
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When trying to use a different size than the `VisionEncoderDecoderModel` was trained with, you are supposed to pass `interpolate_pos_encoding=True` to the forward method. That in itself works fine and hence the training does too, but when using the `generate()` method, it rejects `interpolate_pos_encoding` during the validation of the model kwargs.

For example using TrOCR, which was trained on an image size of 384x384, and changing the input size of the images to 128x768:

```py
import torch
from transformers import VisionEncoderDecoderModel

model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten")

images = torch.rand((1, 3, 128, 768))

# interpolate_pos_encoding=True should be passed to the forward method in order to use
# different image sizes, but the validation of .generate() rejects it.
model.generate(pixel_values=images, interpolate_pos_encoding=True)
```

Produces the error:

```
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/mjungo/ocr/summer-school/repro_interpolate_pos_encoding.py:10 in <module>                  │
│                                                                                                  │
│    7                                                                                             │
│    8 # interpolate_pos_encoding=True should be passed to the forward method in order to use      │
│    9 # different image sizes, but the validation of .generate() rejects it.                      │
│ ❱ 10 model.generate(pixel_values=images, interpolate_pos_encoding=True)                          │
│   11                                                                                             │
│                                                                                                  │
│ /home/mjungo/miniconda3/lib/python3.9/site-packages/torch/utils/_contextlib.py:115 in            │
│ decorate_context                                                                                 │
│                                                                                                  │
│   112 │   @functools.wraps(func)                                                                 │
│   113 │   def decorate_context(*args, **kwargs):                                                 │
│   114 │   │   with ctx_factory():                                                                │
│ ❱ 115 │   │   │   return func(*args, **kwargs)                                                   │
│   116 │                                                                                          │
│   117 │   return decorate_context                                                                │
│   118                                                                                            │
│                                                                                                  │
│ /home/mjungo/miniconda3/lib/python3.9/site-packages/transformers/generation/utils.py:1271 in     │
│ generate                                                                                         │
│                                                                                                  │
│   1268 │   │   generation_config = copy.deepcopy(generation_config)                              │
│   1269 │   │   model_kwargs = generation_config.update(**kwargs)  # All unused kwargs must be m  │
│   1270 │   │   generation_config.validate()                                                      │
│ ❱ 1271 │   │   self._validate_model_kwargs(model_kwargs.copy())                                  │
│   1272 │   │                                                                                     │
│   1273 │   │   # 2. Set generation parameters if not already defined                             │
│   1274 │   │   logits_processor = logits_processor if logits_processor is not None else LogitsP  │
│                                                                                                  │
│ /home/mjungo/miniconda3/lib/python3.9/site-packages/transformers/generation/utils.py:1144 in     │
│ _validate_model_kwargs                                                                           │
│                                                                                                  │
│   1141 │   │   │   │   unused_model_args.append(key)                                             │
│   1142 │   │                                                                                     │
│   1143 │   │   if unused_model_args:                                                             │
│ ❱ 1144 │   │   │   raise ValueError(                                                             │
│   1145 │   │   │   │   f"The following `model_kwargs` are not used by the model: {unused_model_  │
│   1146 │   │   │   │   " generate arguments will also show up in this list)"                     │
│   1147 │   │   │   )                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: The following `model_kwargs` are not used by the model: ['interpolate_pos_encoding'] (note: typos in the
generate arguments will also show up in this list)
```

### Expected behavior

It was expected to accept `interpolate_pos_encoding=True` and pass it to the forward method.

Note: When removing the verification check, it works as expected, so this is purely an issue of the validation being incorrect, i.e. missing the additional arguments that are allowed for this kind of model.

Comments:
Hi @jungomi

Thank you for raising this issue! Looks valid, I will take a look quickly.

Hi @gante 

It looks we need to update `_validate_model_kwargs` in some way, so some extra but necessary (for correctness) arguments could be allowed even if not in `prepare_inputs_for_generation` (here of the generic class `VisioniEncoderDecoder`.

Let me try a fix.

@jungomi 

The PR is merged. You can try the latest commit on the `main` branch if you would like to 🤗