IdeficsProcessor: Changing `image_size` will result in `RuntimeError`
Author: xhluca
Pull request: no

### System Info

- `transformers` version: 4.33.0
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (gpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@VictorSanh 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

If we change the `image_size` in the official example:

```python
import torch
from transformers import IdeficsForVisionText2Text, AutoProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"

checkpoint = "HuggingFaceM4/tiny-random-idefics"

model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)
processor = AutoProcessor.from_pretrained(checkpoint, image_size=224*2)

# We feed to the model an arbitrary sequence of text strings and images. Images can be either URLs or PIL Images.
prompts = [
    [
        "User: What is in this image?",
        "https://upload.wikimedia.org/wikipedia/commons/8/86/Id%C3%A9fix.JPG",
        "<end_of_utterance>",

        "\nAssistant: This picture depicts Idefix, the dog of Obelix in Asterix and Obelix. Idefix is running on the ground.<end_of_utterance>",

        "\nUser:",
        "https://static.wikia.nocookie.net/asterix/images/2/25/R22b.gif/revision/latest?cb=20110815073052",
        "And who is that?<end_of_utterance>",

        "\nAssistant:",
    ],
]

# --batched mode
inputs = processor(prompts, add_end_of_utterance_token=False, return_tensors="pt").to(device)
# --single sample mode
# inputs = processor(prompts[0], return_tensors="pt").to(device)

# Generation args
exit_condition = processor.tokenizer("<end_of_utterance>", add_special_tokens=False).input_ids
bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

generated_ids = model.generate(**inputs, eos_token_id=exit_condition, bad_words_ids=bad_words_ids, max_length=100)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
for i, t in enumerate(generated_text):
    print(f"{i}:\n{t}\n")
```


We get the following error:
```
RuntimeError: The size of tensor a (50177) must match the size of tensor b (226) at non-singleton dimension 1
```

Here's the full output:
```

[4]:

!transformers-cli env

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.5
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.33.0
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.2 (gpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

[3]:

import torch

from transformers import IdeficsForVisionText2Text, AutoProcessor

​

device = "cuda" if torch.cuda.is_available() else "cpu"

​

checkpoint = "HuggingFaceM4/tiny-random-idefics"

​

model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)

processor = AutoProcessor.from_pretrained(checkpoint, image_size=224*2)

[3]:

# We feed to the model an arbitrary sequence of text strings and images. Images can be either URLs or PIL Images.

prompts = [

    [

        "User: What is in this image?",

        "https://upload.wikimedia.org/wikipedia/commons/8/86/Id%C3%A9fix.JPG",

        "<end_of_utterance>",

​

        "\nAssistant: This picture depicts Idefix, the dog of Obelix in Asterix and Obelix. Idefix is running on the ground.<end_of_utterance>",

​

        "\nUser:",

        "https://static.wikia.nocookie.net/asterix/images/2/25/R22b.gif/revision/latest?cb=20110815073052",

        "And who is that?<end_of_utterance>",

​

        "\nAssistant:",

    ],

]

​

# --batched mode

inputs = processor(prompts, add_end_of_utterance_token=False, return_tensors="pt").to(device)

# --single sample mode

# inputs = processor(prompts[0], return_tensors="pt").to(device)

​

# Generation args

exit_condition = processor.tokenizer("<end_of_utterance>", add_special_tokens=False).input_ids

bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

​

generated_ids = model.generate(**inputs, eos_token_id=exit_condition, bad_words_ids=bad_words_ids, max_length=100)

generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)

for i, t in enumerate(generated_text):

    print(f"{i}:\n{t}\n")

Both `max_new_tokens` (=128) and `max_length`(=100) seem to have been set. `max_new_tokens` will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[3], line 37
     34 exit_condition = processor.tokenizer("<end_of_utterance>", add_special_tokens=False).input_ids
     35 bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids
---> 37 generated_ids = model.generate(**inputs, eos_token_id=exit_condition, bad_words_ids=bad_words_ids, max_length=100)
     38 generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
     39 for i, t in enumerate(generated_text):

File /opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py:1602, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)
   1585     return self.assisted_decoding(
   1586         input_ids,
   1587         assistant_model=assistant_model,
   (...)
   1598         **model_kwargs,
   1599     )
   1600 if generation_mode == GenerationMode.GREEDY_SEARCH:
   1601     # 11. run greedy search
-> 1602     return self.greedy_search(
   1603         input_ids,
   1604         logits_processor=logits_processor,
   1605         stopping_criteria=stopping_criteria,
   1606         pad_token_id=generation_config.pad_token_id,
   1607         eos_token_id=generation_config.eos_token_id,
   1608         output_scores=generation_config.output_scores,
   1609         return_dict_in_generate=generation_config.return_dict_in_generate,
   1610         synced_gpus=synced_gpus,
   1611         streamer=streamer,
   1612         **model_kwargs,
   1613     )
   1615 elif generation_mode == GenerationMode.CONTRASTIVE_SEARCH:
   1616     if not model_kwargs["use_cache"]:

File /opt/conda/lib/python3.10/site-packages/transformers/generation/utils.py:2450, in GenerationMixin.greedy_search(self, input_ids, logits_processor, stopping_criteria, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, synced_gpus, streamer, **model_kwargs)
   2447 model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
   2449 # forward pass to get next token
-> 2450 outputs = self(
   2451     **model_inputs,
   2452     return_dict=True,
   2453     output_attentions=output_attentions,
   2454     output_hidden_states=output_hidden_states,
   2455 )
   2457 if synced_gpus and this_peer_finished:
   2458     continue  # don't waste resources running the code we don't need

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.10/site-packages/transformers/models/idefics/modeling_idefics.py:1506, in IdeficsForVisionText2Text.forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, pixel_values, image_encoder_embeddings, perceiver_embeddings, image_attention_mask, labels, use_cache, output_attentions, output_hidden_states, return_dict)
   1503 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
   1505 # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
-> 1506 outputs = self.model(
   1507     input_ids=input_ids,
   1508     attention_mask=attention_mask,
   1509     position_ids=position_ids,
   1510     past_key_values=past_key_values,
   1511     inputs_embeds=inputs_embeds,
   1512     pixel_values=pixel_values,
   1513     image_encoder_embeddings=image_encoder_embeddings,
   1514     perceiver_embeddings=perceiver_embeddings,
   1515     image_attention_mask=image_attention_mask,
   1516     use_cache=use_cache,
   1517     output_attentions=output_attentions,
   1518     output_hidden_states=output_hidden_states,
   1519     return_dict=return_dict,
   1520 )
   1522 hidden_states = outputs[0]
   1523 logits = self.lm_head(hidden_states)

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.10/site-packages/transformers/models/idefics/modeling_idefics.py:1215, in IdeficsModel.forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, pixel_values, image_encoder_embeddings, perceiver_embeddings, image_attention_mask, use_cache, output_attentions, output_hidden_states, return_dict)
   1212     pixel_values = pixel_values.contiguous().view(batch_size * num_images, *pixel_values.shape[2:])
   1214     # Get sequence from the vision encoder
-> 1215     image_hidden_states = self.vision_model(pixel_values=pixel_values).last_hidden_state
   1217 elif image_encoder_embeddings is not None:
   1218     batch_size, num_images, image_seq_len, image_hidden_size = image_encoder_embeddings.size()

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.10/site-packages/transformers/models/idefics/vision.py:411, in IdeficsVisionTransformer.forward(self, pixel_values, output_attentions, output_hidden_states, return_dict)
    408 if pixel_values is None:
    409     raise ValueError("You have to specify pixel_values")
--> 411 hidden_states = self.embeddings(pixel_values)
    412 hidden_states = self.pre_layrnorm(hidden_states)
    414 encoder_outputs = self.encoder(
    415     inputs_embeds=hidden_states,
    416     output_attentions=output_attentions,
    417     output_hidden_states=output_hidden_states,
    418     return_dict=return_dict,
    419 )

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.10/site-packages/transformers/models/idefics/vision.py:98, in IdeficsVisionEmbeddings.forward(self, pixel_values)
     96 class_embeds = self.class_embedding.expand(batch_size, 1, -1)
     97 embeddings = torch.cat([class_embeds, patch_embeds], dim=1)
---> 98 embeddings = embeddings + self.position_embedding(self.position_ids)
     99 return embeddings

RuntimeError: The size of tensor a (50177) must match the size of tensor b (226) at non-singleton dimension 1
```

### Expected behavior

224px is very low for images with finegrained details. We should be able to use bigger images (with more image patches).

Comments:
Could you have a look into this @leot13 ? related to what you've been working on

Yes this is being solved in this [PR](https://github.com/huggingface/transformers/pull/26029). 
You should be allowed to interpolate the embeddings. However the model was trained with a fixed image size of 224x224, so it's not guaranteed you'll have improvements in performance

Thanks. @leot13 what version should I upgrade to? If it's not released yet, let me know which version you plan to release it with.

You should install from source if you want to have this. It will be in the next release 😉