Add Idefics 3!
Author: andimarafioti
Pull request: yes
Labels: name: run-slow

# What does this PR do?
Adding the Idefics 3 model.

There are still a few things to do before merging this PR. The results are not exactly the same as with our codebase and the tests are not done. We are opening it to unblock our release. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Models:

- vision models: @amyeroberts


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32473). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

updated main

@andimarafioti I can see that you re-requested review, but there's still some debugging commits being pushed so will hold of reviewing until this has been resolved. I'm going to unsubscribe to prevent getting notifications for every push - as soon as you have a question or want to let me know it's ready, just ping me with `@amyeroberts` and I'll get a notification :) 

@amyeroberts ready to review! There is still the multi-gpu tests that is queued but if those fail I would skip them. They run OOM in the CI on single GPU and there is an issue open for the same on idefics2 https://github.com/huggingface/transformers/issues/32288. If my fix here works, I already opened a PR for that fix as well: https://github.com/huggingface/transformers/pull/32840 

Talked to @molbap and he said that there is an issue with the multi-gpu workers getting stuck in the queue. But it's not related to this PR. 

@andimarafioti Yes, unfortunately we're having issues at the moment with single GPU and multi-GPU runners taking a long time to run / never running cc @ydshieh. 

It looks like the multi GPU tests did eventually run and there's at least one test which is currently failing to be addressed 

rebased on main

Hi, I am trying to run this together with structured generation with outlines (which supports Vision Transformers in a package not yet in PyPI) and am having trouble with that. Crossposting this as an issue in Outlines [here](https://github.com/outlines-dev/outlines/issues/1123) as I am not sure who is supposed to change the code. I try the code below and get the error below that. With some print statements, I found that the "<" in "Got <." in the error message is actually the first character of the prompt "<image> detailed description:", which means that each character in the prompt list gets misinterpreted as an image URL.

Roughly following the instructions in the doc [here](https://outlines-dev.github.io/outlines/reference/models/transformers_vision/) (using `load_image` rather than the `image_from_url` as defined here), I am trying the following code:

```
import outlines
from transformers import Idefics3ForConditionalGeneration

model = outlines.models.transformers_vision(
    "HuggingFaceM4/Idefics3-8B-Llama3",
    model_class=Idefics3ForConditionalGeneration,
    device="cuda",
)

from transformers.image_utils import load_image

description_generator = outlines.generate.text(model)
description_generator(
    ["<image> detailed description:"],
    [[load_image("https://upload.wikimedia.org/wikipedia/commons/2/25/Siam_lilacpoint.jpg")]]
)
```
I get the following error:
```
---------------------------------------------------------------------------
UnidentifiedImageError                    Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/transformers/image_utils.py:372, in load_image(image, timeout)
    371     b64 = base64.decodebytes(image.encode())
--> 372     image = PIL.Image.open(BytesIO(b64))
    373 except Exception as e:

File /opt/conda/lib/python3.10/site-packages/PIL/Image.py:3309, in open(fp, mode, formats)
   3308 msg = "cannot identify image file %r" % (filename if filename else fp)
-> 3309 raise UnidentifiedImageError(msg)

UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7f2d007367f0>

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[9], line 4
      1 from transformers.image_utils import load_image
      3 description_generator = outlines.generate.text(model)
----> 4 description_generator(
      5     ["<image> detailed description:"],
      6     [[load_image("https://upload.wikimedia.org/wikipedia/commons/2/25/Siam_lilacpoint.jpg")]]
      7 )

File /opt/conda/lib/python3.10/site-packages/outlines/generate/api.py:555, in VisionSequenceGeneratorAdapter.__call__(self, prompts, media, max_tokens, stop_at, seed, **model_specific_params)
    549 prompts, media = self._validate_prompt_media_types(prompts, media)
    551 generation_params = self.prepare_generation_parameters(
    552     max_tokens, stop_at, seed
    553 )
--> 555 completions = self.model.generate(
    556     prompts,
    557     media,
    558     generation_params,
    559     self.logits_processor,
    560     self.sampling_params,
    561     **model_specific_params,
    562 )
    564 return self._format(completions)

File /opt/conda/lib/python3.10/site-packages/outlines/models/transformers_vision.py:46, in TransformersVision.generate(self, prompts, media, generation_parameters, logits_processor, sampling_parameters)
     15 def generate(  # type: ignore
     16     self,
     17     prompts: Union[str, List[str]],
   (...)
     21     sampling_parameters: SamplingParameters,
     22 ) -> Union[str, List[str], List[List[str]]]:
     23     """Generate text using `transformers`.
     24 
     25     Arguments
   (...)
     44     The generated text
     45     """
---> 46     inputs = self.processor(prompts, media, padding=True, return_tensors="pt").to(
     47         self.model.device
     48     )
     50     generation_kwargs = self._get_generation_kwargs(
     51         prompts,
     52         generation_parameters,
     53         logits_processor,
     54         sampling_parameters,
     55     )
     56     generated_ids = self._generate_output_seq(prompts, inputs, **generation_kwargs)

File /opt/conda/lib/python3.10/site-packages/transformers/models/idefics3/processing_idefics3.py:288, in Idefics3Processor.__call__(self, images, text, audio, videos, **kwargs)
    286             new_images[-1].append(im)  # already loaded
    287         elif isinstance(im, str):
--> 288             new_images[-1].append(load_image(im))
    290 images = new_images
    291 del new_images

File /opt/conda/lib/python3.10/site-packages/transformers/image_utils.py:374, in load_image(image, timeout)
    372             image = PIL.Image.open(BytesIO(b64))
    373         except Exception as e:
--> 374             raise ValueError(
    375                 f"Incorrect image source. Must be a valid URL starting with `http://` or `https://`, a valid path to an image file, or a base64 encoded string. Got {image}. Failed with {e}"
    376             )
    377 elif isinstance(image, PIL.Image.Image):
    378     image = image

ValueError: Incorrect image source. Must be a valid URL starting with `http://` or `https://`, a valid path to an image file, or a base64 encoded string. Got <. Failed with cannot identify image file <_io.BytesIO object at 0x7f2d007367f0>
```

I tracked down that my issue is due to swapping the order of images and text, see [here](https://github.com/huggingface/transformers/pull/32473/files#r1734779753). Swapping them back in the code makes it work for me, at least when passing a list of prompts and a list of lists of images.

@amyeroberts I'm back from your review :) 

@amyeroberts ready for re-review 

Hi @andimarafioti @amyeroberts,
When I load the processor with `processor.image_processor.do_image_splitting = False`, I encounter an error in the `pixel_shuffle` function located in `transformers/src/transformers/models/idefics3/modeling_idefics3.py`. 
The error message states: "shape '[1, 90, 90, 1152]' is invalid for input of size 9345024" (line 581).

If I use a square image it breaks in `def input_merger()` in `modeling_idefics3.py` with a `shape mismatch` error.

Could you please advise on how to resolve this bug?


I get the same error as @efenocchi when I use `image_splitting=False` in the Idefics3 processor.

When I set `image_splitting=True` I get the following error during accelerated fine-tuning:

```  
File "/home/ec2-user/SageMaker/LLMs/VLM_fine-tuning/Idefics3/finetune.py", line 135, in train
    self.trainer.train()
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/trainer.py", line 1955, in train
    return inner_training_loop(
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/trainer.py", line 2296, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/trainer.py", line 3380, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/trainer.py", line 3427, in compute_loss
    outputs = model(**inputs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1523, in forward
    else self._run_ddp_forward(*inputs, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1359, in _run_ddp_forward
    return self.module(*inputs, **kwargs)  # type: ignore[index]
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/accelerate/utils/operations.py", line 820, in forward
    return model_forward(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/accelerate/utils/operations.py", line 808, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/models/idefics3/modeling_idefics3.py", line 1180, in forward
    outputs = self.model(
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/models/idefics3/modeling_idefics3.py", line 1004, in forward
    inputs_embeds = self.inputs_merger(
  File "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/transformers/models/idefics3/modeling_idefics3.py", line 896, in inputs_merger
    inputs_embeds[special_image_token_mask] = reshaped_image_hidden_states
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
```

Which could be related to #31373  Edited for formatting & context.

The second one is probably why there was that input_embeds.clone 💡 I was asked and wasn't sure why that was there so I removed it, but the error certainly points towards that

@Quinn-Meyer-Sustainment , the error you showed should be fixed with my last commit 🙏 

Thank you @andimarafioti for the super quick fix! Commenting to confirm that I can now finetune without error😊

@andimarafioti any suggestions for the error I reported?

Hi @efenocchi , can you provide more information on your bug? A reproducible snippet would be ideal.
There is a test for not splitting the images which passes, so I'm wondering if the test is not catching your situation or if something else is happening. 

@amyeroberts back :) 

@andimarafioti  I can reproduce the same error reported by @efenocchi by setting:

```
self.processor = AutoProcessor.from_pretrained("HuggingFaceM4/Idefics3-8B-Llama3", size= {"longest_edge": 2*364}, do_image_splitting = False)` 
```
and then run the pretrained model. I guess this is not the right way to disable image splitting when running the pretrained model? 

Since my inputs contain multiple images, I basically want to disable the image splitting and also reduce the size of the image (not using the default 4x364). What is the right way to do here? Thanks!

I rebased on main and a test from the chat_template is failing. I'm investigating it. 

There's a PR already addressing the test that's failing here: https://github.com/huggingface/transformers/pull/33462
ie, it's not related to my code

Investigating @woodfrog s report now

I can't reproduce your error @woodfrog . Can you provide more details? This is my snippet that works:

```python
from io import BytesIO

import requests
from PIL import Image

from transformers.models.auto.processing_auto import AutoProcessor


processor = AutoProcessor.from_pretrained(
    "HuggingFaceM4/Idefics3-8B-Llama3", size={"longest_edge": 2 * 364}, do_image_splitting=False
)

image1 = Image.open(
    BytesIO(
        requests.get(
            "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
        ).content
    )
)
inputs = processor(images=image1)

print(len(inputs["pixel_values"]))
# 1
print(len(inputs["pixel_values"][0]))
# 1
print(inputs["pixel_values"][0][0].shape)
# (3, 486, 728)
```

@andimarafioti  Thank you for investigating!

I followed the docs [here](https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3) while changing the `size` and `do_image_splitting` of the processor.

```python
import torch

from transformers.image_utils import load_image
from transformers import AutoProcessor, AutoModelForVision2Seq


DEVICE = "cuda:0"

processor = AutoProcessor.from_pretrained(
    "HuggingFaceM4/Idefics3-8B-Llama3", size={"longest_edge": 2 * 364}, do_image_splitting=False
)

model = AutoModelForVision2Seq.from_pretrained(
    "HuggingFaceM4/Idefics3-8B-Llama3", torch_dtype=torch.bfloat16
).to(DEVICE)

image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")
image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg")
image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "In this image, we can see the city of New York, and more specifically the Statue of Liberty."},
        ]
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "And how about this image?"},
        ]
    },       
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image1, image2], return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}


# Generate
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)


```

And I got:

```
transformers/models/idefics3/modeling_idefics3.py", line 581, in pixel_shuffle
    x = x.view(bsz, height, width, embed_dim)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: shape '[2, 45, 45, 1152]' is invalid for input of size 4792320
```

(Maybe this is not the correct way to change the image resolution and the splitting setting when running the pretrained model. Could you advise on the right solution?)

Okay, now I see the issue. The vision encoder expects square images, and when you don't do the image splitting, the images are not squared, leading to reshaping issues. I'm pushing a change that will fix that and I adapted the tests.

Beware that this will reshape your images to 364x364, since that is the `max_image_size`. If you want to have the images be 728x728, you will need to make more changes.

The amount of tokens that you pass to the language model is determined by the size of the output of the vision encoder. If you feed the vision encoder larger images, you get more tokens. That relationship is determined by the formula:
`int(((image_size // patch_size) ** 2) / (scale_factor**2))`
where `scale_factor` is by default `2` and the `patch_size` for this vision encoder is `14`. Since you are passing `image_size=2*364`, the formula gives: `image_seq_len=676`.

Your code would then be:

```python
import torch

from transformers.image_utils import load_image
from transformers import AutoProcessor, AutoModelForVision2Seq


DEVICE = "cuda:0"

processor = AutoProcessor.from_pretrained(
    "HuggingFaceM4/Idefics3-8B-Llama3", size={"longest_edge": 2 * 364}, max_image_size={"longest_edge": 2 * 364}, image_seq_len=676, do_image_splitting=False
)

model = AutoModelForVision2Seq.from_pretrained(
    "HuggingFaceM4/Idefics3-8B-Llama3", torch_dtype=torch.bfloat16
).to(DEVICE)

image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")
image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg")
image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What do we see in this image?"},
        ]
    },
    {
        "role": "assistant",
        "content": [
            {"type": "text", "text": "In this image, we can see the city of New York, and more specifically the Statue of Liberty."},
        ]
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "And how about this image?"},
        ]
    },       
]

prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=prompt, images=[image1, image2], return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}


# Generate
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)
```


It works now :) Appreciate the formula. Thank you for fixing it so quickly! 

Btw, I found that I have to specify CUDA_VISIBLE_DEVICES=X to use a single GPU. Otherwise, there will be a device error. I roughly went through the discussions above and suppose you have been aware of this.

One question on the formula: Both // and ``int()`` round down, not up, is this intended? E.g. when image_size <
patch_size, the formula implies that 0 tokens are being passed.

@joris-sense , yes if `image_size < patch_size` then it would be zero. The patches for our vision encoder are `14x14`, so an image smaller than that would be rare. But in practice, it would probably still output one latent value, so a `max(formula, 1)` would make sense there. The rest of the rounding is intended and related to how the vision encoder processes the image and how the `pixel_shuffle` works. 

@woodfrog , I wasn't aware of that, but @merveenoyan showed me some errors she had when trying to do multi-gpu training on a computer with too many gpus. I'm testing this with a single GPU and it works well, or with several GPUs but using all of them. Do you only have one GPU? A bit more info would also help here (your setup, a reproducible script).

You're welcome for the quick fix. Thank you for flagging it! I don't think I would have noticed this as I would always split the images (the model was trained this way, so it works better if you keep it)

@andimarafioti  I'm using an 8-GPU machine. I have a complex query pipeline and sometimes have ~10 images (so it's not easy to put a concise snippet here). It works well if I run it with CUDA_VISIBLE_DEVICES=0, but got the error below if not:

```
...
...
transformers/models/idefics3/modeling_idefics3.py", line 957, in forward
    inputs_embeds = self.text_model.get_input_embeddings()(input_ids).to(self.device)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
...
...
torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument index in method wrapper_CUDA__index_select)
```

Thank you for sharing the training details! I have to do this for some multi-image queries; otherwise, the number of image tokens explodes after the splitting. Do you have recommendations for multi-image use cases (say we can have 1-10 images)?  I believe the current processing pipeline doesn't support dynamic image processing, so the safe way for me right now is to disable the image splitting.


hey @andimarafioti I realized IDEFICS3 processor doesn't have a part for labels like PaliGemma does. it was handling things under the hood e.g. [adding EOS token at the end](https://github.com/huggingface/transformers/blob/db72894b48990860a9b8d1ebbfdd9a0342309b8f/src/transformers/models/paligemma/processing_paligemma.py#L241) and also it's an indicator of fine-tuning, so [it preps the data for user](https://github.com/huggingface/transformers/blob/db72894b48990860a9b8d1ebbfdd9a0342309b8f/src/transformers/models/paligemma/processing_paligemma.py#L284). I realized when I was fine-tuning IDEFICS3 I forgot to add EOS token myself so it was just generating forever. WDYT? 

also cc @amyeroberts 

I rebased on main