new model: IDEFICS via HuggingFaceM4
Author: stas00
Pull request: yes

**important: The following notes are for my team mates and they won't work for anybody else as the data isn't ready for the public yet. should be made public next week **

Meanwhile to try it out:

```
$ git clone https://github.com/huggingface/transformers -b add-model-idefics
$ cd transformers

$ cat generate.py
import torch
from transformers import IdeficsForVisionText2Text, AutoProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"

checkpoint = "HuggingFaceM4/idefics-9b"
#checkpoint = "HuggingFaceM4/tiny-random-idefics"

model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)
processor = AutoProcessor.from_pretrained(checkpoint)

prompts = [
    [
        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg",
        "Describe this image.",
        
        "Assistant: An image of two kittens in grass.",
        
        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg",
        "Describe this image.",
        
        "Assistant:",
    ],
    [
        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg",
        "Describe this image.",
        
        "Assistant: An image of a dog wearing funny glasses.",

        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg",
        "Describe this image.",

        "Assistant:",
    ],
]

# batched mode
inputs = processor(prompts, return_tensors="pt").to(device)
# single sample mode
#inputs = processor(prompts[0], return_tensors="pt").to(device)

generated_ids = model.generate(**inputs, max_length=100)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
for i,t in enumerate(generated_text):
    print(f"{i}:\n{t}\n")
```

and then run:
```
CUDA_VISIBLE_DEVICES=0 PYTHONPATH=src python generate.py
```

# Demos 

A PR with examples/demos, including finetuning, is here: https://github.com/huggingface/notebooks/pull/418

# TODOs before merging
- [ ] make the models public - which coincides with the announcement/release


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Is it possible to be a private repo ? ;-) The m4 repo from huggingface organisation does not exist

Thank you for your interest, @flozi00 - please give us some time. It says WIP because it's not ready for a public consumption. I edited the OP to clarify that.

I'm not able to rebase as this recently merged PR https://github.com/huggingface/transformers/pull/25174 breaks `tests/models/idefics/test_image_processing_idefics.py::IdeficsImageProcessingTest::test_torchvision_numpy_transforms_equivalency`

cc: @amyeroberts, if I need to adapt our image processing code please let me know - the function in question is called here:

https://github.com/huggingface/transformers/pull/24796/files#diff-e1b90eb52340b91c2471bac7c6fd34c67c7cd530050c607852fd426f397b3b3fR162

@sgugger, I addressed your feedback and this PR is ready for a detailed review. 

Thank you!

Thank you, @sgugger, @HugoLaurencon and @leot13 for your reviews - I have addressed everything you have raised.

> ```
> prompts = [
>     [
>         "User:",
>         "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg",
>         "Describe this image."
>         
>         "Assistant: An image of two kittens in grass.",
>         
>         "User:",
>         "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg",
>         "Describe this image".
>         
>         "Assistant:",
>     ],
>     [
>         "User:",
>         "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg",
>         "Describe this image."
>         
>         "Assistant: An image of a dog wearing funny glasses.",
> 
>         "User:",
>         "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg",
>         "Describe this image".
> 
>         "Assistant:",
>     ],
> ]
> ```

For posterity, that part of the OP (i can't edit unfortunately) is missing some "," (commas) at some end of string (for instance `"Describe this image".` -> `"Describe this image",`). this is important for the tokenization in particular when we call processor with `add_end_of_utterance_token=True`.

I can edit if need be. You should also be able to push commits to this branch, since it's in the main fork and you have write permissions @VictorSanh :-)

Thanks a lot, @gante, for the suggestions - merged