The Demo Provided on Defics2 Document Cannot Run
Author: Kamichanw
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.40.2
- Platform: Linux-4.15.0-76-generic-x86_64-with-glibc2.27
- Python version: 3.10.14
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@ArthurZucker @stevhliu

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. The official [demo](https://huggingface.co/docs/transformers/main/en/model_doc/idefics2#usage-tips)  for `chat_template` cannot work due to out of index of text embedding. Although `processor.tokenizer` has token `<end_of_utterance>`，the  text model still isn't able to get its embedding.

2. In official [demo](https://huggingface.co/docs/transformers/main/en/model_doc/idefics2#transformers.Idefics2ForConditionalGeneration) for `Idefics2ForConditionalGeneration`, the `images` is not passed to the model, which is definitely a bug.


### Expected behavior

1. The model should be able to process `<end_of_utterance>`.
2. The variable `images` should also be passed to `processor` with texts.

Comments:
cc @Rocketknight1 for the chat template maybe

I think this is more related to the VLM docs! cc @NielsRogge or @zucchini-nlp maybe, but feel free to ask me if there are any chat template problems!

Hey @Kami-chanw !

1. The `<end_of_utterance>` should be present in model's embedding. Can you verify that you're using the correct checkpoints for model and processor, because that token is absent in `base` model and `base` model should not be used with chat templates. I can confirm that the demo code works and token 32002 is in the embedding layer
2. Right, seems to be a typo. Will make a fix ;)

Thanks for your help @zucchini-nlp, and I still have some problems on chat template @Rocketknight1 and `Idefics2` model itself.
I'm currently running an in-context learning task with Idefics2 (both `Idefics2-8b` and `Idefics2-8b-base`), but the results confuse me.

I fed the following prompt into Idefics2-8b, where the questions and images are from VQAv2 and COCO2014 dataset, respectively.
```
Instruction:Provide an answer to the question. Use the image to answer.
Question:<image>Where is he looking?
Short answer: down
Question:<image>What are the people in the background doing?
Short answer: watching
Question:<image>What is he on top of?
Short answer: 
```
The three QA pair share the same [image](https://cocodataset.org/#explore?id=262148).
But the `Idefics2-8b` generated (The part enclosed in parentheses is the prompt I fed), and `Idefics2-8b-base` also gave a similarly bad result:
```
(Instruction:Provide an answer to the question. Use the image to answer.
Question: Where is he looking?
Short answer: down
Question: What are the people in the background doing?
Short answer: watching
Question: What is he on top of?
Short answer:) 
Question: where is
```
I wonder:
1. Is it because I am using prompt incorrectly or is it a problem with the model itself?
2. Can I change the processor's `chat_template` to my prompt template, and then apply `apply_chat_template` to the original QA pairs? (Although it's not really used in conversation mode)
3. If I need to use the `<end of utterance>` token, do I need to add it to the end of every `Instruction: ...`, `Question:...`, and `Short answer:...` sentence?

@Kami-chanw From the example you gave, the chat seems to be formatted incorrectly for `Ideifics-8b` model. 

1. I strongly recommend to use the processor's chat template, the general format for multi-turn is like this:

```bash
'User:<image>{PROMPT}<end_of_utterance>\nAssistant: {ANSWER}<end_of_utterance>\nUser:<image>{PROMPT}<end_of_utterance>\nAssistant:'
```

Also note that this model was tuned with QA in zero-shot setting so adding more turns and trying to do few-shot does not guarantee better generation quality. But using the official template could get a better performance than you have now.

2. You surely can use your own template, but note that it might degrade generation quality (as we can see from above) because the model was trained in a different format than you are using

3. I added an example above for formatting

@zucchini-nlp Thanks, I now decide to use `idefics-8b-base` model to continue my in-context learning experiments, since `idefics-8b` is fine-tuned by some specific templates and may not obtain better generation quality when doing few-shot.

Will my current prompt template work properly on `idefics-8b-base` ? Or, could you give me any other suggestions? Thanks again.

@Kami-chanw yes, you can try, but still that doesn't guarantee a good ICL, If you want to do ICL, it's better to 
- either use another model which is trained specifically fro ICL. Afaik there's no such model in transformers yet, but there's VILA which we might support later. Or you can try llava-interleave models trained with multi-image setting as another optio
-  or try to tune base Idefics model for your own use-case with your own formatting