Idefics2 model seems to ignore the user textual prompt
Author: jlmeunier
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.46.3
- Platform: Linux-4.18.0-477.27.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.12.7
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes A100-SXM4-40GB
- GPU type: NVIDIA A100-SXM4-40GB

### Who can help?

@amyeroberts 

Hi,

Seems like the textual part of the user's prompt is ignored by the idefics2 model.

I simply took the official example of decoding and replaced "_What’s the difference between these two images?_" by "_Is there a cow in the image? Answer yes or no_."

The result is always "_Assistant: The dog and cat are sleeping on the couch._"

Seems not normal to me.

I must say that an error message is shown before decoding:
`Chat templates should be in a 'chat_template.json' file but found key='chat_template' in the processor's config. Make sure to move your template to its own file.`

I tested with 2 different versions of transformers, as reported in the community page.

Thanks,
JL

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Change the user prompt in the demo script, like below, and see that the assistant response does not change.

Example below:
``` 
import requests
from PIL import Image
from transformers import Idefics2Processor, Idefics2ForConditionalGeneration
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

url_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
url_2 = "http://images.cocodataset.org/val2017/000000219578.jpg"

image_1 = Image.open(requests.get(url_1, stream=True).raw)
image_2 = Image.open(requests.get(url_2, stream=True).raw)
images = [image_1, image_2]

messages = [{
    "role": "user",
    "content": [
        #{"type": "text", "text": "What’s the difference between these two images?"},
        {"type": "text", "text": "Is there a cow in the image? Answer yes or no."},
        {"type": "image"},
        {"type": "image"},
    ],
}]

processor = Idefics2Processor.from_pretrained("HuggingFaceM4/idefics2-8b")
model = Idefics2ForConditionalGeneration.from_pretrained("HuggingFaceM4/idefics2-8b")
model.to(device)

# at inference time, one needs to pass `add_generation_prompt=True` in order to make sure the model completes the prompt
text = processor.apply_chat_template(messages, add_generation_prompt=True)
print(text)
# 'User: What’s the difference between these two images?<image><image><end_of_utterance>\nAssistant:'

inputs = processor(images=images, text=text, return_tensors="pt").to(device)

generated_text = model.generate(**inputs, max_new_tokens=500)
generated_text = processor.batch_decode(generated_text, skip_special_tokens=True)[0]
print("Generated text:", generated_text)
```


### Expected behavior

I would expect a clear change in the assistant response, due to the change of user's prompt.

Comments:
@jlmeunier hey!

Is this a regression compared to some older version? If the model has been generating same way, I don't think it is a bug. Most VLMs are not very good when it comes to multiple images

You might want to try the recently released, SmolVLM or Qwen2.5VL models to see if they are better 🤗 

Hi,
I've no idea if it is a regression or not.
I've tried with a single image (the first one, and same question). The output is also disappointing since it seems the user's question is ignored.
Of course, I can use another model. But the point here is to document, and maybe fix, this issue in Idefics2. It is an issue IMO.

```
User: Is there a cow in the image? Answer yes or no.<image><end_of_utterance>
Assistant:
Generated text: User: Is there a cow in the image? Answer yes or no. 
Assistant: Two cats are on a bed.
```


@jlmeunier I see! Would be nice to check if this is a regression or not, because otherwise it is rather related to the trained model quality than to how model is implemented. I just checked that there has been no regression from v4.40

If you feel like this should be an issue, please open a discussion on model page in the hub. 

@zucchini-nlp thanks for the responses. Actually, I commented on the community page of the Idefics2 model two months ago. Fine with me if we close this issue.

Oke, then let's close this one :)