Incorrect image tokens error while calling PaliGemma (google/paligemma-3b-ft-vqav2-448)
Author: souradipp76
Pull request: no
Labels: name: bug

name: VLM

### System Info

- `transformers` version: 4.48.2
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.11.11
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.2
- Accelerate version: 1.2.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): 2.18.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.10.2 (gpu)
- Jax version: 0.4.33
- JaxLib version: 0.4.33
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: Tesla T4

### Who can help?

@zucchini-nlp @ArthurZucker @amyeroberts

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am trying to use the PaliGemma multimodal model (`google/paligemma-3b-ft-vqav2-448`) for Visual Question Answering. I want to get the loss and logits using `model.forward` by passing the input_ids, attention_mask, pixel_values, and labels. The shape of the inputs are set as per the documentation (https://huggingface.co/docs/transformers/en/model_doc/paligemma#transformers.PaliGemmaForConditionalGeneration)). But the script below throws`ValueError: Number of images does not match number of special image tokens in the input text. Got 2 image tokens in the text but 2048 tokens from image embeddings.` on doing `model.forward`.

1. Run the following code snippet
```
from PIL import Image
import requests
from transformers import AutoProcessor, AutoModelForVision2Seq
import torch

device = torch.device('cuda')
dtype = torch.float16
processor = AutoProcessor.from_pretrained("google/paligemma-3b-ft-vqav2-448")

prompts = ["<image> What is the color of the car? Answer: ", "<image> What is the color of the car? Answer: "]
labels = ["A. Red", "B. Green"]
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)
images = processor.image_processor([image, image], return_tensors="pt").data
inputs = processor.tokenizer(prompts)

input_ids = torch.LongTensor(inputs['input_ids'])
masks = torch.Tensor(inputs['attention_mask'])
labels = torch.LongTensor(processor.tokenizer(labels)['input_ids'])

print(input_ids.shape, labels.shape, images['pixel_values'].shape) # torch.Size([2, 12]) torch.Size([2, 3]) torch.Size([2, 3, 448, 448])

model = AutoModelForVision2Seq.from_pretrained("google/paligemma-3b-ft-vqav2-448", torch_dtype=dtype, device_map = device)
output = model(input_ids=input_ids.to(device), attention_mask=masks.to(device), pixel_values=images["pixel_values"].to(device),labels=labels.to(device))
print(output.loss)
print(output.logits.shape)

```

### Expected behavior

The script should run without error.

print(output.logits.shape) # torch.Size([2, 3, 257152])

Comments:
@souradipp76 hey!

Any reason for calling the `tokenizer` and `image_processor` separately? For multimodal models,  it is strongly advised to call the `processor` directly, Processor usually adds extra steps on top, for VLMs it expands `image` tokens to match the `image_seq_length`. 

```python
inputs = processor(images=[image, image], text=prompts, return_tensors="pt")
```

@zucchini-nlp I was trying to use the padding and truncation parameters for my use case. Is there a way to pass those arguments to the processor directly?
```
 inputs = processor.tokenizer(prompts, padding=True, truncation=True)
```

@souradipp76 yes, you can pass any argument used by tokenizer or image processor directly to `processor`, it will be picked up :)

@zucchini-nlp Thanks for the clarification. Will use the the processor call directly.