use_cache=False makes a huge difference in Paligemma
Author: edmondja
Pull request: no

### System Info

Hello I think something is not right with Paligemma, I figured out the logloss was great but my accuracy or other metrics related to generated sequence were bad.
I realized generating tokens one by one improved performance, and then I realized .generate gave different results (even on the example given from the HF page of Paligemma).
This could be more easily observed with complex problems, where the differences are bigger. @molbap @ArthurZucker 

```from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
from PIL import Image
import requests
import torch
 
model_id = "google/paligemma-3b-mix-224"
 
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)
 
model = PaliGemmaForConditionalGeneration.from_pretrained(model_id).eval()
processor = AutoProcessor.from_pretrained(model_id)
 
# Instruct the model to create a caption in Spanish
prompt = "segment wall"
model_inputs = processor(text=prompt, images=image, return_tensors="pt")
input_len = model_inputs["input_ids"].shape[-1]
 
with torch.inference_mode():
    print('generate automatically')
    generation = model.generate(**model_inputs, max_new_tokens=100, do_sample=False)
    generation = generation[0][input_len:]
    decoded = processor.decode(generation, skip_special_tokens=True)
    print(decoded)
    
    print('generate manually token per token')
    while True:
        pred = model(input_ids= model_inputs["input_ids"], pixel_values=model_inputs['pixel_values'],
                      attention_mask=torch.ones_like(model_inputs["input_ids"]), ).logits
        new_tok = torch.argmax(pred[0,-1])
        model_inputs["input_ids"] = torch.cat([model_inputs["input_ids"], new_tok[None][None]], dim=1)
        decoded = processor.tokenizer.decode(model_inputs["input_ids"][0])
        print(decoded)
        if '<eos>' in decoded:
            break
```
PS: use_cache=False makes the problem disappear.


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Generate something difficult to generate (like segmentation of a hardly recognizable item like "segment wall" on the example in HF with the car)
2. Generate tokens one by one and see the difference


### Expected behavior

.generate by default should use greedy decoding I think, therefore in my opinion generating tokens one by one should give the same result, not better ones.

Comments:
Hi @edmondja, thanks for the issue! I'll take a look soon, but one note is that PaliGemma is creating a fully noncausal attention mask when you're passing `attention_mask` (full attention on image+ text prefix), but decoding is supposed to be causal if you use generate(). 

> Hi @edmondja, thanks for the issue! I'll take a look soon, but one note is that PaliGemma is creating a fully noncausal attention mask when you're passing `attention_mask` (full attention on image+ text prefix), but decoding is supposed to be causal if you use generate().

Alright thanks

Hi @edmondja check out this discussion: https://huggingface.co/google/paligemma-3b-mix-448/discussions/6 I think I found a fix, will be able to push tomorrow, thought to let you know! thanks again for the investigation!

Great, thanks for paying careful attention to our issue.