Paligemma causal attention still not causal ?
Author: edmondja
Pull request: no

### System Info

The bug was supposed not to appear anymore with 4.41.1, yet I the model still doesnt seem causal to me (at least at the end of the sequence, where pure text tokens are supposed to be inputed and predicted).
@ArthurZucker @molbap https://github.com/huggingface/transformers/pull/30967#issuecomment-2126589661

I am adding the zipped code and console results
here https://filedn.eu/lB85dAuefYtHyPANtFCdHfB/arthur_debug_attempt.zip
and here 
[arthur_debug_attempt.zip](https://github.com/huggingface/transformers/files/15420339/arthur_debug_attempt.zip)
And the code again here copied and pasted :
```python
from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
from PIL import Image
import requests

model_id = "google/paligemma-3b-pt-224"

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

model = PaliGemmaForConditionalGeneration.from_pretrained(model_id).eval()
processor = AutoProcessor.from_pretrained(model_id)

prompt = "caption en"
model_inputs = processor(text=prompt, images=image, return_tensors="pt")

# FIRST DEBUGGING METHOD ANALYZE INFLUENCE OF EXTRA TOKEN ON CAUSALITY
pred0 = model(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'],
             attention_mask=model_inputs['attention_mask'], ).logits
print((pred0, pred0[0, -2].argmax()))
pred1 = model(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'],
             attention_mask=model_inputs['attention_mask'], ).logits
print((pred1, pred1[0, -2].argmax())) # same before last token and logits
model_inputs['input_ids'][0,-1] *= 0 + 12 # modify last token
pred2 = model(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'],
             attention_mask=model_inputs['attention_mask'], ).logits
print((pred2, pred2[0, -2].argmax())) # different before last (penultimate) token and logits



# # SECOND DEBUGGING METHOD ANALYZE DECODING
# #THE TWO FIRST METHODS GIVE THE SAME RESULT
# input_len = model_inputs["input_ids"].shape[-1]
# model_inputs = processor(text=prompt, images=image, return_tensors="pt")
# with torch.inference_mode():
#     generation = model.generate(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'], 
#                                 max_new_tokens=1000, attention_mask=model_inputs['attention_mask'], 
#                                 do_sample=False)
#     generation = generation[0][input_len:]
#     decoded = processor.decode(generation, skip_special_tokens=False)
#     print(decoded)

# #SECOND METHOD GIVING THE SAME RESULTS (manual greedy decoding)
# model_inputs = processor(text=prompt, images=image, return_tensors="pt")
# eos_not_met = True
# while eos_not_met:
#     # model_inputs['cache_position']
#     # model_inputs['position_ids']
#     with torch.inference_mode():
#         pred = model(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'],
#                       attention_mask=model_inputs['attention_mask'], ).logits
#     new_tok = torch.argmax(pred[0,-1])
    
#     toks = torch.cat([model_inputs['input_ids'], new_tok[None][None]], dim=1)
#     print(processor.decode(toks[0], skip_special_tokens=False))
    
#     eos_not_met = new_tok.item() != processor.tokenizer.eos_token_id
#     model_inputs['input_ids'] = toks
#     model_inputs['attention_mask'] = torch.ones_like(model_inputs['input_ids']) 

# #THIRD METHOD NOT GIVING THE RIGHT RESULT (manual greedy decoding with extra token in the end)
# model_inputs = processor(text=prompt, images=image, return_tensors="pt")
# eos_not_met = True
# while eos_not_met:
#     # model_inputs['cache_position']
#     # model_inputs['position_ids']
#     toks = torch.cat([model_inputs['input_ids'], new_tok[None][None]], dim=1) # add an extra token to interfere
#     with torch.inference_mode():
#         pred = model(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'],
#                       attention_mask=model_inputs['attention_mask'], ).logits
#     new_tok = torch.argmax(pred[0,-2])
#     toks[0, -1] = new_tok # replacing the interefence token with actual next token
    
#     print(processor.decode(toks[0], skip_special_tokens=False))
    
#     eos_not_met = new_tok.item() != processor.tokenizer.eos_token_id
#     model_inputs['input_ids'] = toks
#     model_inputs['attention_mask'] = torch.ones_like(model_inputs['input_ids']) 

```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Predict twice and I get the same logits vectors
2. Add an extra token to the input sequence, and see the first logits vectors changed

### Expected behavior

Adding an extra token shouldn't influence the first logits'values.

Comments:
```
model_inputs = processor(text=prompt, images=image, return_tensors="pt")
```
you need to include `suffix=suffix` in this call to get returned labels, and token_type_ids
If you add suffix (your target labels after the prompt) like
```
model_inputs = processor(text=prompt, suffix=suffix, images=image, return_tensors="pt")
````
the output keys will have the required `token_type_ids`
```
model_inputs.keys() # dict_keys(['input_ids', 'token_type_ids', 'attention_mask', 'pixel_values', 'labels'])
```
the model relies on `token_type_ids` to build the causal mask:
https://github.com/huggingface/transformers/blob/6d3d5b1039559a7cc03965cc8c2a6ff0291d83fc/src/transformers/models/paligemma/modeling_paligemma.py#L319


the main thing to note is that paligemma is fully attended to over the image block AND prefix, (text in your case). There isn't supposed to be causality on the first half of the text input.

>Expected behavior
Adding an extra token shouldn't influence the first logits'values.

[This isn't true for tokens in the prefix!](https://github.com/google-research/big_vision/blob/main/big_vision/configs/proj/paligemma/README.md)

Hi @edmondja, +1 to Peter's comment above - the model is non-causal for the prefix and causal for the suffix. If you were to observe what you do at suffix level, that would be worrying indeed, but not in this case. Closing this issue, feel free to reopen if your investigations bring something new! 

Thanks ! @molbap @probicheaux 
But why not raising an exception ? Because generation mode doesnt need token_type_ids ?

Yes, exactly. When generating, everything is a "prefix" and there is full block attention on it. 

In line:

https://github.com/huggingface/transformers/blob/main/src/transformers/models/paligemma/modeling_paligemma.py#L399

The token_type_ids for the pad_tokens is 0, which changes the causal mask during training, and the model pays attention to the pad tokens too. The image and prefix tokens should be paid full attention too, but not the pad tokens. This does not happen during inference as is_training is False during inference.