Paligemma causal attention mask
Author: molbap
Pull request: yes
Labels: name: run-slow

Continuation of #30918 

cc @ArthurZucker ! 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30967). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

sorry to keep nitpicking while you're working on this but our timezones don't overlap so much so I want to communicate while we have the chance! I think this PR should add the `<eos>` token directly to the end of the suffix just like it adds `<bos>` in the correct spot

No problem! I think we can add that eos at the end of the suffix, yes.

Just ran my finetune script to completion and everything is looking correct!!!

I just pushed the <eos> addition but nothing else!  seems good to me :) 

@probicheaux the tokenization issue should be fixed, and all the rest! Feel free to test ! 

Thanks a lot @probicheaux for your work!

Finetuned LoRA looks good

@ArthurZucker I still have the bug using this code

`
vlm = PaliGemmaForConditionalGeneration.from_pretrained(**llm_args)

pred = vlm(pixel_values=tensor, input_ids=input_ids[:, :-1],attention_mask=torch.ones_like(input_ids[:, :-1])).logits

pred = pred[:, -nb_tokens_answer:]

loss = F.cross_entropy(pred.permute((0, 2, 1)), input_ids[:, -nb_tokens_answer:], reduction='mean')
`

can you share the traceback as well? 

> can you share the traceback as well?

I am sure I use transformers 4.41.1, yet I the model doesnt seem causal to me.
I am adding the zipped code and console results 
here https://filedn.eu/lB85dAuefYtHyPANtFCdHfB/arthur_debug_attempt.zip
and here [arthur_debug_attempt.zip](https://github.com/huggingface/transformers/files/15418931/arthur_debug_attempt.zip)

And the code again here copied and pasted :

```python3
from transformers import AutoProcessor, PaliGemmaForConditionalGeneration
from PIL import Image
import requests

model_id = "google/paligemma-3b-pt-224"

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw)

model = PaliGemmaForConditionalGeneration.from_pretrained(model_id).eval()
processor = AutoProcessor.from_pretrained(model_id)

prompt = "caption en"
model_inputs = processor(text=prompt, images=image, return_tensors="pt")

# FIRST DEBUGGING METHOD ANALYZE INFLUENCE OF EXTRA TOKEN ON CAUSALITY
pred0 = model(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'],
             attention_mask=model_inputs['attention_mask'], ).logits
print((pred0, pred0[0, -2].argmax()))
pred1 = model(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'],
             attention_mask=model_inputs['attention_mask'], ).logits
print((pred1, pred1[0, -2].argmax())) # same before last token and logits
model_inputs['input_ids'][0,-1] *= 0 + 12 # modify last token
pred2 = model(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'],
             attention_mask=model_inputs['attention_mask'], ).logits
print((pred2, pred2[0, -2].argmax())) # different before last (penultimate) token and logits



# # SECOND DEBUGGING METHOD ANALYZE DECODING
# #THE TWO FIRST METHODS GIVE THE SAME RESULT
# input_len = model_inputs["input_ids"].shape[-1]
# model_inputs = processor(text=prompt, images=image, return_tensors="pt")
# with torch.inference_mode():
#     generation = model.generate(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'], 
#                                 max_new_tokens=1000, attention_mask=model_inputs['attention_mask'], 
#                                 do_sample=False)
#     generation = generation[0][input_len:]
#     decoded = processor.decode(generation, skip_special_tokens=False)
#     print(decoded)

# #SECOND METHOD GIVING THE SAME RESULTS (manual greedy decoding)
# model_inputs = processor(text=prompt, images=image, return_tensors="pt")
# eos_not_met = True
# while eos_not_met:
#     # model_inputs['cache_position']
#     # model_inputs['position_ids']
#     with torch.inference_mode():
#         pred = model(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'],
#                       attention_mask=model_inputs['attention_mask'], ).logits
#     new_tok = torch.argmax(pred[0,-1])
    
#     toks = torch.cat([model_inputs['input_ids'], new_tok[None][None]], dim=1)
#     print(processor.decode(toks[0], skip_special_tokens=False))
    
#     eos_not_met = new_tok.item() != processor.tokenizer.eos_token_id
#     model_inputs['input_ids'] = toks
#     model_inputs['attention_mask'] = torch.ones_like(model_inputs['input_ids']) 

# #THIRD METHOD NOT GIVING THE RIGHT RESULT (manual greedy decoding with extra token in the end)
# model_inputs = processor(text=prompt, images=image, return_tensors="pt")
# eos_not_met = True
# while eos_not_met:
#     # model_inputs['cache_position']
#     # model_inputs['position_ids']
#     toks = torch.cat([model_inputs['input_ids'], new_tok[None][None]], dim=1) # add an extra token to interfere
#     with torch.inference_mode():
#         pred = model(input_ids=model_inputs['input_ids'], pixel_values=model_inputs['pixel_values'],
#                       attention_mask=model_inputs['attention_mask'], ).logits
#     new_tok = torch.argmax(pred[0,-2])
#     toks[0, -1] = new_tok # replacing the interefence token with actual next token
    
#     print(processor.decode(toks[0], skip_special_tokens=False))
    
#     eos_not_met = new_tok.item() != processor.tokenizer.eos_token_id
#     model_inputs['input_ids'] = toks
#     model_inputs['attention_mask'] = torch.ones_like(model_inputs['input_ids']) 

  
```  

Hey @edmondja, thanks for your interest in this! Can you open a separate Issue with your reproducer, including expected outputs and obtained ones and ping me on it? I can't find an exact reference to a bug in your current description. 

Thank you for your help. I thought it was still the same issue, I will create a new issue. 
@molbap