Mistral with flash attention 2 and right padding
Author: dakinggg
Pull request: no

### System Info

- `transformers` version: 4.34.0
- Platform: Linux-5.4.0-148-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.20.3
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@younesbelkada 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

If you run a batch through mistral with flash attention 2 with right padding, you get

```
ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to  call `tokenizer.padding_side  = 'left'` before tokenizing the 
input.
```

I am not doing generation, just calling forward. Is the error message incorrect and you actually meant to prevent _all_ usage of right padding here? Or is the implementation wrong and this was meant to only prevent _generate_ usage of right padding? Or perhaps I am missing something else. Thanks!

### Expected behavior

Either right padding is ok for calling forward, or the error message correctly states the problem.

Comments:
Having the same issues
ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to  call `tokenizer.padding_side  = 'left'` before tokenizing the input. 

cc @younesbelkada 

Indeed forward should be supported but not generation, will raise a patch soon for this

Hey @younesbelkada,  I am still seeing this error after set `tokenizer.padding_side  = 'left'`
This is my demo notebook: https://colab.research.google.com/drive/1sVqbYEOqjJYl7CzNzXzviEBB6A984cMq?usp=sharing

Tokenizer already set with left padding
![截屏2024-01-10 下午10 53 07](https://github.com/huggingface/transformers/assets/25971665/0ce9f431-8a8f-4236-8c71-846f183f455e)

Still have: `ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Mistral. Make sure to  call `tokenizer.padding_side  = 'left'` before tokenizing the input.`
![截屏2024-01-10 下午10 54 31](https://github.com/huggingface/transformers/assets/25971665/f031de6c-6b25-4be8-8b4a-7075e70785a5)

Not sure is this because of trl or something wrong within transformer?

transformers             4.36.2
trl                      0.7.7
peft                     0.6.0


I am having the same issue. Even after I set the tokenizer padding side = left, this error still occurs during training.

Thanks everyone for reporting, might be an issue with TRL I think, let me have a deeper look and get back ASAP

> Thanks everyone for reporting, might be an issue with TRL I think, let me have a deeper look and get back ASAP

I opened a issue in trl as well: https://github.com/huggingface/trl/issues/1217#issuecomment-1888582975

You need to set `use_cache = False` for both the main and reference model. See my comment here: https://github.com/huggingface/trl/issues/1217#issuecomment-1889282654

> You need to set `use_cache = False` for both the main and reference model. See my comment here: [huggingface/trl#1217 (comment)](https://github.com/huggingface/trl/issues/1217#issuecomment-1889282654)

I try your solution and it work like a charm but does set use_cache to False make tokenizer.padding_side = 'left' during evaluation and 'right' during training. I read the doc about use cache here: [link](https://discuss.huggingface.co/t/what-is-the-purpose-of-use-cache-in-decoder/958/4) but seem like it just reduce the inference time.  Can you explain why it work like a charm sir ?

https://blog.csdn.net/qq_41496421/article/details/136654890?spm=1001.2014.3001.5502

There is a solution here, welcome to discuss