text-generation results inconsistent depending on padding side
Author: fxmarty
Pull request: no

### System Info

transformers main (but same issue on transformers 4.33.3)

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi, this issue is probably a duplicate but at least here's a simple reproduction.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "meta-llama/Llama-2-7b-chat-hf"

tokenizer = AutoTokenizer.from_pretrained(model_id)
with torch.device("cuda"):
    model = AutoModelForCausalLM.from_pretrained(model_id)

tokenizer.pad_token_id = tokenizer.eos_token_id
tokenizer.padding_side = "right"
inp = tokenizer(["Today I am in Paris and", "Hi there!"], padding=True, return_tensors="pt").to("cuda")

res = model.generate(**inp, num_beams=1, do_sample=False, min_new_tokens=10, max_new_tokens=10)


print(tokenizer.batch_decode(res))
```

With `tokenizer.padding_side = "right"`:
`['
 Today I am in Paris and I am feeling very grateful for this opportunity to explore', "
 Hi there!



\n\nI'm a software engineer with"]`

With `tokenizer.padding_side = "left"`:
`['
 Today I am in Paris and I am feeling very grateful for this opportunity to explore', "



 Hi there! I'm a software engineer with a passion for"]`

You can see that `\n\n` is generated with padding side right, but not with padding side left.

The same issue exist for GPTJ, so probably for many models:
```
['Today I am in Paris and I am going to share with you my experience of', 'Hi there!


Q: I have a question about the new version']

['Today I am in Paris and I am going to share with you my experience of', "


Hi there! I'm a newbie to the forum and I"]
```

### Expected behavior

No different tokens generated

Comments:
Thanks for the report! Feel free to investigate @fxmarty, pinging @gante for when he's back from leave

This is also a duplicate of #25921, and #26380! 

@fxmarty The results are supposed to be different on all `generate`-compatible, decoder-only models :D 

In the attention layer, the token that is used for the `query` to generate the next token is different -- `
` for right-padding and `!` for left-padding (using the original text example, a padded `Hi there!`). Different `query` -> different attention values -> different logits -> very likely different tokens.

Llama 2 kinda works with right-padding, the results are sensible. Most models throw gibberish if left-padding is not used at generation time. Left-padding is our default recommendation for `generate`

@gante Thank you! Not sure to get it. With left padding for `Hi there!`:

```
{
    'input_ids': tensor([
        [1, 20628, 306, 626, 297, 3681, 322],
        [2, 2, 2, 1, 6324, 727, 29991]
    ]),
    'attention_mask': tensor([
        [1, 1, 1, 1, 1, 1, 1],
        [0, 0, 0, 1, 1, 1, 1]
    ])
}
```

With right padding:

```
{
    'input_ids': tensor([
        [1, 20628, 306, 626, 297, 3681, 322],
        [1, 6324, 727, 29991, 2, 2, 2]
    ]),
    'attention_mask': tensor([
        [1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 0, 0, 0]
    ])
}
```

We can see that the attended tokens are identical.

@fxmarty the attended tokens are equal according to the `attention_mask`, yes. 

However, what gets masked is the matmul product between the `query` (from the last token) and the `key` (from all tokens in the sequence) -- [here is the corresponding llama code](https://github.com/huggingface/transformers/blob/e893b1efbbce80b7eaaf24f9e0134450820782b5/src/transformers/models/llama/modeling_llama.py#L384). In other words, what gets masked is `query * key` in the positions where the attention mask is `0`.

This means that, despite `attention_mask` masking the position corresponding to the same tokens, we will observe different results depending on the padding side, as the `query` will be different in the two cases. The model is trained without padding halfway through a sentence, and this is why left-padding produces no distribution shift, and thus better generation results :)

(Note: this is not a Llama-only property, it's a general property of decoder-only LLMs. We have it as one of the most common pitfalls when using LLMs in our [docs](https://huggingface.co/docs/transformers/llm_tutorial#wrong-padding-side).)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @gante I want to know that would it work to take the logit/hidden_state of last non-pad token instead of the last token so that right padding could be used?

something around the line of
replacing 
```outputs.logits[:, -1, :]```
with 
```
# inputs_id: (batch_size, seq_len)
# outputs.logits: (batch_size, seq_len, vocab_size)
pad_mask = (inputs_id == pad_token_id)
batch_size, seq_len = inputs_id.shape[0], inputs_id.shape[1]

last_non_pad_indices = seq_len - 1 - torch.argmax((~pad_mask).to(torch.float32).flip(dims=[1]), dim=1)
logit_for_next_step = outputs.logits[torch.arange(batch_size), last_non_pad_indices, :]
```
Thanks!!