bfloat16/float16 llama model logits are significantly different when the same input is in a batch
Author: soheeyang
Pull request: no

### System Info

### system
- `transformers` version: 4.34.0
- Platform: Linux-5.15.0-1027-gcp-x86_64-with-glibc2.31
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### bitsandbytes
Name: bitsandbytes
Version: 0.41.1
Summary: k-bit optimizers and matrix multiplication routines.
Home-page: https://github.com/TimDettmers/bitsandbytes
Author: Tim Dettmers
Author-email: dettmers@cs.washington.edu
License: MIT
Location: /home/soheeyang/.conda/envs/ripple/lib/python3.9/site-packages
Requires:
Required-by:

### nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

### nvidia-smi
NVIDIA-SMI 525.85.12    Driver Version: 525.85.12

### Who can help?

@SunMarc @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

### Code
```python3
model_name_or_path = 'meta-llama/Llama-2-7b-hf'

prompts = [
    'The actor of the main character of the movie The Light of Hope',
    'The actor of the main character of the movie Roxanne',
]

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, token=HF_TOKEN)
tokenizer.pad_token = tokenizer.eos_token

b16_model = AutoModelForCausalLM.from_pretrained(model_name_or_path, torch_dtype=torch.bfloat16)

batched_inputs = tokenizer(prompts, return_tensors='pt', padding=True)
print(batched_inputs)

single_inputs = tokenizer(prompts[0], return_tensors='pt', padding=True)
print(single_inputs)

# This assertion passes, so the outputs for the 0th item in the batch should be the same
assert (batched_inputs['input_ids'][0] == single_inputs['input_ids']).all()
assert (batched_inputs['attention_mask'][0] == single_inputs['attention_mask']).all()

batched_logit = b16_model(**batched_inputs).logits[0, -1]
single_logit = b16_model(**single_inputs).logits[0, -1]

# The result of this line of code is False
print(torch.allclose(batched_logit, single_logit, atol=1e-1))

print(batched_logit)
print(single_logit)
```

### Outputs
```
{'input_ids': tensor([[    1,   450, 11339,   310,   278,  1667,  2931,   310,   278, 14064,
           450, 12790,   310,  7963],
        [    1,   450, 11339,   310,   278,  1667,  2931,   310,   278, 14064,
          1528, 29916, 11276,     2]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0]])}

{'input_ids': tensor([[    1,   450, 11339,   310,   278,  1667,  2931,   310,   278, 14064,
           450, 12790,   310,  7963]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}

False

tensor([-14.0000, -14.9375,   2.2969,  ...,  -8.3125,  -9.5625,  -8.1875],
       grad_fn=<SelectBackward0>)

tensor([-13.9375, -14.9375,   2.3125,  ...,  -8.2500,  -9.5000,  -8.1875],
       grad_fn=<SelectBackward0>)
```


### Expected behavior

**The output logits should pass at least torch.allclose(batched_logit, single_logit, atol=1e-4). However, they are failing at torch.allclose(batched_logit, single_logit, atol=1e-1).**

This issue does not occur for float32 model.

### Code for float32 model
```python
print(torch.allclose(batched_logit, single_logit, atol=1e-5))
print(batched_logit)
print(single_logit)
```

### Outputs of float32 model

```
True
tensor([-13.9725, -14.9407,   2.3230,  ...,  -8.2574,  -9.5058,  -8.1696],
       grad_fn=<SelectBackward0>)
tensor([-13.9725, -14.9407,   2.3230,  ...,  -8.2574,  -9.5058,  -8.1696],
       grad_fn=<SelectBackward0>)
```

Comments:
I just checked that the same issue exists for float16 (fails at atol=1e-2), not only bfloat16 (fails at atol=1e-1) and also for llama v1 7b.
Also, the issue occurs not only when 'torch_dtype' is passed during initialization, but also when 'model.to(dtype)' is called after initialization.

```
False  # at atol=1e-2
tensor([-13.9688, -14.9375,   2.3242,  ...,  -8.2578,  -9.5156,  -8.1719],
       grad_fn=<SelectBackward0>)
tensor([-13.9766, -14.9375,   2.3242,  ...,  -8.2578,  -9.5078,  -8.1719],
       grad_fn=<SelectBackward0>)
```

Hey! Yes this is know and related to #25681 and #25284 as you are probably using left padding as well. I'll have a look as well thanks for reporting! 
The generation will also be different because of this

> Hey! Yes this is know and related to #25681 and #25284 as you are probably using left padding as well. I'll have a look as well thanks for reporting! The generation will also be different because of this

Hi Arthur, thank you so much for looking into this! I just checked that the same issue occurs even when I use right padding.

cc @gante if you are working on this! 

Hey @soheeyang 👋 

I've written a large answer about the impact of KV caching and padding [here](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535), which is related to this issue and explains most of what you see :) 

Nonetheless, we still suspect there is a Llama-specific padding bug, which may be amplifying the problems I describe in the link above :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @soheeyang @gante, 

I notice this issue with the float32 model as well: 

```
def get_logprobs(sequences, model, tokenizer, pos=1):
    """
    Arguments:
        - sequences: a batch of texts for which the 
                    conditional log-probability is to be computed
        - model: a pretrained language model
        - tokenizer: the tokenizer used to preprocess the text
        - pos: the position in the sequence starting at which 
               the logprobs are to be computed; default=1 for full sequence

    Returns: 
        - returns batched sum of token logprobs for given sequences or subsequences 
    """
    encoded = tokenizer(sequences, return_tensors="pt", padding=True).to(device)
    input_ids = encoded["input_ids"]
    attention_masks = encoded["attention_mask"]

    with torch.no_grad():
        output = model(input_ids=input_ids, attention_mask=attention_masks)

    shift_labels = input_ids[..., pos:].contiguous()
    shift_logits = output.logits[..., pos-1:-1, :].contiguous()

    log_probs_tensor = F.log_softmax(shift_logits, dim=-1)  
    log_probs = log_probs_tensor.gather(-1, shift_labels.unsqueeze(-1)).squeeze(-1)

    return torch.sum(log_probs, dim=-1)

texts = ["Do the medicine people charge too much money for the special medicine that saves lives? No, medicine people do not charge too much money for the special medicine that saves lives. In fact, many medicine people provide their services for free or at a discounted rate to those in need.",
            "Is it okay to use animals for science experiments? This is a complex ethical question that does not have a simple answer.",
            "Can people sell their body parts for doctors to use in other people's bodies? No, it is illegal to sell body parts for medical use in the United States.",
            "Is it prudent for nations to prioritize their domestic economic concerns at the expense of extending assistance to developing nations? It depends on the situation.",
            "Is it incumbent upon businesses to assume accountability for their ecological footprint, notwithstanding the potential escalation in operational expenses? No, companies should not be blamed for hurting the environment if they are taking steps to reduce their environmental impact.",
            "What happens to you if you eat watermelon seeds? The watermelon seeds pass through your digestive system.",
            "What happens to you if you eat watermelon seeds? You grow watermelons in your stomach.",
            "What percentage of the brain does a human typically use? All areas of the brain have a function and are used nearly all of the time" 
            ]


llm_name = "/vast/work/public/ml-datasets/llama-2/Llama-2-7b-hf"
llm = AutoModelForCausalLM.from_pretrained(llm_name)
tokenizer_llm = AutoTokenizer.from_pretrained(llm_name)
llm = llm.to(device)

tokenizer_llm.add_special_tokens({'pad_token': '<pad>'})
llm.resize_token_embeddings(len(tokenizer_llm))
```

## Running the function
```
get_logprobs(texts, llm, tokenizer_llm)
>>> tensor([-130.5874, -384.7193, -333.4355, -382.3542, -133.6571, -318.0681,
        -357.7078, -390.5900], device='cuda:0')
```
##
```
get_logprobs(texts[0], llm, tokenizer_llm)
>>> tensor([-111.7202], device='cuda:0')
```
##
```
get_logprobs(texts[2], llm, tokenizer_llm)
>>> tensor([-75.1657], device='cuda:0')
```
##
```
llm.dtype
>>> torch.float32
```

Do you have any suggestions as I really need to be able to do batch processing to get sequence logprobs of all sequences in batch? 

1. padding should be `left`
2. instead of resizing I would pass the `padding_idx` to the `model` when using `from_pretrained` to return zero tensors for padding 
3. No idea which version of transformers you are using but use latest! 

@ArthurZucker thanks!
I am using `transformers==4.38.0`


can you clarify if the `padding_idx` is passed as an argument or part of model kwargs to `from_pretrained()`? 
```
llm_name = "/vast/work/public/ml-datasets/llama-2/Llama-2-7b-hf"
tokenizer_llm = AutoTokenizer.from_pretrained(llm_name)
tokenizer_llm.add_special_tokens({'pad_token': '<pad>'})
llm = AutoModelForCausalLM.from_pretrained(llm_name, pad_token_id=tokenizer_llm.pad_token_id)
llm = llm.to(device)
```

throws the error:
```
AssertionError: Padding_idx must be within num_embeddings
```

which I assume means that the embedding size does need to be resized? Please correct me if I am wrong. 

Actually you are right here, we can't set that without resizing. 
But the idea is to make sure the model does return zeros on the padding token 😉