8-Bit LlamaModel returns different outputs for the same input when in different batches
Author: maximek3
Pull request: no

### System Info

- `transformers` version: 4.36.0.dev0
- Platform: Linux-4.18.0-372.58.1.el8_6.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.6
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Tensorflow version (GPU?): 2.9.3 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@SunMarc @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behavior:

Run the following script:

```python
import torch
from transformers import LlamaForCausalLM, LlamaTokenizer

# load a 7B Llama model in 8bit
model = LlamaForCausalLM.from_pretrained(
    "huggyllama/llama-7b", load_in_8bit=True
)
model.eval()

# create two batches that share the same first datapoint
b_01 = {}
b_02 = {}
b_01["input_ids"] = torch.randint(10, 1000, (2, 30))
b_02["input_ids"] = torch.randint(10, 1000, (2, 30))
b_01['input_ids'][0] = b_02['input_ids'][0]

# get the model outputs
with torch.no_grad():
    o_01 = model(b_01["input_ids"])
    o_02 = model(b_02["input_ids"])

# this returns "tensor(True)" and "tensor(False)"
print(torch.all(b_01["input_ids"][0] == b_02["input_ids"][0]))
print(torch.all(o_01["logits"][0] == o_02["logits"][0]))
```

### Expected behavior

We'd expect:

```
tensor(True)
tensor(True)
```

But we get:

```
tensor(True)
tensor(False)
```

,i.e., the model outputs for that same datapoint (`b_01["input_ids"][0]` and `b_02["input_ids"][0]`) are different.

When we load the model with `load_in_8bit=False`, we get the expected behavior, i.e., the model output is consistent for the same datapoint.

Comments:
Hi @maximek3 
I think those small instabilities are expected, can you test out with `torch.allclose()` with different values of `rtol` and  `atol` ? cc @SunMarc @Titus-von-Koeller in case I am missing anything

Hi @younesbelkada, thanks for your quick response.

`torch.allclose()` returns `False` even with the following parameters: `torch.allclose(o_01["logits"][0], o_02["logits"][0], atol=1e-01, rtol=1e-01)`.

Just looking at the values myself, I get differences such as `o_01["logits"][0][0,2] = -0.2825` and `o_02["logits"][0][0,2] = -0.1675`.

Attention is bi-directional, so I'm not surprised that the logits depend on the hidden states no? 

Hi @ArthurZucker, thanks for the reply.

Is it expected that the output for a given datapoint is influenced by the other datapoints in the same batch?

Maybe a clearer way to illustrate what seems unexpected is given below:

```ruby
from transformers import LlamaForCausalLM, LlamaTokenizer

# load a 7B Llama model in 8bit
model = LlamaForCausalLM.from_pretrained(
    "huggyllama/llama-7b", load_in_8bit=True
)
model.eval()

tokenizer = LlamaTokenizer.from_pretrained("huggyllama/llama-7b")
tokenizer.pad_token_id = 0  

prompt1 = "Hey, are you conscious? Can you talk to me?"
prompt2 = "Give me a recipe for porridge."
prompt3 = "How is the weather today?"

batch_12 = tokenizer.batch_encode_plus([prompt1, prompt2], padding=True, return_tensors="pt")
batch_13 = tokenizer.batch_encode_plus([prompt1, prompt3], padding=True, return_tensors="pt")

# Generate
generate_ids_12 = model.generate(batch_12.input_ids, max_length=50)
generate_ids_13 = model.generate(batch_13.input_ids, max_length=50)

response_1_a = tokenizer.batch_decode(generate_ids_12, skip_special_tokens=True)[0]
response_1_b = tokenizer.batch_decode(generate_ids_13, skip_special_tokens=True)[0]

print(response_1_a)
print(response_1_b)
``` 

When loading with `load_in_8bit=True`, `response_1_a` and `response_1_b` are different. They are the response to the same prompt, it's just the other prompt(s) in the batch that change. This does not happen when we load the model with `load_in_8bit=False`.

1. You are right in this case, same prompt but different *batch* is not related to bi-directional attention
2. Quantized results are not always deterministic so yes would expect some differences
3. You are padding the input, probably on the right which changes what the models sees. An awesome explanation of what is happening is available [here](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535)

Thanks for the pointers.

I did encounter the issues with padding earlier as well. However, I also ran experiments where the prompts were all the exact same length (i.e. no padding), and I still got significant variance in Natural Language Generation metrics for the output when I was shuffling the test set or changing the batch size (both leading to different batch constellations). Are those expected shortcomings when using quantisation?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.