LlavaForConditionalGeneration logit values are batch size dependent.
Author: ShahRutav
Pull request: no

### System Info

- `transformers` version: 4.39.0.dev0
- Platform: Linux-5.4.0-163-generic-x86_64-with-glibc2.31
- Python version: 3.9.18
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

 @ArthurZucker  @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
import torch
import requests
from PIL import Image
import peft
from transformers import BitsAndBytesConfig
from transformers import LlavaForConditionalGeneration, AutoProcessor

dtype = torch.bfloat16
model_id = "llava-hf/llava-1.5-7b-hf"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=dtype
)
processor = AutoProcessor.from_pretrained(
    model_id,
    padding_side="left",
    dtype=dtype,
)
prompt = "<image>\nUSER: What's the content of the image?\nASSISTANT:"
url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(text=[prompt], images=[image], return_tensors="pt")
inputs_2 = {k: torch.cat([v, v], dim=0).clone() for k, v in inputs.items()}

model = LlavaForConditionalGeneration.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    torch_dtype=dtype,
).eval()
peft.prepare_model_for_kbit_training(model)

with torch.autocast(device_type="cuda", dtype=dtype):
    with torch.no_grad():
        model_output = model(**inputs).logits
        model_output_2 = model(**inputs_2).logits

model_output_2 = model_output_2[:model_output.shape[0]]
assert model_output.shape == model_output_2.shape, \
    f"Shapes must be same for comparing. {model_output.shape} vs {model_output_2.shape}"
assert torch.allclose(model_output, model_output_2, atol=1e-5), \
    f"Logits are not the same with maximum difference {torch.max(torch.abs(model_output - model_output_2))}.\n \
    Values:\n{model_output}\nvs\n{model_output_2}"
```
The code snippet creates an input for the `LlavaForConditionalGeneration` model. Replicates the same input along the batch dimension.

### Expected behavior

The logits returned in both cases (batch_size = 1 and batch_size = 2) should be the same. However, the difference is larger than expected (maximum difference is ~0.3).


Comments:
Padding tokens are never ignored, they thus affect the model. See [this comment](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535). 
I'll close this as it's pretty much a duplicate! Feel free to re-open if you are not satisfied with this answer / if float32 has the issue



Hi @ArthurZucker,
Thanks for the response. From the [comment](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535), the possible reasons in llama models are -
1. Left-side padding: There is no padding in my script.
2. KV Caching in lower precision operations.  

I tested with float32, my observations are as follows - 
1. No issue (mismatch < 1e-5) if I don't use images in the input and float32 precision. This is equivalent to using only the llama model.
2. The mismatch error is very high, ~11, if I use images in input (and random for different batch sizes), even in float32.  

I am attaching the updated script to reproduce the mismatch,
```python
import torch
import requests
from PIL import Image
import peft
from transformers import BitsAndBytesConfig
from transformers import LlavaForConditionalGeneration, AutoProcessor

test_image = True
dtype = torch.float32
model_id = "llava-hf/llava-1.5-7b-hf"
processor = AutoProcessor.from_pretrained(
    model_id,
    padding_side="left",
    dtype=dtype,
)
if test_image:
    prompt = "<image>\nUSER: What's the content of the image?\nASSISTANT:"
    url = "https://www.ilankelman.org/stopsigns/australia.jpg"
    image = Image.open(requests.get(url, stream=True).raw)
    inputs = processor(text=[prompt], images=[image], return_tensors="pt")
else:
    prompt = "USER: What's the content of the image?\nASSISTANT:"
    inputs = processor.tokenizer(prompt, return_tensors="pt")
inputs_2 = {k: torch.cat([v, v], dim=0).clone() for k, v in inputs.items()}

model = LlavaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=dtype,
).eval()
if dtype != torch.float32: # no need to go through this if dtype is already float32
    peft.prepare_model_for_kbit_training(model)

with torch.no_grad():
    print(inputs['input_ids'])
    model_output = model(**inputs).logits
    model_output_2 = model(**inputs_2).logits

model_output_2 = model_output_2[:model_output.shape[0]]
assert model_output.shape == model_output_2.shape, \
    f"Shapes must be same for comparing. {model_output.shape} vs {model_output_2.shape}"
assert torch.allclose(model_output, model_output_2, atol=1e-5), \
    f"Logits are not the same with maximum difference {torch.max(torch.abs(model_output - model_output_2))}.\n \
    Values:\n{model_output}\nvs\n{model_output_2}"
```

https://github.com/huggingface/transformers/issues/29282#issuecomment-1965910448
I cannot re-open the issue since a collaborator has closed it.