LlavaForConditionalGeneration logit values are batch size dependent.
Author: ShahRutav
Pull request: no

Hi @ArthurZucker,
Thanks for the response. From the [comment](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535), the possible reasons in llama models are -
1. Left-side padding: There is no padding in my script.
2. KV Caching in lower precision operations.  

I tested with float32, my observations are as follows - 
1. No issue (mismatch < 1e-5) if I don't use images in the input and float32 precision. This is equivalent to using only the llama model.
2. The mismatch error is very high, ~11, if I use images in input (and random for different batch sizes), even in float32.  

I am attaching the updated script to reproduce the mismatch,
```python
import torch
import requests
from PIL import Image
import peft
from transformers import BitsAndBytesConfig
from transformers import LlavaForConditionalGeneration, AutoProcessor

test_image = True
dtype = torch.float32
model_id = "llava-hf/llava-1.5-7b-hf"
processor = AutoProcessor.from_pretrained(
    model_id,
    padding_side="left",
    dtype=dtype,
)
if test_image:
    prompt = "<image>\nUSER: What's the content of the image?\nASSISTANT:"
    url = "https://www.ilankelman.org/stopsigns/australia.jpg"
    image = Image.open(requests.get(url, stream=True).raw)
    inputs = processor(text=[prompt], images=[image], return_tensors="pt")
else:
    prompt = "USER: What's the content of the image?\nASSISTANT:"
    inputs = processor.tokenizer(prompt, return_tensors="pt")
inputs_2 = {k: torch.cat([v, v], dim=0).clone() for k, v in inputs.items()}

model = LlavaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=dtype,
).eval()
if dtype != torch.float32: # no need to go through this if dtype is already float32
    peft.prepare_model_for_kbit_training(model)

with torch.no_grad():
    print(inputs['input_ids'])
    model_output = model(**inputs).logits
    model_output_2 = model(**inputs_2).logits

model_output_2 = model_output_2[:model_output.shape[0]]
assert model_output.shape == model_output_2.shape, \
    f"Shapes must be same for comparing. {model_output.shape} vs {model_output_2.shape}"
assert torch.allclose(model_output, model_output_2, atol=1e-5), \
    f"Logits are not the same with maximum difference {torch.max(torch.abs(model_output - model_output_2))}.\n \
    Values:\n{model_output}\nvs\n{model_output_2}"
```

_Originally posted by @ShahRutav in https://github.com/huggingface/transformers/issues/29282#issuecomment-1967487567_
            

Comments:
Sorry but I cannot reproduce your error: 
<img width="993" alt="image" src="https://github.com/huggingface/transformers/assets/48595927/71164caf-3365-4b2e-8541-5dc0119c014b">


The logits are close enough. 
I used this: 
```python 
import torch
import requests
from PIL import Image
import peft
from transformers import BitsAndBytesConfig
from transformers import LlavaForConditionalGeneration, AutoProcessor

test_image = True
dtype = torch.float32
model_id = "llava-hf/llava-1.5-7b-hf"
processor = AutoProcessor.from_pretrained(
    model_id,
    padding_side="left",
    dtype=dtype,
)
if test_image:
    prompt = "USER: <image>\nWhat's the content of the image?\nASSISTANT:"
    url = "https://www.ilankelman.org/stopsigns/australia.jpg"
    image = Image.open(requests.get(url, stream=True).raw)
    inputs = processor(text=[prompt], images=[image], return_tensors="pt")

inputs_2 =  processor(text=[prompt, prompt], images=[image, image], return_tensors="pt")

model = LlavaForConditionalGeneration.from_pretrained(model_id, torch_dtype=dtype,).eval()

with torch.no_grad():
    print(inputs['input_ids'])
    model_output = model(**inputs).logits
    model_output_2 = model(**inputs_2).logits

model_output_2 = model_output_2[:model_output.shape[0]]
assert model_output.shape == model_output_2.shape, "Shapes must be same for comparing. {} vs {}".format(model_output.shape, model_output_2.shape)
assert torch.allclose(model_output, model_output_2, atol=1e-5), "Logits are not the same with maximum difference {}. Values:\n{} vs\n{}".format(torch.max(torch.abs(model_output - model_output_2)), model_output, model_output_2)
```
issue might be how you create your inputs. 

Also the format you are using is wrong

https://github.com/huggingface/transformers/issues/29327#issuecomment-1968086162
Thanks for experimenting with it. 
- I copied exactly the test script you attached, and I got the maximum difference in logits ~  `0.00145` (which is quite larger than yours)
- This difference does not change by creating the inputs differently.

Unless you use a different transformer version, is this likely due to different system specifications?

I am using the latest release of transformers.

If there was a bug, it was fixe 😉 


> [#29327 (comment)](https://github.com/huggingface/transformers/issues/29327#issuecomment-1968086162) Thanks for experimenting with it.
> 
> * I copied exactly the test script you attached, and I got the maximum difference in logits ~  `0.00145` (which is quite larger than yours)
> * This difference does not change by creating the inputs differently.
> 
> Unless you use a different transformer version, is this likely due to different system specifications?

I am also using the latest transformers from the main branch. I cannot explain the difference between your execution (~1e-5) and mine (~1e-3). I'm unsure if this is relevant, but I use CPU since the model won't fit in my GPU without quantization. I am avoiding quantization of the base model to isolate different sources of error.

1e-3 is already close enough. I don't know which CPU you are using but I think we can agree that the outputs are not batch dependent.

sounds good. I am closing the issue. Thanks for the help!