Weird behaivour of LlavaNextForConditionalGeneration during batched generation
Author: heli-qi
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.42.4
- Platform: Linux-5.15.0-94-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.24.2
- Safetensors version: 0.4.3
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 4090 D

### Who can help?

@ArthurZucker  @zucchini-nlp @amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm using model_name `llava-hf/llava-v1.6-vicuna-7b-hf` and model_class `LlavaNextForConditionalGeneration` to generate answers for my own VQA dataset and I met two issues of batched generation.

I'm using the following image as a sample:
![sample_image](https://github.com/user-attachments/assets/dea72dd9-b52b-48d6-8296-06e85627996d)

# 1st: padding_side='left' doesn't work
I notice in [llavanext docs](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/llava_next.md#usage-tips) that `padding_side='left'` should be set when we want to use batched generation, but I find only `padding_side='right'` works for my data. Below is my minimalist codes:
```
import torch
import transformers
from PIL import Image
from peft import PeftModel

from transformers.testing_utils import (
    torch_device,
)

model_id = "llava-hf/llava-v1.6-vicuna-7b-hf"
model = transformers.LlavaNextForConditionalGeneration.from_pretrained(
    model_id, low_cpu_mem_usage=True, load_in_4bit=True
)
processor = transformers.AutoProcessor.from_pretrained(model_id)

prompt = [
    'A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user\'s questions. USER: <image>\nAre there any buildings in this scene? ASSISTANT:',
    'A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user\'s questions. USER: <image>\nWhat is the area of buildings? ASSISTANT:'
]
image = Image.open("sample_image.png")

# # RIGHT
inputs = processor(
    prompt,
    images=[image, image],
    return_tensors="pt",
    padding=True,
).to(torch_device)

output = model.generate(**inputs, max_new_tokens=20)
output = processor.batch_decode(output, skip_special_tokens=True)
print(output)
# A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: \nAre there any buildings in this scene? ASSISTANT: Yes, there are buildings in this scene. The image shows a satellite view of a town or village
# A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: \nWhat is the area of buildings? ASSISTANT:\nThe image you've provided appears to be an aerial or satellite view of a residential

# # LEFT
processor.tokenizer.padding_side = "left"
inputs = processor(
    prompt,
    images=[image, image],
    return_tensors="pt",
    padding=True,
).to(torch_device)

output = model.generate(**inputs, max_new_tokens=20)
output = processor.batch_decode(output, skip_special_tokens=True)
print(output)
# A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: \nAre there any buildings in this scene? ASSISTANT: Yes, there are buildings in this scene. The image shows a satellite view of a town or village
# A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user\'s questions. USER: \,What is the area of buildings? ASSISTANT:
```
The second prompt is shorter than the first one, so the second one will be padded in the batch.
It turns out that if `padding_side` remains its default value `right`, I can have reasonable answers except for there is an extra `\n` at the beginning of the second one. But when `padding_side` is set to `left`, the second answer will be a blank.

I have tried to feed the second prompt to the model alone and it turns out that it works well in the single prompt case:
```
inputs = processor(
    [prompt[1]],
    images=[image],
    return_tensors="pt",
    padding=True,
).to(torch_device)

output = model.generate(**inputs, max_new_tokens=20)
output = processor.batch_decode(output, skip_special_tokens=True)
print(output)
# A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: \nWhat is the area of buildings? ASSISTANT: The image you've provided appears to be an aerial or satellite view of a residential area
```

# 2nd: padding_side='right' doesn't work if pretrained LoRA weights are introduced
I have fine-tuned an LoRA adapter for `llava-hf/llava-v1.6-vicuna-7b-hf` and `LlavaNextForConditionalGeneration` on my own training data. I want to initialized the trained LoRA weights in `LlavaNextForConditionalGeneration` and do batched generation on my test set, but it turns out that `padding_side=right` even failed in this case. Below is my minimalist codes:
```
model = transformers.LlavaNextForConditionalGeneration.from_pretrained(
    model_id, low_cpu_mem_usage=True, device_map="auto", torch_dtype=torch.float16
)
model = PeftModel.from_pretrained(model, "/path/to/my/lora/ckpt")
model = model.merge_and_unload()
model = model.to(torch.float16)
processor = transformers.AutoProcessor.from_pretrained(model_id)

# # RIGHT
inputs = processor(
    prompt,
    images=[image, image],
    return_tensors="pt",
    padding=True,
).to(torch_device)
output = model.generate(**inputs, max_new_tokens=20)
output = processor.batch_decode(output, skip_special_tokens=True)
print(output)
# A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user\'s questions. USER: \nAre there any buildings in this scene? ASSISTANT: Yes
# A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user\'s questions. USER: \nWhat is the area of buildings? ASSISTANT:

# # LEFT
processor.tokenizer.padding_side = "left"
inputs = processor(
    prompt,
    images=[image, image],
    return_tensors="pt",
    padding=True,
).to(torch_device)
output = model.generate(**inputs, max_new_tokens=20)
output = processor.batch_decode(output, skip_special_tokens=True)
print(output)
# A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user\'s questions. USER: \nAre there any buildings in this scene? ASSISTANT: Yes
# A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user\'s questions. USER: \nWhat is the area of buildings? ASSISTANT:

# # SINGLE
inputs = processor(
    [prompt[1]],
    images=[image],
    return_tensors="pt",
    padding=True,
).to(torch_device)
output = model.generate(**inputs, max_new_tokens=20)
output = processor.batch_decode(output, skip_special_tokens=True)
print(output)
# A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user\'s questions. USER: \nWhat is the area of buildings? ASSISTANT: 10%-20%
```


### Expected behavior

I wish to enable batched generation of my tuned model on my test set to enjoy the multi-GPU speed up. Currently, I could only do generation on a single GPU with batch_size=1 which is quite slow.

Comments:
Taking a look 👀 

@heli-qi seems like the error is present only when BnB-int and Eager attention is used. 

If we change the attn to "sdpa" (for that you need to upgrade torch), the generated text is a lot better. To dig deeper, I verified that the attention masks are constructed correctly, and all inputs are same in sdpa vs eager settings. Also I can assure that sdpa and eager generate same text in fp16 precision.

It means this is not a bug in how llava is implemented but rather due to the numerical precision errors in this particular setting with this particular image & prompt. For ex, using another image doesn't generate empty for left-padding



Thanks a lot for your response! Switching "eager" attn to either "sdpa" or "flash_attention_2" fixed the blank generation issue with `padding_side='left'`.

It seems like "eager" attn is the least stable attention implementation. Should I prioritize "sdpa" or "flash_attention_2" for LlaVA inference?

SDPA is usually the default attention used in all models, if you have torch version higher than 2.1.0. In your case the torch version was lower and eager dispatched as default. 

SDPA is default because it can leverage several optimization tricks to make your generation efficient