Better llava next.
Author: nxphi47
Pull request: yes

- Batched forward with multiple image of different sizes (number of patches).
- Support training, for cases with and without image.
- Support multi-image in same sequence. e.g: `["<image> <image> the first image is a dog while the second is a cat", "<image> <image> <image> <image> these 4 image are..."]`
- Support batched generation (require padding_side = "left")

```python

device = "cuda:0"
model_path = "llava-hf/llava-v1.6-mistral-7b-hf"
processor = LlavaNextProcessor.from_pretrained(model_path)
processor.tokenizer.padding_side = "left"

model = LlavaNextForConditionalGeneration.from_pretrained(
    model_path, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="cuda",
)


# ! Differnt images, same prompt

cat_img = Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)
chart_img = Image.open(requests.get("https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true", stream=True).raw)

prompts = [
    "
 <image>\nWhat is shown in this image? 
",
    "
 <image>\nWhat is shown in this image? 
",
]
inputs = processor(prompts, [chart_img, cat_img], return_tensors='pt', padding=True).to("cuda")
processor.tokenizer.padding_side = "left"

# just in case, pass padding_side = "left" to generate
output = model.generate(**inputs, max_new_tokens=1024, do_sample=False, pad_token_id=processor.tokenizer.pad_token_id, padding_side=processor.tokenizer.padding_side)

for o in output:
    print(processor.decode(o, skip_special_tokens=True))

# expected output
"""

  
What is shown in this image? 
 The image appears to be a radar chart, which is a type of multivariate chart that displays values for multiple variables represented on axes starting from the same point. This particular radar chart is showing the performance of different models or systems across various metrics.

The axes represent different metrics or benchmarks, such as MM-Vet, MM-Vet-GQA, MM-Vet-GQA-VizWiz, LLaVa-Bench, SLED-Bench, and several others. Each axis is labeled with the name of the metric and a numerical value, which likely represents a score or a performance measure.

The colored areas within the chart represent different models or systems, such as MME, BLIP-2, InstructionBLIP, and others. The size of the area on each axis indicates the performance of the model or system on that particular metric.

The chart is color-coded to differentiate between the different models or systems, and it provides a visual comparison of their performance across the various metrics. This kind of chart is often used in machine learning and artificial intelligence to compare the performance of different models or algorithms. 

  
What is shown in this image? 
 The image shows two cats lying on a pink blanket. The cat on the left is curled up in a relaxed position, while the cat on the right is stretched out with its head resting on the blanket. There is a remote control next to the cat on the left, suggesting that this scene might be taking place in a living room or a similar space where people might watch television. The cats appear to be sleeping or resting. 
"""
```

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Thanks for your PR, feel free to add the changes to `modeling_llava_next.py` so that we can see an easier diff

Just tried out your implementation from a [new branch](https://github.com/huggingface/transformers/compare/main...NielsRogge:transformers:llava_batched_inference_bis?expand=1) I made.

It gives me the following outputs (by running `python src/transformers/models/llava_next/test.py`) :
"""
['
 
  \nWhat is shown in this image? 
 
 The image appears to be a radar chart, which is a type of multi-dimensional plot that displays data in the form of a two-dimensional chart of three or more quantitative variables represented on axes starting from the same point.\n\nIn this particular radar chart, there are several axes labeled with different metrics or benchmarks, such as "MM-Vet," "MM-Bench," "LLa-Va-Bench," "LLa-Va-B', '
 
  \nHow many cats are there? 
 
 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 cats are two cats are there are there are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats are two cats']
"""

Whereas the original implementation gives me the following on the same inputs (by running `CUDA_VISIBLE_DEVICES=0 python llava/eval/run_llava_batched_inference.py --model-path "liuhaotian/llava-v1.6-mistral-7b" --image-file "images/llava_v1_5_radar.jpg" --query "What is shown in this image?"` from [this branch](https://github.com/haotian-liu/LLaVA/compare/main...NielsRogge:LLaVA:understand_llava_batched_inference?expand=1)):

"""
['The image appears to be a radar chart, which is a type of multi-dimensional plot that displays data in the form of a two-dimensional chart of three or more quantitative variables represented on axes starting from the same point.\n\nIn this particular radar chart, there are several axes labeled with different metrics or benchmarks, such as "MM-Vet," "MM-Bench," "LLa-Va-Bench," "LLa-Va-B', 'There are two cats in the image.']
"""
Note that the inputs use right padding, I pushed them here: https://huggingface.co/datasets/nielsr/llava-batched-inference/tree/main

So it looks like the first one in the batch is correct, whereas the second one (regarding the cats image) goes off the rails.

I've also noticed that the padding token ID is currently set wrongly for the tokenizer of https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf, it should be \<unk> instead of \<pad>

@NielsRogge Thanks. Let me check it out. I thought batched generation require left-padding, unless the 2 of the sample are exactly same # of tokens, because otherwise pad tokens will be in the middle of the generation?


@NielsRogge I thought the pad token id being wrong was fixed as mentioned by you in https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf/discussions/2

@NielsRogge I have added batched generation with left padding in the latest commit. Try it here:

```python
import torch
from huggingface_hub import hf_hub_download
import requests
from PIL import Image

from transformers import LlavaNextForConditionalGeneration, LlavaNextProcessor
from transformers.models.llava_next.modeling_better_llava_next import BetterLlavaNextForConditionalGeneration
processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
processor.tokenizer.padding_side = "left"

model = BetterLlavaNextForConditionalGeneration.from_pretrained(
    "llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="cuda",
)

# ! Chart and cat
cat_img = Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)
chart_img = Image.open(requests.get("https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true", stream=True).raw)


prompts = [
    "
 <image>\nWhat is shown in this image? 
",
    "
 <image>\nWhat is shown in this image? 
"
]
inputs = processor(prompts, [chart_img, cat_img], return_tensors='pt', padding=True).to("cuda")

output = model.generate(**inputs, max_new_tokens=20, do_sample=False, pad_token_id=processor.tokenizer.pad_token_id)

for o in output:
    print(processor.decode(o, skip_special_tokens=True))

"""output

  
What is shown in this image? 
 The image appears to be a radar chart, which is a type of multivariate chart that

  
What is shown in this image? 
 2 cats lying on a pink blanket. The cat on the left is stretching and has its
"""
```

However, I still find that batch=1 and batch=2 still has inconsistency issue. So need to debug further

We got repetition degeneration when one item in the batch has finish generating.
```
# batch=2

  
What is shown in this image? 
 2 cats lying on a pink blanket. The cat on the left is stretching and has its paws extended, while the cat on the right is curled up and appears to be sleeping. There is a remote control next to the cats, and a red couch is visible in the background. 

  
What is shown in this image? 
 The image appears to be a radar chart, which is a type of multivariate chart that displays values for multiple variables represented on axes starting from the same point. This particular radar chart is showing the performance of different models or systems across various metrics.

The axes represent different metrics or benchmarks, such as MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-Vet, MM-V
```

batch=1
```

  
What is shown in this image? 
 The image appears to be a radar chart, which is a type of multivariate chart that displays values for multiple variables represented on axes starting from the same point. This particular radar chart is showing the performance of different models or systems across various metrics.

The axes represent different metrics or benchmarks, such as MM-Vet, MM-Vet-GQA, MM-Vet-GQA-VizWiz, LLaVa-Bench, SLED-Bench, and several others. Each axis is labeled with the name of the metric and a numerical value, which likely represents a score or a performance measure.

The colored areas within the chart represent different models or systems, such as MME, BLIP-2, InstructionBLIP, and others. The size of the area and the position of the models on the axes indicate their performance on each metric.

The chart is color-coded to differentiate between the different models or systems, and it provides a visual comparison of their performance across the various benchmarks. This kind of chart is often used in machine learning and artificial intelligence to compare the performance of different models or algorithms. 
```
So suspect it has something to do with this part https://github.com/huggingface/transformers/blob/b98581b09521b47fc597d2077819895ab4c059dd/src/transformers/models/llava_next/modeling_better_llava_next.py#L950

Could you try without cached keys/values (by setting `model.config.use_cache=False`)?

I'm not able to reproduce your code snippet, I get:
"""
['
  \nWhat is shown in this image? 
 The image appears to be a radar chart, which is a type of multivariate chart that displays values for multiple variables represented on axes starting from the same point. This particular radar chart is showing the performance of different models or systems across various', '
  \nWhat is shown in this image? 
 ']
"""
My branch is [here](https://github.com/huggingface/transformers/compare/main...NielsRogge:llava_batched_inference_bis?expand=1). I'm running the script `python src/transformers/models/llava_next/test_bis.py`.

@NielsRogge Ok, I also got your response when using this PR alone. And I found that this PR and my internal code (which I port the PR from) is same regarding llava_next codes., but different at `modeling_mistral.py`

This is SdpaAttention. Left is my code, right is most updated transformer code.

<img width="2137" alt="Screenshot 2024-03-26 at 4 52 47 PM" src="https://github.com/huggingface/transformers/assets/19323568/7a51960b-cc3b-41f5-b40b-ca4f49bb7911">

My internal code is older and I guess adopted by previous commits. Do you notice if the problem can arise from this mistral difference?


I got the same results either with or without `model.config.use_cache=False `

@aliencaocao I discussed this offline with @ArthurZucker, the padding token was [added](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava/convert_llava_weights_to_hf.py#L80) for llava 1.5 as at the time the original repository didn't support batched generation, and a padding token didn't seem to be used. Having a clear separation between an unk and padding token was the reason why it was added. Hence the same was done for llava 1.6.

However I'm now able to perform batched inference, and it uses \<unk> as padding token with right padding, which is why I think we may need to update the tokenizer.

Edit: I don't think right padding makes sense for batched generation, one should use left padding. I assume it's fine to use \<pad> instead of \<unk> for batched generation (we do the same for llava 1.5)

@nxphi47 latest transformers' mistral implementation should be exact, is this a `sliding_window` issue? 

@NielsRogge @ArthurZucker I figure it out and has push the updated code. Please give a test!
Basically what went wrong:
- batched generation always require `padding_side = "left"`
- the line `attention_mask = torch.cat((attention_mask, extended_attention_mask), dim=1)` should be `attention_mask = torch.cat((extended_attention_mask, attention_mask), dim=1)` for padding_side = left. Otherwise you will get mask: `[1,1,1,0,0,0,0,1,1,1,1]`
- in `_merge_input_ids_with_image_features`, sometimes you cannot tell from the inputs whether this is left or right padding. 
   - e.g1: (same image, different prompt) `[f"
 <image> What are the things I should be cautious about when I visit here? 
", f"
 <image> Describe what you see. 
",]`. You can tell padding_side from `input_ids`.
   - eg2: (different image (different visual token), same prompt) `["
 <image>\nWhat is shown in this image? 
", "
 <image>\nWhat is shown in this image? 
",]` . Here, input `attention_mask` is all `1`, but `final_attention_mask` has padding because image1 and image2 has different visual tokens. In this case, have to pass another arg `padding_side` to tell it we are doing left-padding, so that it will put the padding correctly.

Check out the test script: [`tests/models/llava_next/test_llava_next_batched_gen.py`](https://github.com/huggingface/transformers/blob/9b84e7dc3d2118c7f9f865ee63eb2300f897012c/tests/models/llava_next/test_llava_next_batched_gen.py)

Example run:
```python
device = "cuda:0"
model_path = "llava-hf/llava-v1.6-mistral-7b-hf"
processor = LlavaNextProcessor.from_pretrained(model_path)
processor.tokenizer.padding_side = "left"

model = LlavaNextForConditionalGeneration.from_pretrained(
    model_path, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="cuda",
)


# ! Differnt images, same prompt

cat_img = Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)
chart_img = Image.open(requests.get("https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true", stream=True).raw)

prompts = [
    "
 <image>\nWhat is shown in this image? 
",
    "
 <image>\nWhat is shown in this image? 
",
]
inputs = processor(prompts, [chart_img, cat_img], return_tensors='pt', padding=True).to("cuda")
processor.tokenizer.padding_side = "left"
output = model.generate(**inputs, max_new_tokens=1024, do_sample=False, pad_token_id=processor.tokenizer.pad_token_id, padding_side=processor.tokenizer.padding_side)

for o in output:
    print(processor.decode(o, skip_special_tokens=True))

# expected output
"""

  
What is shown in this image? 
 The image appears to be a radar chart, which is a type of multivariate chart that displays values for multiple variables represented on axes starting from the same point. This particular radar chart is showing the performance of different models or systems across various metrics.

The axes represent different metrics or benchmarks, such as MM-Vet, MM-Vet-GQA, MM-Vet-GQA-VizWiz, LLaVa-Bench, SLED-Bench, and several others. Each axis is labeled with the name of the metric and a numerical value, which likely represents a score or a performance measure.

The colored areas within the chart represent different models or systems, such as MME, BLIP-2, InstructionBLIP, and others. The size of the area on each axis indicates the performance of the model or system on that particular metric.

The chart is color-coded to differentiate between the different models or systems, and it provides a visual comparison of their performance across the various metrics. This kind of chart is often used in machine learning and artificial intelligence to compare the performance of different models or algorithms. 

  
What is shown in this image? 
 The image shows two cats lying on a pink blanket. The cat on the left is curled up in a relaxed position, while the cat on the right is stretched out with its head resting on the blanket. There is a remote control next to the cat on the left, suggesting that this scene might be taking place in a living room or a similar space where people might watch television. The cats appear to be sleeping or resting. 
"""

```

I have also remove the BetterLlavaNext class and integrate it into LlavaNext

So this would close https://github.com/huggingface/transformers/issues/29832 right?
Any idea on if this change could solve https://github.com/huggingface/transformers/issues/29835?

Awesome @nxphi47 !! Will take a look tomorrow. Yes @aliencaocao this issue could fix batched generation. The other issue #29835 seems to happen with both llava and llava-next, so that's a different one

Thanks, I'm able to reproduce it so it works fine! Great work. Final todo's:

- [ ] address comments I made above
- [ ] make sure everything is backwards compatible (by running `RUN_SLOW=yes pytest tests/models/llava_next`)
- [ ] make sure CI is green (can be done by running `make fixup` locally and address reported issues)

When that's done, I'll assign @ArthurZucker for final review

@NielsRogge There you go, kindly review the updated code.

The current implementation of the processor no longer support multi-image in same sequence. I would suggest we support that by add another `num_images` dimension to pixel_values `(batch_size, num_images, num_patches, 3, 336, 336)` and image_sizes `(batch_size, num_images, 2)` with zero padding.

@NielsRogge is this pad function what you expected?
https://github.com/huggingface/transformers/blob/685892cb6c13e0d17043d58f8997011b6fd75823/src/transformers/models/llava_next/image_processing_llava_next.py#L449

It's in the right direction, but any image processing method should always take a numpy array as input and produce a numpy array. In case of padding, one can leverage the pad method available in the image_transforms module. An example is [this method](https://github.com/huggingface/transformers/blob/6cdbd73e01a9719bfaec07d91fd108e8d932bbbb/src/transformers/models/donut/image_processing_donut.py#L182-L222).

Also wondering @amyeroberts why `pad` was [deprecated](https://github.com/huggingface/transformers/blob/6cdbd73e01a9719bfaec07d91fd108e8d932bbbb/src/transformers/models/donut/image_processing_donut.py#L224) for this model in favor of `pad_image`

> It's in the right direction, but any image processing method should always take a numpy array as input and produce a numpy array. In case of padding, one can leverage the pad method available in the image_transforms module. An example is [this method](https://github.com/huggingface/transformers/blob/6cdbd73e01a9719bfaec07d91fd108e8d932bbbb/src/transformers/models/donut/image_processing_donut.py#L182-L222).
> 
> Also wondering @amyeroberts why `pad` was [deprecated](https://github.com/huggingface/transformers/blob/6cdbd73e01a9719bfaec07d91fd108e8d932bbbb/src/transformers/models/donut/image_processing_donut.py#L224) for this model in favor of `pad_image`

@NielsRogge Can you be more specific by providing a concrete example for llava_next ? How pad should be implemented, given that it should receive a list of np array for multiple images to be padded into a single np.array ? I couldn't draw a correlation given donut or DETR examples.

Ok I see, in that case we can just call the method `_pad_pixel_values` I assume. I'll let @amyeroberts confirm.

Discussed this offline with @zucchini-nlp, we'd like to have batched generation of llava-next to be available by the next Transformers release (given that it's the current best open-source VLM along with Idefics-2), @nxphi47 do you think you're able to address the remaining comments + having a green CI (by running `make fixup`)? Else we're happy to help and get this merged quickly

> Also wondering @amyeroberts why pad was [deprecated](https://github.com/huggingface/transformers/blob/6cdbd73e01a9719bfaec07d91fd108e8d932bbbb/src/transformers/models/donut/image_processing_donut.py#L224) for this model in favor of pad_image

@NielsRogge `pad` was deprecated across the feature extractors because the API was inconsistent: some pad implementations would pad a single image, some a batch of images; as well as input arguments. There were also inconsistent return types: sometimes an image, sometimes a batch of images; sometimes a batchfeature. All standard the image transforms should follow the pattern of taking and returning a single numpy array image. 

To make this more consistent, `pad_image` is used for a single image. `pad` is implemented for some image processors in the case when it takes a batch of images, which separately calls the `pad_image` method. 

@NielsRogge @zucchini-nlp Resolved the issues. Should be ready to merge.

Thanks, made some final comments, after that I'll assign Amy for approval.

> Thanks, made some final comments, after that I'll assign Amy for approval.

Added the "remove comments" update

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29850). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey, guys! @nxphi47 @NielsRogge 
I just pulled this branch and facing problems with generation with repetition_penalty parameter. Im using vicuna version of LLaVA

Generation without  penalty works as it should


```python
from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
from PIL import Image
import requests
import torch

model_path = "llava-hf/llava-v1.6-vicuna-7b-hf"
processor = LlavaNextProcessor.from_pretrained(model_path)
processor.tokenizer.padding_side = "left"

model = LlavaNextForConditionalGeneration.from_pretrained(
    model_path, torch_dtype=torch.float16, low_cpu_mem_usage=True, device_map="cuda",
)

cat_img = Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)
chart_img = Image.open(requests.get("https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true", stream=True).raw)

prompts = [
    'A chat between a curious human and an artificial intelligence assistant. '
    'The assistant gives helpful, detailed, and polite answers to the human\'s questions. '
    'USER: <image>\nWhat is shown in this image? ASSISTANT:'
    for _ in range(2)
]
inputs = processor(prompts, [cat_img, chart_img], return_tensors='pt', padding=True).to(model.device)
output = model.generate(**inputs, max_new_tokens=10, do_sample=False, pad_token_id=processor.tokenizer.pad_token_id)
for o in output:
    print(processor.decode(o, skip_special_tokens=False), '\n')

# output
"""

 A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image> 
What is shown in this image? ASSISTANT:
The image shows two cats lying on a 


 A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image> 
What is shown in this image? ASSISTANT: The image appears to be a graphical representation of 
"""
```

Now let's generate the same with repetition_penalty. There will be unknown tokens - zeros in output

```python
output = model.generate(**inputs, max_new_tokens=10, do_sample=False, pad_token_id=processor.tokenizer.pad_token_id,
                       repetition_penalty=1.5)
for o in output:
    print(processor.decode(o, skip_special_tokens=False), '\n')
# output
"""

 A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image> 
What is shown in this image? ASSISTANT:
<unk><unk><unk><unk><unk><unk><unk><unk><unk> 


 A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image> 
What is shown in this image? ASSISTANT: This appears to be a graphical representation of some 
"""
```

In that time if we will pass first image twice, with penalty, all will be good

```python
inputs = processor(prompts, [cat_img, cat_img], return_tensors='pt', padding=True).to(model.device)
output = model.generate(**inputs, max_new_tokens=10, do_sample=False, pad_token_id=processor.tokenizer.pad_token_id,
                       repetition_penalty=1.5)
for o in output:
    print(processor.decode(o, skip_special_tokens=False), '\n')
# output
"""

 A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image> 
What is shown in this image? ASSISTANT: This photo shows two cats lying on their sides 


 A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image> 
What is shown in this image? ASSISTANT: This photo shows two cats lying on their sides 
"""
```

@messlav Can you try turning this line back to `new_token_positions += new_token_positions[:, -1].max() - new_token_positions[:, -1:]` ?

https://github.com/huggingface/transformers/blob/a0102a425dc8d01fddf215444aa2e54dfd8b7eb2/src/transformers/models/llava_next/modeling_llava_next.py#L565

@nxphi47 Didn't help, same behaviour :(

@messlav I could not reproduce it. Can you share stack trace? If it is failing only with `repetition_penalty`, can be related to indexing errors

Hi! Thanks for the great work. I just have a question.

> Support training, for cases without any image.

Does this mean
1. Existing code doesn't support training at all, now it can train but only without any image
2. Existing code supports training but requires both images and texts, now it also supports training only with texts

2

> 2

Thank you for the answer!

@nxphi47 can you rebase main and resolve conflicts, before we ask core maintainers review. Also I see some unresolved comments, will be addressed?

@zucchini-nlp Resolved the comments. But I wont have time for another round of rebase and conflict resolution. Can you help with that?

@nxphi47 thanks a lot! Okay will rebase main and I believe we'll be ready for re-review 

@amyeroberts ready for review, all comments addressed

Thanks for building this! Excited to take advantage of it. 

> It might not be necessary to add anything upstream. If we want padding to happen by default (which I'm assuming we do) then we can just set do_pad: int = True in the init. In this case, even if we're loading in old config files, this will still be the default behaviour.

Okay, then we don't need to get the `do_pad` attribute from the config I guess. We will have a default `True` and overwrite it by user-defined kwargs.

 

@zucchini-nlp Yes, although one thing to note is any image processor save will have `do_pad` in its config

Ready to merge. All slow-tests are passing on my end. @amyeroberts merging it later today to ship it for the next release, lmk if anything needs to be corrected before :)

Thanks so much @nxphi47 for this heroic effort and thanks @zucchini-nlp for finishing it!

Wow it finally got approved . Thanks @zucchini-nlp for finishing it!

@NielsRogge and @zucchini-nlp I am unable to perform VSFT with TRL and the LLaVANext models. Can you take a look at the following issue: https://github.com/huggingface/trl/issues/1785

> Yes, now the user has to set the `padding_side` explicitly if the batch has all texts of equal length. Okay for me to bring back the hard-coded `self._padding_side = "left"` as it was in the initial state of the PR.
> 
> If user wants to train the model, the padding_side will have to be set to "right" in model's config manually

trying to ft llava-1.6 and kept having issues with anything but batch=1 as long as model is set to model.train().

certainly predominantly on me for not setting the padding side explicitly, but since there is so little llava-next transformers training code out there - i didn't even consider setting it until i uncovered this issue in the depths of the [hf discussions](https://huggingface.co/llava-hf/llava-v1.6-34b-hf/discussions/5). i feel that the error produced during training is quite non-intuitive and not descriptive of the issue as it talks about images not being of expected number. (my set-up is a little weird thought with fabric wrapping a hf model but idt that matters). if we know what will break during training for someone who doesn't set it explicitly - potentially worth throwing in as a comment into a exception?

my traceback in the simplest possible case:

```
^MLoading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]^MLoading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:05,  1.74s/it]^MLoading checkpoint shards:  50%|█████     | 2/4 [00:03<00:02,  1.50s/it]^MLoading checkpoint shards:  75%|███████▌  | 3/4 [00:04<00:01,  1.48s/it]^MLoading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.02it/s]^MLoading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.18s/it]
Using bfloat16 Automatic Mixed Precision (AMP)
tensor([[572, 503],
        [255, 502]], device='cuda:0')
[torch.Size([5, 3, 336, 336]), torch.Size([5, 3, 336, 336])]
^MTraining:   0%|          | 0/25430 [00:00<?, ?it/s]^M                                                   ^MTraceback (most recent call last):
  File "/gpfs/data/oermannlab/users/alyaka01/CNS-LLaVA/cns_llava/llava-next-med/llava-next-med.py", line 173, in <module>
    outputs = model(**batch)
              ^^^^^^^^^^^^^^
  File "/gpfs/data/oermannlab/users/alyaka01/.conda/envs/cns-llava-hf-12-scratch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/data/oermannlab/users/alyaka01/.conda/envs/cns-llava-hf-12-scratch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/data/oermannlab/users/alyaka01/.conda/envs/cns-llava-hf-12-scratch/lib/python3.12/site-packages/lightning/fabric/wrappers.py", line 141, in forward
    output = self._forward_module(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/data/oermannlab/users/alyaka01/.conda/envs/cns-llava-hf-12-scratch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/data/oermannlab/users/alyaka01/.conda/envs/cns-llava-hf-12-scratch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/data/oermannlab/users/alyaka01/.conda/envs/cns-llava-hf-12-scratch/lib/python3.12/site-packages/transformers/models/llava_next/modeling_llava_next.py", line 806, in forward
    inputs_embeds, attention_mask, position_ids, labels, _ = self._merge_input_ids_with_image_features(
                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/gpfs/data/oermannlab/users/alyaka01/.conda/envs/cns-llava-hf-12-scratch/lib/python3.12/site-packages/transformers/models/llava_next/modeling_llava_next.py", line 626, in _merge_input_ids_with_image_features
    raise ValueError(
ValueError: image_to_overwrite.sum()=tensor(3607, device='cuda:0') != num_image_features=3516 The input provided to the model are wrong. The number of image tokens is 3514 while the number of image given to the model is 2. This prevents correct indexing and breaks batch generation.
srun: error: sp-0001: task 0: Exited with exit code 1
```


@alyakin314 Thanks for reporting. We'll raise a warning in case the padding side is different from expected when https://github.com/huggingface/transformers/pull/32386 is merged! Agreed that llava-next is a bit harder than other llava models, we're working on simplifying the logic 

https://github.com/huggingface/transformers/blob/174890280b340b89c5bfa092f6b4fb0e2dc2d7fc/src/transformers/models/llava_next/modeling_llava_next.py#L758-L762
Hi everyone, from my perspective, this is a great feature for not requiring the embedding for extra image token. Could you please share the reasons behind its removal in later versions? And is there any workaround for not having the corresponding embedding for image token?

@lcxrocks let's move the discussion on that to https://github.com/huggingface/transformers/issues/35683, I will reply under the issue soon