Add llama3-llava-next-8b to llava_next conversion script
Author: jamt9000
Pull request: yes

# What does this PR do?

Adds support for the [lmms-lab/llama3-llava-next-8b](https://huggingface.co/lmms-lab/llama3-llava-next-8b) model to the convert_llava_next_weights_to_hf.py script, along with an example prompt generated from the [llava_llama_3 conv_template](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/6944062b9bb2e61c48436f1a65c3ea339095ec91/llava/conversation.py#L357) in the LLaVA-NeXT repo.

I've confirmed the model seems to convert without errors and the output seems similar (but not identical) to [the inference example from the LLaVA-NeXT repo](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/inference/docs/LLaVA-NeXT.md#quick-start-with-huggingface), however I'm nor sure if the logic around the added tokens is correct.

The batched generation also seems to be off:
```
Batched generation...
Setting `pad_token_id` to `eos_token_id`:128009 for open-end generation.
['system\n\nYou are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.user\n\n\nWhat is shown in this image?assistant\n\n\nThe image appears to be a radar chart, also known as a spider chart or a web chart', '
 \nHow many cats are there? 
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n']
```

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/issues/31394


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@NielsRogge @zucchini-nlp


Comments:
Yes would be great to have all checkpoints converted :)

Please also make sure to verify the outputs of the model against the original implementation, also on a logits level (ideally with `torch.assertallclose(original_logits[0,:3,:3], huggingface_logits[0,:3,:3])` on a dummy input). Getting sensible output is usually not enough

Currently doing a bunch of debugging to try to get near-identical logits. Findings so far are:

I'd added `
` in the prompt example, but it looks like this gets added automatically in the HF processor, so the input_ids had two begin of text ids. I've also checked the 5-crop image preprocessing is completely identical.

After fixing the begin of text the logits are:

```py
# LLaVA-NeXT repo
tensor([[ -3.9648,   1.1396,   3.3145],
        [ -5.3398,  -1.5537,  -1.9512],
        [-12.3828, -10.6797,  -9.3047]], device='cuda:0',
       grad_fn=<SliceBackward0>)

# HF LlavaNextForConditionalGeneration
tensor([[ -3.9648,   1.1396,   3.3145],
        [ -5.3594,  -1.5654,  -1.9619],
        [-12.3750, -10.6797,  -9.3125]], device='cuda:2', dtype=torch.float32)
```

Which is close but not within the `atol=1e-4` in the script. However the generated text is now identical (and matches what is in the [quickstart example](https://github.com/LLaVA-VL/LLaVA-NeXT/blob/inference/docs/LLaVA-NeXT.md#quick-start-with-huggingface)):

`The image shows a radar chart, also known as a spider chart or a web chart, which is a type of graph used to display multivariate data in the form of a two-dimensional chart of three or more quantitative variables represented on axes starting from the same point. Each axis represents a different variable, and the values are plotted along each axis and connected to form a polygon.\n\nIn this particular radar chart, there are several axes labeled with different variables, such as "MM-Vet," "LL`
 

Also making a note that LLama3 and Qwen actually have extra unused space in the embeddings, so we don't need to resize.

For LLama3 they are included in the tokenizer as `
` etc (you can see them all in https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/blob/main/tokenizer_config.json) and from the discussion here https://github.com/meta-llama/llama3/issues/77 it looks like they are available for use (although renaming a reserved token to `<image>` might involve some hacky stuff: https://github.com/huggingface/transformers/issues/27974#issuecomment-1854188941)

For Qwen, the unused space isn't already allocated in the tokenizer, but `len(tokenizer)==151646` whereas `config.text_config.vocab_size==152064` due to padding to a multiple of 128 https://github.com/QwenLM/Qwen/issues/553

hey, i wonder why this PR is in pending.

Sorry, totally forgot about this PR. I think the only thing left now is to get batched generation fixed, if not done yet, and create section in docs for new models, as outlined above

@jamt9000 let me know if you're stuck and need help or ping me if it's ready for review :) 

I've added the prompt examples to the docs and checked that with the correct prompt and `padding_side = "left"` the batched generation is working (but with `padding_side = "right"` it still doesn't generate any answer to the cats question).

```
[
'system\n\nYou are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.user\n\n\nWhat is shown in this image?assistant\n\n\nThe image shows a radar chart, also known as a spider chart or a web chart, which',
'system\n\nYou are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.user\n\n\nHow many cats are there?assistant\n\n\nThere are two cats in the image.'
]
```

What I'm still stuck on is:

* I'm unsure if the llama3 logits need to be within a tighter tolerance to the reference
* Should the batched generation be working with padding=right? Why does it say "Setting `pad_token_id` to `eos_token_id`:128009 for open-end generation." when there's a padding token added?
* Whether it's necessary to resize the vocabulary (or if it could use a reserved token intead https://github.com/huggingface/transformers/pull/31395#issuecomment-2171694828)

@jamt9000 again sorry for delayed reply, was off on a vacation. Regarding the questions:

- Yes, `padding_side='left'` for batched inference is exactly what we need, as right padding is used only in training
- if we can make use of reserved tokens, then it's not necessary to resize vocab embeddings. Just make sure that tokenizer and model both have correct image token ids in their config. Simple test comparing with original implementation should be okay 
- Hmm, from the prev example seems like the higher logit difference is a result of using different dtypes in original and HF implementation. Afaik the original always uses bf16, so might interesting to check. In any case, the current state logits difference look okay to me


@zucchini-nlp Thanks! I'm on vacation this week, so do go ahead if you'd like to finish the PR yourself!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31395). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ready for review, made final changes and uploaded the weights on the Hub, along with their chat templates

@amyeroberts ready for review, addresses the comments

Thanks @jamt9000 💛  The converted model weights are already uploaded to llava-hf org in the hub!