Add LLaVa-1.6, bis
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds [LLaVa-1.6](https://llava-vl.github.io/blog/2024-01-30-llava-next/) (also called LLaVa-NeXT) as a standalone model rather than modifying the existing modeling_llava.py. As discussed in #29012 this might be better.

To do:

- [x] convert all checkpoints once PR is approved, along with corresponding integration test, which will also fix the failing documentation test.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29586). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for opening @NielsRogge! I can see some model related tests are failing - ping me for review once they're all passing 

@amyeroberts I've fixed all tests, only the documentation test is failing because the official checkpoint hasn't been uploaded yet.

Thanks for your review, addressed all comments and uploaded all checkpoints to a [collection](https://huggingface.co/collections/llava-hf/llava-next-65f75c4afac77fd37dbbe6cf).

The documentation test fails, seems like the machine crashed due to the size of the model (cc @ydshieh).

> Thanks for your review, addressed all comments and uploaded all checkpoints to a [collection](https://huggingface.co/collections/llava-hf/llava-next-65f75c4afac77fd37dbbe6cf).
> 
> The documentation test fails, seems like the machine crashed due to the size of the model (cc @ydshieh).

If you can make use lower resource (anyway like fp16, 4-bit etc) that would be great.
If nothing could avoid the crash, just put the file to `utils/not_doctested.txt`

@NielsRogge Please review the PR before asking for review. There's files which should not be included as part of the PR here

Apologies, scripts are removed

Regarding your comment regarding the `aspect_ratio_setting` of [LlavaNextImageProcessor](url) -> actually only `aspect_ratio_setting = "anyres"` is used for llava 1.6 (`LlavaNextForConditionalGeneration`).

The other 2 settings ("clip" and "pad") are there to be used for `LlavaForConditionalGeneration` (the previous generation). This one currently uses `CLIPImageProcessor`, which is technically incorrect, it should use the "pad" setting of `LlavaNextImageProcessor`. But I assume we can't simply replace the image processor of `LlavaForConditionalGeneration` by a new one (cc @ArthurZucker).

Should I keep the `aspect_ratio_setting`, or remove it and only support "anyres" by default? 

> This one currently uses CLIPImageProcessor, which is technically incorrect, it should use the "pad" setting of LlavaNextImageProcessor. But I assume we can't simply replace the image processor of LlavaForConditionalGeneration by a new one (cc @ArthurZucker).

I don't quite understand this statement. "This one" is referring to `LlavaForConditionalGeneration` or `LlavaNextForConditionalGeneration`? 

> But I assume we can't simply replace the image processor of LlavaForConditionalGeneration by a new one (cc @ArthurZucker).

Again, not sure which object you're talking about replacing, but if there's an incorrect implementation it would be better to fix  it, possibly with some kind of deprecation cycle / legacy behaviour flag. If the issue is with the pad method, I don't see why we need to completely replace the image processor, instead of updating `pad`

> Should I keep the aspect_ratio_setting, or remove it and only support "anyres" by default?

We should remove the `aspect_ratio_setting` flag entirely

It doesn't make sense to have an image processor for a specific model, which is preparing inputs for a different model which already has its own image processor.

Ok, I've removed the flag as it leads to confusion. Feel free to approve the PR, if possible.

https://github.com/huggingface/transformers/commit/a2a95164b82960ee1b36b01735b81a8a48ee1958 there were some unprotected imports 😈 

This model when loading gives `The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.`. I am using device_map='auto'. Anyway to fix this?

Hi @aliencaocao, could you open an issue detailing this error and a minimal code snippet to reproduce? This will help us track what has and hasn't been addressed and help people find it if they're experiencing something similar. 

https://github.com/huggingface/transformers/issues/30001