[WIP] Change LlavaProcessor to include more (any) tokenizer and image processor.
Author: muhark
Pull request: yes

# What does this PR do?

This PR changes `processing_llava.py` to permit users to use language and vision backbones with tokenizers and image processors other than `LlamaTokenizer` and `CLIPImageProcessor`.

We are adding to permit out-of-the-box functionality for the [recently uploaded Gemma-based LLaVA model](https://huggingface.co/Intel/llava-gemma-2b) (see #29887).

Fixes #29887.

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

### Regarding documentation:

I made relevant changes to the docstring in the LlavaProcessor class, but decided against modifying the source docs for the LLaVA models because nothing in those suggests that you cannot use alternate tokenizers/processors.

I am happy to change the source docs to give alternate examples, if helpful.

### Regarding tests:

I do not think any new tests are necessary. I ran the existing ones for tokenizers and processors.

## Who can review?

@ArthurZucker


Comments:
## Minimum Working Example

The below code demonstrates an example of using a LLaVA model with a non-Llama language backbone.

```python
import requests
from PIL import Image
from transformers import LlavaProcessor, LlavaForConditionalGeneration

checkpoint = "Intel/llava-gemma-2b"

# Load processor
processor = LlavaProcessor.from_pretrained(checkpoint)

# Prepare inputs
# Use gemma chat template
prompt = processor.tokenizer.apply_chat_template(
    [{'role': 'user', 'content': "What's the content of the image?<image>"}],
    tokenize=False,
    add_generation_prompt=True
)
url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(text=prompt, images=image, return_tensors="pt", add_special_tokens=False)  # Don't double-add <bos>
      
# Generate
model = LlavaForConditionalGeneration.from_pretrained(checkpoint)
generate_ids = model.generate(**inputs, max_new_tokens=10)
output = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
print(output)
```

## Other Notes

I tried to think of a good way to create a tokenizer/image processor "whitelist" (in this case, I need to be able to use `GemmaTokenizer` and `BitImageProcessor`), but I could not find a good way to build this into the existing `ProcessorMixin` functionality.

I am not sure if using `AutoTokenizer` and `AutoImageProcessor` might be removing important guardrails, but I could change the suggested usage to separately loading the tokenizer and image processor (and forgoing the use of the `AutoProcessor` entirely) if that's the case.

Setting to `WIP` while @JoeyTPChou adds support for Pipelines as well.

@ArthurZucker sorry for the delay in getting back to you.

I was not sure how to make the changes to `image_processing_auto.py` and `tokenization_auto.py`.

What is the right way to allow multiple options? e.g.

```python
 ("llava", "CLIPImageProcessor", "BitImageProcessor"), 
```

or

```python
 ("llava", "CLIPImageProcessor"),
 ("llava", "BitImageProcessor"), 
```

Apologies--I was not able to figure this out.

If this functionality is not supported, is there some way to modify the auto loading to:

1. Read from the config
2. If the value is absent from the config, use the default value from `transformers.models.auto`


No most probably `(llava, AutoProcessor)` or `llava, ImageProcessor` ? 

It was actually a good Idea, just did not have the time to guide you sorry 

Closing as completed

Hello, just wonder, if images can be None or not? If I set images to None, it will raise error.

MLLM can talk with text only right?

this might depend on the model so I am not sure. If it feels like a bug feel free to open a PR for a fix!