Granite Vision Support
Author: alex-jw-brooks
Pull request: yes
Labels: name: New model

name: Multimodal

# What does this PR do?

This PR adds compatibility for IBM's upcoming Granite Vision models (which are based on LLava Next). The main changes here are:
- The vision feature layer, which is currently expected to be an integer can now also be a list of integers; if a list of integers are provided, the image features are the concatenated before applying the feature selection strategy
- The validation which breaks visual encoders with no CLS (discussed a bit [here](https://github.com/huggingface/transformers/pull/35262)) is removed. I did add a warning if the feature packing explodes with the `default` strategy as well.

This change was applied in a lot of places to make the checks for repository consistency happy + the config consistent, but the multimodal granite models are instances of `LlavaNextForConditionalGeneration`. I added a test for each changed model to ensure that things don't blow up if a list of vision feature layers is provided, but if there is another path forward that is preferred to changing several models at the same time to add compatibility with llava next, I'm happy to revise this PR as needed.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts, @qubvel, @zucchini-nlp 

Comments:
Thanks a lot for the quick review @zucchini-nlp 😄 For `vipllava`, do you mean allowing the `vision_feature_layers` to be of type `Union[int, List[int]]`, allowing a strategy, or something else?

@alex-jw-brooks 

> allowing the vision_feature_layers to be of type Union[int, List[int]]

exactly, this is what I meant so we can be consistent for all llava models :) 

Awesome, thank you for the clarification @zucchini-nlp 😄  I made the change in vipllava and rebased this PR, it should be ready for another look when you have a moment!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35579). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Cool, thanks a lot! Let's get one more review from the core maintainer and then we'll merge

We would also want to add an integration test with expected values!

Awesome, thank you for the feedback @ArthurZucker! 

> LGTM, but we need to add a granite_multimodal.md with model details!

Given that these models will be run in `transformers` as an instance of llava next and not their own class, would you prefer that we add a separate `granite_multimodal.md`, or add a note in the llava next docs? I'm happy to do either, and will update this PR tomorrow, once I've heard back from my colleagues on which paper would be best to cite for these models 😄 

> We would also want to add an integration test with expected values!

I've added an integration test that works for the version of the 2b model that I have locally. However, since the models are still being finalized (experimental release planned for the next week or so), I've marked it with `@unittest.skip` - would it be possible to merge this PR so that there is day zero support in `transformers`, and submit a follow-up PR enabling the integration test with the correct model ID + value once the 2b model is available?



A separate `granite_multimodal.md` would be nice IMO! 

yep completely possible! 


Great, thanks so much @ArthurZucker! It looks like the models are actually going to be released as `granite vision` - I went ahead and updated references from `granite multimodal` -> `granite vision` and added a short description of the model + example to `docs/source/en/model_doc/granitevision.md`. It should be ready for another look when you have a moment!

It seems like there will be a publication,  but it's not available quite yet - I'll be sure to open follow-up PRs for the integration test and doc updates (e.g., paper citation and model ID fix if the name ends up incorrect in the example) in the coming weeks as things are published 😄 

Congrats on the merge! 🚀