[feat] LlavaNext add feature size check to avoid CUDA Runtime Error
Author: laurentd-lunit
Pull request: yes

# What does this PR do?

In `LlavaNextForConditionalGeneration`, in the forward pass the following is applied:
```
special_image_mask = (
                    (input_ids == self.config.image_token_index).unsqueeze(-1).expand_as(inputs_embeds)
                )
                image_features = image_features.to(inputs_embeds.device, inputs_embeds.dtype)
                inputs_embeds = inputs_embeds.masked_scatter(special_image_mask, image_features)
```
It happened to me that for some edge cases the `special_image_mask` and `image_features` shapes are incompatible which leads to a `RuntimeError: CUDA error: device-side assert triggered` in the `masked_scatter` operation. 
The problem is it only happens very rarely and I couldn't pin point yet what causes the missmatch. 

Rather than fixing the issue for edge cases, in this PR I'm suggesting to first check the respective size of image tokens and image features and ensure they match before applying the `masked_scatter` operation. This allows to raise a `ValueError` instead of getting the CUDA Runtime error which is useful because one can then handle the exception as they see fit and still continue using the GPU while the CUDA Runtime error even if caught as an exception will throw the same CUDA Runtime error if any other CUDA operation is applied, in other words it can't really be handled and necessarily breaks running of the script.

In short, this PR allows nicer error handling in some edge cases in LlavaNext when GPU is used.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@zucchini-nlp 


Comments:
@zucchini-nlp thanks for checking this PR!

> Would be super nice if you can get a reproducer to see why this happens, maybe a script that fails once in 10 runs?

I've encountered this issue using a large custom dataset but I haven't pinned down which sample(s) it happens to yet. I suspect it's an edge case of image dimensions that maybe causes this issue. I'll try and find some time to investigate it and make a reproducer but here this PR was really more about getting around it by simply avoiding CUDA to crash if the dimensions do not match. 

> And regarding changes, can we do same error in all llava models if you can? And I thinkg Paligemma and Qwen2-VL also use masked scatter to replace image tokens

Right, I'll check other Llava models and Paligemma and Qwen2-VL and see if I can make the same changes!

@zucchini-nlp 
After doing some digging, I was able to pin down a sample where it happened within my custom dataset + custom vision backbone but it seems the issue has been fixed by https://github.com/huggingface/transformers/pull/33564. So I don't think it's needed to make a reproducer here since the bug itself has been fixed.

However, I think it could still be useful to check feature sizes and return `ValueErrror`  before doing the `masked_scatter` in case a new bug arises in the future. I added the check to all Llava architectures and found that Paligemma already had it implemented.

Let me know if there are any other changes required and if you think it's still worth merging this PR.


@zucchini-nlp Thanks for your review again! I incorporated your changes and suggestion. Hopefully should be good to merge now.

@LysandreJik  would be good if you could check this when you get the chance, especially I had to modify some of the reference values in `test_modeling_llava.py` and `test_modeling_vip_llava` because it seems the assumed number of tokens was 1 too low: 

Using [llava processor logic](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava/processing_llava.py#L150) to compute the expected number of image tokens:
`num_image_tokens = (height // self.patch_size) * (width // self.patch_size) + 1` 

Here we have high,width = 30 and patch_size = 2 which would make: 15*15+1 = 226 then we do minus 1 because CLS token is ignored so it should be 225 (not 224).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33608). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

the test should have been fixed in the `main`, @laurentd-lunit can you rebase main?