Add tests for batching support
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #29179 . Currently the tests are passing for all models, and batched and unbatched outputs have nearly same outputs.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?
@gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29297). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp Running your test script on my setup I get

```
tensor(0., device='cuda:0')
tensor(4.7684e-07, device='cuda:0')
```

Could it be due to different hardware and/or versions? 🤔 For context, I'm running with CUDA 12.1, `torch==2.3.0.dev20240208+cu121`, and an nvidia RTX3090 

Using cosine distance to check cv/audio models helped to solve the issue. The way we check if the model is conv-based or not might not be the best option,  I will see if I can make it better. 

Now everything is passing the test, yay!

Next steps:
- `make fixup` :D
- `LayoutLMv2ModelTest` seems to be failing. Perhaps skip it as well?
- Update the PR header (we no longer have issues with CV models)
- Hit `ready to review` so it is no longer a draft PR

Then:
- Tag both core maintainers (Arthur to review since he's on rotation, Amy to have a look on the CV model skips)

@amyeroberts @ArthurZucker ready for review

@amyeroberts: you're not on rotation, but this PR includes a new test that, to make CI green, @zucchini-nlp had to a) make minor CV model changes; b) add skips on CV models 🤗 

Let us know if you find anything wrong! If the skips are not supposed to be skips, then I'd like to ask to keep them as TODOs so as to be tended by someone with CV expertise 🙏 

@zucchini-nlp the change in batch size has made the whisper tests unhappy :D after that is cleared up, we can merge this PR 🤗 

This PR is now ready to be merged. An unexpected bug was found yesterday, but it's fixed already. I will also merge `main` to resolve conflicts 
@gante 

@zucchini-nlp needs a `make fixup` and it is ready to be merged 

Hi @zucchini-nlp Thank you for your work. Our CI reports 2 failing tests:

```
tests/models/mamba/test_modeling_mamba.py::MambaModelTest::test_batching_equivalence
(line 726)  AttributeError: 'MambaCache' object has no attribute 'dim'
```

and 

```
tests/models/seggpt/test_modeling_seggpt.py::SegGptModelTest::test_batching_equivalence
(line 765)  AssertionError: tensor(False, device='cuda:0') is not true : Batched and Single row outputs are not equal in SegGptModel for key=hidden_states. Difference=0.1510644555091858.
```

If you can take a look on these, that would be great 🙏 Thank you.

@ydshieh (and @amyeroberts, since you're working on the test fetcher): in this case, the new failures arise because the branch merged in this PR, which adds a new mixin test, was out of sync with `main` with respect to new models. Is there any way we can prevent this without too much effort? 

I'm not seeing an automated way to do it. The only thing that comes to mind is to ask to rebase before merging this type of PRs :D 

@gante Yeah, I don't see an satisfying way neither.

One possibility is to change circleCI config file to rebase (on the fly) on the latest main during the CI runs.

> One possibility is to change circleCI config file to rebase (on the fly) on the latest main during the CI runs.

@ydshieh That could be... interesting! Try to rebase with `main`, roll back if a conflict arises, then run tests. If an informative message is added in the report (e.g. "the tests were run after rebasing on main"), we might save ourselves a few situations like this :D 

> One possibility is to change circleCI config file to rebase (on the fly) on the latest main during the CI runs.

I'm not sure this is a great idea, for two reasons: 
* the code version being used to run tests is not the version on the branch, which can lead to all sorts of confusion
* It's pretty common in development to not rebase for a few commits. Even though _technically_ PRs should be opened when the branch is review ready, this isn't what happens. It's going to end up forcing people to constantly rebase and force push, even if there aren't meaningful commits being added. As someone who gets a notification for every commit being pushed, I'd really rather avoid this. 

To test this out, I'd rather have an optional feature, similar to the run all tests command we can have in commits e.g. run-with-base, than we can comment on github, or ask users to run asa final step. 

>  than we can comment on github

That would also be quite cool -- and could save some time from the core maintainers, if the other reviewers diligently test against `main` on wide-reaching changes before merging 🤗 

Totally agree with @amyeroberts that's  why I say `I don't see an satisfying way`