Add support for GIT model in VQA pipelines
Author: marechaux
Pull request: yes

# What does this PR do?

This PR implement support for generative model in VQA pipeline (and more precisely GIT model).

Fixes part of #21110

This is my first contribution here, I am uncertain if my approach is correct. Please advise me if any modifications are necessary 😃 

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. => #21110 
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@NielsRogge 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23348). All of your documentation changes will be reflected on that endpoint.

I have two remarks here : 
* The CI is red due to this change : https://github.com/huggingface/transformers/pull/23348/files#diff-b452cc4f093e4b991e92054cf7d504edab44be07c4957969df85a5562238313cR48 the model now used in test is `hf-internal-testing/tiny-random-ViltForQuestionAnswering` instead of `hf-internal-testing/tiny-vilt-random-vqa` (and it seems like the new model image processor can't process images, I am not sure about how to fix it)
* The test for GIT in VQA pipeline doesn't run because `hf-internal-testing/tiny-random-GitForVisualQuestionAnswering` doesn't exist, I need help about this point as well 

Thanks for the review, I updated my changes following your comments. However, I have several doubt on my approach. 

### Beam search for scores 
I use beam scores to provide a score to follow the "signature" of the pipeline described [here](https://github.com/huggingface/transformers/blob/c3c9b03d55f2d8094a2ac058db566d469baa8bbd/src/transformers/pipelines/visual_question_answering.py#L106-L110). Is it a correct ? 

The beam search is so slow that it makes the pipeline test [timeout](https://app.circleci.com/pipelines/github/huggingface/transformers/65019/workflows/be9d4fde-5cbb-4db0-ba85-0110d8988953/jobs/806494), it runs locally but in more than 120s,  which make me think I'm not in the right way here

### Tokenizer padding
Also when I use the pipeline with `microsoft/git-base-textvqa` in batch mode, I have this warning : 
```
A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set `padding_side='left'` when initializing the tokenizer.
```
This warning is legitimate, but I don't know how to fix it as `padding_side` can only be set at tokenizer init. 

### Broken `Vilt` tests
Due to [this change](https://github.com/huggingface/transformers/pull/23348/files#diff-b452cc4f093e4b991e92054cf7d504edab44be07c4957969df85a5562238313cR48) , the model used in unit test for Vilt model is now `hf-internal-testing/tiny-random-ViltForQuestionAnswering` instead of `hf-internal-testing/tiny-vilt-random-vqa`.

It seems like the new model image processor can't process images, I am not sure about how to fix it

Should I fix the model directly in the hub ? 

Gently ping here @NielsRogge  :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.