Fix Aria tests
Author: jiqing-feng
Pull request: yes

Reproduce:
`TRANSFORMERS_TEST_DEVICE=cuda RUN_SLOW=1 pytest -rA tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationIntegrationTest::test_batched_generation`

Error log:
```
FAILED tests/models/aria/test_modeling_aria.py::AriaForConditionalGenerationIntegrationTest::test_batched_generation - RuntimeError: INDICES element is out of DATA bounds, id=100352 axis_dim=100352
```

I found 4 issues in this case:
1. The pad_token_id is the same as vocab_size which will cause out of bounds. It cames from both [config.json](https://huggingface.co/rhymes-ai/Aria/blob/main/config.json) and [added_tokens.json](https://huggingface.co/rhymes-ai/Aria/blob/main/added_tokens.json) have pad token.
2. The input image and input prompt should have same batch size
3. The input pixel value shoule be cast to the model.dtype, see the [model hub usage](https://huggingface.co/rhymes-ai/Aria#inference)
4. This model uses torch MHA, which will read the weight directly and apply torch.matmul without dequantizing the weights. We should skip quantizing the MHA weight.

After fixing the 4 issues, the test can correctly run.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Hi @ydshieh @SunMarc . Please review this PR. Thanks!

cc @zucchini-nlp for VLMs as well

run-slow: aria

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/aria']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37444). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hm slow tests are oom-ing for aria, do you think we need to add `@require_torch_large_gpu` @ydshieh ?

> Hm slow tests are oom-ing for aria, do you think we need to add `@require_torch_large_gpu` @ydshieh ?

Add `@require_torch_large_gpu` means it will not be run in daily CI, only run in some occasions that team members have less focus to check. 

However, all those integration tests already use 4-bits and there is no smaller model size for this model architecture.  Let's add it anyway.

Thank you for bring this issue up 👍 

Hi @ydshieh @zucchini-nlp . I have fixed your comments, please review it again. Thanks!

> Thanks @jiqing-feng ! Looks good to me, left one tiny comment so we get better error messages when tests fails
> 
> Approving but before merging can you update hub config as suggested in above comment, and remove the line ` processor.tokenizer.pad_token_id = model.config.pad_token_id` from tests (if pad token is assigned correctly, we shouldn't explicitly set it to eos again)

Yes, I will remove it once [20](https://huggingface.co/rhymes-ai/Aria/discussions/20) is merged

Hi @ydshieh @zucchini-nlp  The model hub PR is merged, I also removed the pad_token setting. The failed tests seem not related to my changes.

@ydshieh can you merge?

Tests were fixed on `main`, a rebase should help now

Hi @zucchini-nlp . Yes, the tests all passed after I rebased the main branch. This PR is ready to be merged.