[Whisper] Reduce batch size in tests
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

The slowest PyTorch tests as of 10th May were reported as follows:

```
68.24s call     tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_maximum_encoding_length_pair_input
64.24s call     tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_maximum_encoding_length_single_input
62.17s call     tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_internal_consistency
60.57s call     tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_add_special_tokens
55.37s call     tests/models/whisper/test_modeling_whisper.py::WhisperEncoderModelTest::test_model_outputs_equivalence
52.59s call     tests/models/mobilebert/test_modeling_mobilebert.py::MobileBertModelTest::test_save_load_fast_init_from_base
...
```

Source: https://huggingface.slack.com/archives/C01NE71C4F7/p1683734816738159

Taking a deeper look, we see that the first of these four tests were Whisper tokeniser tests. Running locally, the Whisper tokenisation tests take a fraction of the time reported above. For instance, I can run the **entire** Whisper tokenisation test suite in 48s, with the longest tests from the CI taking less than 10x the time locally:

```
3.43s call     tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_maximum_encoding_length_pair_input
3.52s call     tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_maximum_encoding_length_single_input
3.43s call     tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_internal_consistency
0.34s call     tests/models/whisper/test_tokenization_whisper.py::WhisperTokenizerTest::test_add_special_tokens
```

Checking more recent CI runs, we see that these tokenisation tests have disappeared from the top tests by time, e.g. for the PR [#23223](https://circleci-tasks-prod.s3.us-east-1.amazonaws.com/forks/storage/artifacts/d5b57382-7f67-4274-9623-7f238ef4fb6f/636881279/6382baaf-cb48-4b59-ad00-ab8ab3d42763/0/~/transformers/reports/tests_torch/durations.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQVFQINEOOAPIWJIW%2F20230524%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230524T150859Z&X-Amz-Expires=60&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEMj%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJHMEUCIFSTbUmgA5nKTMzcsI888FWe4%2BqsFRrE42wJsJkxpXrjAiEA7NGVGvb6DVyu%2FmuTm%2BLg07L6KkB5v%2Fv4yFpxIqql4AUqtAII8P%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FARADGgwwNDU0NjY4MDY1NTYiDHUW%2F9eGKhbQqo67WyqIAt%2FowvbsfkuLoidqZLHQeXrqZl55RY4AJnMGRfT0xY12jCvpjZ7F89p9cJbln9stqj6NpUSSaqzrO4L7XaHhmJ8I8ZRoUo7Uwk4J7ll3CvohzUJqVzYnEzeLvinEF0%2Bi0mx6ZT2DyP8bYjJI%2BWAu25gTByeny05l6xH9PNy7kxurax9scDnBc7Be0Gs56y74F2%2FVvrdCaxigY0wSNfgCXyasfX%2FIq87UP7y%2BVZIxDoL5zD1gbZmUulf3gL5VAcaOwOtkmhCwisjc%2BbTbUSHMTpJZ1D77U3mSmJVXUSQnzpNavl%2FVHXY3DOh45KFoQETemsTehhUlOCMyV91IrO%2BwLacXlzbLsQUMBDCo0LijBjqdAVgdasvAQc5feVYL4SuV5Re4TIrGh6cLLj689oFfoVilLj8AjcOj5GSFHUBCzH792fYCmTZIF%2B66qJ3ieBTup5C%2B1PkaoEZ3mQjAFi8fjUiDDwrFGWlCwTtOklha1HAMKPPmEIW4Q5nhp5OkvPU5CPrZcIm90VNLGCy7CunrrugV74llZpDAU5UwwkUJRYLp3aP8nifKGMDj924ubj8%3D&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=9c473476aa6883dc476f641125a3b9ea33a8c5df489676e0ff1bab13129831b0) from the 18th May. Note that these are just generic tokenisation tests, and the Whisper tokeniser is one-to-one the same as GPT2, so we’d expect the same runtime here.

In this PR, we speed-up the Whisper modelling tests by a factor of ~4x by reducing the batch size from 13 -> 2, which should address the slow modelling tests. We'll monitor the Whisper tokenisation tests to see if they keep cropping up as the slowest PyTorch tests in the future and amend as necessary.

Comments:
_The documentation is not available anymore as the PR was closed or merged._