Make tests run in less time by reducing `batch_size`
Author: ydshieh
Pull request: yes

# What does this PR do?

Saving 40 seconds for 

>  python -m pytest -v -m 'not generate' tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest --make-reports=test_dummy

The most significant one is

> test_model_outputs_equivalence

20 sec --> 4 sec.

Have no clear idea why this model use more time, but it's encoder-decoder, each of 2 layers ... and this test `test_model_outputs_equivalence` have many test cases, and being even slower if we have `output_attentions=True`.

Let's simply use smaller `batch_size` for this model, and in general, for more other models.

### Before

```
slowest durations
20.34s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_model_outputs_equivalence
3.74s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing_use_reentrant
3.71s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing_use_reentrant_false
3.71s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing
3.59s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_resize_tokens_embeddings
2.75s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_attention_outputs
2.40s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_bc_torch_dtype
2.01s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_feed_forward_chunking
2.01s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_determinism
1.74s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_save_load
1.66s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_generate_fp16
1.59s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_hidden_states_output
1.45s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_retain_grad_hidden_states_attentions
1.44s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training
1.08s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_encoder_decoder_model_standalone
1.07s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_batching_equivalence
0.79s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_inputs_embeds
0.79s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_inputs_embeds_matches_input_ids
0.78s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_resize_embeddings_untied
0.24s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_can_use_safetensors
0.21s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_load_save_without_tied_weights
0.18s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_from_pretrained_no_checkpoint
0.14s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_save_load_strict
0.09s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_can_be_initialized_on_meta
0.07s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_model_weights_reload_no_missing_tied_weights
509 durations < 0.05 secs were omitted
```

### After

```
slowest durations
3.96s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_model_outputs_equivalence
2.46s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_bc_torch_dtype
1.09s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_resize_tokens_embeddings
0.83s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing
0.81s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing_use_reentrant
0.78s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training_gradient_checkpointing_use_reentrant_false
0.61s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_generate_fp16
0.59s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_save_load
0.50s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_attention_outputs
0.47s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_determinism
0.34s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_feed_forward_chunking
0.33s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_retain_grad_hidden_states_attentions
0.33s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_batching_equivalence
0.32s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_training
0.27s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_encoder_decoder_model_standalone
0.22s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_load_save_without_tied_weights
0.22s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_can_use_safetensors
0.21s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_hidden_states_output
0.21s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_save_load_strict
0.17s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_resize_embeddings_untied
0.17s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_from_pretrained_no_checkpoint
0.11s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_inputs_embeds
0.11s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_model_weights_reload_no_missing_tied_weights
0.10s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_inputs_embeds_matches_input_ids
0.09s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_can_be_initialized_on_meta
0.06s call     tests/models/pegasus_x/test_modeling_pegasus_x.py::PegasusXModelTest::test_torch_save_load
508 durations < 0.05 secs were omitted
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42213). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This test `test_model_outputs_equivalence` appears 447 times in the report of long run tests.

It test many cases (which is not a good approach for testing), and `output_attentions=True` case is even much slower
(large tensor, I guess).

For this model , it's encoder-decoder, each of 2 layers. (But not sure if this play a significant role for slowness here.)


I will wait a bit until the CI on the main branch is clean.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: albert, aria, audio_spectrogram_transformer, autoformer, bamba, bart, beit, bert, bert_generation, big_bird, bigbird_pegasus, biogpt, bitnet, blenderbot_small, bros, canine