Fix reproducible tests in Trainer
Author: sgugger
Pull request: yes

This fixes the tests that were using hardcoded values for reproducible training by testing instead the training yields the same result as one run with the same seed during setup. As a result, it can also run in a multigpu-environment (which is why I removed the decorator).

I also fixed the batch size to count the number of steps, which should also fix those tests for multigpu-environent. @stas00, when you get a chance to run the tests on that file, it would be great to know if the fix worked as intended.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7119?src=pr&el=h1) Report
> Merging [#7119](https://codecov.io/gh/huggingface/transformers/pull/7119?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5636cbb25d248b61bb9027d026dddcd6d1599b0b?el=desc) will **decrease** coverage by `1.26%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/7119/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/7119?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #7119      +/-   ##
==========================================
- Coverage   79.48%   78.22%   -1.27%     
==========================================
  Files         168      168              
  Lines       32281    32281              
==========================================
- Hits        25660    25251     -409     
- Misses       6621     7030     +409     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/7119?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl90NS5weQ==) | `26.05% <0.00%> (-63.52%)` | :arrow_down: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `25.24% <0.00%> (-55.76%)` | :arrow_down: |
| [src/transformers/optimization.py](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb24ucHk=) | `34.28% <0.00%> (-48.00%)` | :arrow_down: |
| [src/transformers/optimization\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9vcHRpbWl6YXRpb25fdGYucHk=) | `33.33% <0.00%> (-24.33%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hdXRvLnB5) | `57.28% <0.00%> (-15.08%)` | :arrow_down: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `13.76% <0.00%> (-14.38%)` | :arrow_down: |
| [src/transformers/modeling\_transfo\_xl\_utilities.py](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190cmFuc2ZvX3hsX3V0aWxpdGllcy5weQ==) | `52.98% <0.00%> (-13.44%)` | :arrow_down: |
| [src/transformers/modeling\_transfo\_xl.py](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190cmFuc2ZvX3hsLnB5) | `67.10% <0.00%> (-12.67%)` | :arrow_down: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `74.18% <0.00%> (-12.29%)` | :arrow_down: |
| [src/transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hdXRvLnB5) | `70.33% <0.00%> (-11.97%)` | :arrow_down: |
| ... and [16 more](https://codecov.io/gh/huggingface/transformers/pull/7119/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/7119?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/7119?src=pr&el=footer). Last update [5636cbb...5d7c2ae](https://codecov.io/gh/huggingface/transformers/pull/7119?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


with both gpus seen:

```
pytest tests/test_trainer.py
Test session starts (platform: linux, Python 3.8.5, pytest 6.0.1, pytest-sugar 0.9.4)
Using --randomly-seed=3075405078
rootdir: /mnt/nvme1/code/huggingface/transformers-master
plugins: xdist-2.1.0, forked-1.3.0, hydra-core-1.0.0, pspec-0.0.4, sugar-0.9.4, randomly-3.4.1, cov-2.10.1, flakefinder-1.0.0
collecting ... 
 tests/test_trainer.py ✓✓✓✓✓                                                                                                            45% ████▋     

――――――――――――――――――――――――――――――――――――――――――――――― TrainerIntegrationTest.test_train_and_eval_dataloaders ―――――――――――――――――――――――――――――――――――――――――――――――

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_train_and_eval_dataloaders>

    def test_train_and_eval_dataloaders(self):
        trainer = get_regression_trainer(learning_rate=0.1, per_device_train_batch_size=16)
>       self.assertEqual(trainer.get_train_dataloader().batch_size, 16)
E       AssertionError: 32 != 16

tests/test_trainer.py:144: AssertionError
```

with zero or 1 gpu it passes.


Could you retest now?

Success!

Thanks for checking @stas00 !