Benchmarking on TPU shows clearly wrong results
Author: sslotin
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

I'm trying to benchmark performance of TPUs and the results don't make sense: they are the same for all batch sizes.

It was mentioned [in the pull request that added the feature](https://github.com/huggingface/transformers/pull/4850#issuecomment-640751636) but the PR was merged anyway.

## To reproduce

```
from transformers import PyTorchBenchmark, PyTorchBenchmarkArguments

args = PyTorchBenchmarkArguments(
    models=["bert-large-uncased"],
    batch_sizes=[i * 1024 for i in range(2, 17)],
    sequence_lengths=[16],
    training=True,
    no_memory=True
)

benchmark = PyTorchBenchmark(args)
results = benchmark.run()
```

Output:

```
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
      bert-large-uncased            2048             16            0.027     
      bert-large-uncased            3072             16            0.029     
      bert-large-uncased            4096             16            0.028     
      bert-large-uncased            5120             16            0.027     
      bert-large-uncased            6144             16            0.027     
      bert-large-uncased            7168             16            0.028     
      bert-large-uncased            8192             16            0.028     
      bert-large-uncased            9216             16            0.027     
      bert-large-uncased           10240             16            0.027     
      bert-large-uncased           11264             16            0.027     
      bert-large-uncased           12288             16            0.027     
      bert-large-uncased           13312             16            0.027     
      bert-large-uncased           14336             16            0.027     
      bert-large-uncased           15360             16            0.028     
      bert-large-uncased           16384             16            0.028     
--------------------------------------------------------------------------------
TPU was used for inference. Note that the time after compilation stabilized (after ~10 inferences model.forward(..) calls) was measured.

====================        TRAIN - SPEED - RESULTS         ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s   
--------------------------------------------------------------------------------
      bert-large-uncased            2048             16            0.089     
      bert-large-uncased            3072             16            0.074     
      bert-large-uncased            4096             16            0.091     
      bert-large-uncased            5120             16            0.091     
      bert-large-uncased            6144             16            0.091     
      bert-large-uncased            7168             16            0.075     
      bert-large-uncased            8192             16            0.089     
      bert-large-uncased            9216             16             0.09     
      bert-large-uncased           10240             16            0.074     
      bert-large-uncased           11264             16             0.09     
      bert-large-uncased           12288             16             0.09     
      bert-large-uncased           13312             16             0.09     
      bert-large-uncased           14336             16            0.077     
      bert-large-uncased           15360             16            0.089     
      bert-large-uncased           16384             16            0.091     
--------------------------------------------------------------------------------
TPU was used for training. Note that the time after compilation stabilized (after ~10 train loss=model.forward(...) + loss.backward() calls) was measured.

```

## Environment info

Running on GKE cluster, TPUv3-8, vanilla tpu-pytorch/xla:r1.5 image, XRT_TPU_CONFIG set

```
====================        ENVIRONMENT INFORMATION         ====================
The current process just got forked. Disabling parallelism to avoid deadlocks...
To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)
- transformers_version: 3.0.0
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.5.0a0+6d48871
- python_version: 3.6.10
- system: Linux
- cpu: 
- architecture: 64bit
- date: 2020-07-01
- time: 14:37:15.608184
- fp16: False
- use_multiprocessing: False
- only_pretrain_model: False
- cpu_ram_mb: 30156
- use_gpu: False
- use_tpu: True
```

Comments:
That looks like some solid batch parallelization :D Yeah these results don't look very accurate.

To be honest TPU Benchmarking is not very well tested yet and probably not very reliable, also partly because PyTorch/XLA is not very robust yet either. I will try to see if I can find the reason for this.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.