Repository: huggingface/transformers

Issue:
Regression in training speed since 4.4.0
## Environment info

- `transformers` version: 4.4.0/4.6.1
- Platform: Linux-5.8.0-53-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: Yes, RTX 3090
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger @patrickvonplaten

## Information

I've noticed a training speed regression between 4.4.0 and 4.6.0:

4.4.0 BS8:
{'train_runtime': 21.363, 'train_samples_per_second': 5.851, 'epoch': 1.0}
{'train_runtime': 21.6148, 'train_samples_per_second': 5.783, 'epoch': 1.0}
{'train_runtime': 21.7867, 'train_samples_per_second': 5.737, 'epoch': 1.0}  
 
4.6.1 BS8:
{'train_runtime': 23.7011, 'train_samples_per_second': 5.274, 'epoch': 1.0}    
{'train_runtime': 24.2845, 'train_samples_per_second': 5.147, 'epoch': 1.0}
{'train_runtime': 23.5801, 'train_samples_per_second': 5.301, 'epoch': 1.0}    
    
4.4.0 BS4:
{'train_runtime': 25.4107, 'train_samples_per_second': 9.838, 'epoch': 1.0}  
  
4.6.1 BS4:
{'train_runtime': 31.2902, 'train_samples_per_second': 7.99, 'epoch': 1.0}

I'm running the Pytorch 1.8.1. release, on my RTX3090/Ryzen 3700X workstation.

The performance loss seems to increase with smaller batch sizes, leading me to think it's something in Trainer.

Although I found the regression with Sequence Classification, I've found the the slowdown transfers to other tasks as well.

## To reproduce

```
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer

BATCH_SIZE = 4

raw_datasets = load_dataset("imdb")

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
full_train_dataset = tokenized_datasets["train"]
full_eval_dataset = tokenized_datasets["test"]

model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)

training_args = TrainingArguments("test_trainer", num_train_epochs=1, per_device_train_batch_size=BATCH_SIZE)

trainer = Trainer(
    model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset
)

trainer.train()
```


Discussion:
Thanks for flagging! I used your reproducer (very nice by the way!) and a git bisect and it comes from #10937, which reworks the way memory metrics are computed. You just have to set `skip_memory_metrics=True` in your `TrainingArguments` to skip the memory metric computation and you will get your performance back.

Resolving patch:
diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
--- a/src/transformers/training_args.py
+++ b/src/transformers/training_args.py
@@ -303,8 +303,9 @@ class TrainingArguments:
             otherwise.
         dataloader_pin_memory (:obj:`bool`, `optional`, defaults to :obj:`True`):
             Whether you want to pin memory in data loaders or not. Will default to :obj:`True`.
-        skip_memory_metrics (:obj:`bool`, `optional`, defaults to :obj:`False`):
-            Whether to skip adding of memory profiler reports to metrics. Defaults to :obj:`False`.
+        skip_memory_metrics (:obj:`bool`, `optional`, defaults to :obj:`True`):
+            Whether to skip adding of memory profiler reports to metrics. This is skipped by default because it slows
+            down the training and evaluation speed.
         push_to_hub (:obj:`bool`, `optional`, defaults to :obj:`False`):
             Whether or not to upload the trained model to the hub after training. This argument is not directly used by
             :class:`~transformers.Trainer`, it's intended to be used by your training/evaluation scripts instead. See
@@ -546,7 +547,7 @@ class TrainingArguments:
         default=True, metadata={"help": "Whether or not to pin memory for DataLoader."}
     )
     skip_memory_metrics: bool = field(
-        default=False, metadata={"help": "Whether or not to skip adding of memory profiler reports to metrics."}
+        default=True, metadata={"help": "Whether or not to skip adding of memory profiler reports to metrics."}
     )
     use_legacy_prediction_loop: bool = field(
         default=False, metadata={"help": "Whether or not to use the legacy prediction_loop in the Trainer."}


Test patch: