Slower training time per batch for increasing dataset size
Author: mattivi
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.0.0-rc-1
- `tokenizers` version: 0.9.4
- `datasets` version: 1.1.3
- Platform: Linux-3.10.0-862.el7.x86_64-x86_64-with-centos-7.8.2003-Core
- Python version: 3.7.2
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, V100
- Using distributed or parallel set-up in script?: No

### Who can help

 @patrickvonplaten @sgugger @LysandreJik 

## Information

Model I am using: BERT, language modelling task.


The problem arises when using:
* [ x] the official example scripts: /examples/language-modeling/run_mlm.py

The tasks I am working on is:
* [ x] my own task or dataset: BERT MLM pre-training with own dataset

I need to pre-train a BERT base model from scratch with own dataset. 
The dataset has millions of lines, each line is a short document.
I am experiencing slower training time given increasing size of the dataset. 

To debug the problem, I have already tried to split original datasets into several smaller files (issue  https://github.com/huggingface/datasets/issues/610), switch on/off the caching mechanism, but no improvements. 

What could it be? I am not able to find the origin of the problem. Thanks a lot!


## To reproduce

Steps to reproduce the behavior:

1. run_mlm.py with increasing dataset sizes.
2. This results in slower training time for each batch.

Below some stats, each batch is 128, and I am running run_mlm.py with --line_by_line option. The increase seems not linear.

| Number lines in dataset  |  seconds per batch |
|---|---|
|  100k |  0.16 |
|  10M |  0.25 |
|  100M |  3 |

BERT parameters:
```
Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 18,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "type_vocab_size": 2,
  "vocab_size": 32000
}
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I expect that the training time for each batch will remain constant given different datasets' sizes. 


Comments:
Mmm, this looks like a problem in Datasets, @lhoestq @thomwolf ?

Could you check if the same speed differences appear if you're iterating through the `tokenized_datasets` ? 
For example doing
```python
for i in range(0, len(tokenized_datasets), batch_size):
    batch = tokenized_datasets[i:i + batch_size]
```
If so, please open an issue on the Datasets repo so we can investigate

I have run the following code for both dataset 10M rows and 100M rows, and the speed is a bit slower for the 100M dataset compared to the 10M dataset. 
However, when training BERT, I have much higher differences (e.g. in the timing above is 3s Vs. 0.25s). 

```python
print("--- Starting test for 10M batches ---")
num_batches = 10000000
batch_size = 128
import time
start_time = time.time()
for i in range(0, num_batches, batch_size):
    batch = tokenized_datasets['train'][i:i + batch_size]
end_time = time.time() - start_time
print("--- %3.3f seconds per 10M batches ---" % (end_time))

```

|  Number of lines in dataset | Seconds   |
|---|---|
| 10M  |  241 |
|  100M |  303  |

It looks like the major slowdown comes from somewhere else then.
It could either be from the PyTorch DataLoader but my best guess would be the PyTorch RandomSampler.
For big datasets the sampler takes a ton of RAM as mentioned in https://github.com/huggingface/datasets/issues/610#issuecomment-731725078, it could slowdown your training signficantly.

Could you run the same experiment with the dataloader and the random sampler to make sure ?

To debug I have replaced RandomSampler with SequentialSampler in Trainer class, https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py at line 374. 
With SequentialSampler it works as expected, with no slower time.

How to fix RandomSampler now? 

`RandomSampler` comes from PyTorch, so you can open an issue there. You will most likely need to implement your own random-ih sampler that goes faster than PyTorch.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

This is still an issue today. I got a 10x speed up in training on a larger dataset, by switching to the sequential sampler (due to the RandomSampler bottleneck). Monkeypatch here to switch to the seqential sampler in case it's useful:

```python
import transformers.trainer as trainer
from transformers.trainer import SequentialSampler


def sampler_monkey_patch(dataset, generator):
    return SequentialSampler(dataset)


trainer.RandomSampler = sampler_monkey_patch
```

Versions this was used with:
```
transformers==4.26.1
pytorch==1.13.1
datasets==2.10.1
python==3.9.16
```

To detect if this is an issue for you, it's useful to compare the rate at which samples are processed (and gpu utilization), for a small dataset slice versus a large one.

You can also use an IterableDataset :

```python 
train_dataset = train_dataset.to_iterable_dataset()
```

PS : pass `num_shards=` with a factor of `num_workers` to distribute the data evenly across DataLoader workers 

PS2 : for distributed, see the "Distributed" section at https://huggingface.co/docs/datasets/use_with_pytorch

@sgugger - this issue was closed prematurely. See the comment above where monkeypatching `Trainer` to use a `SequentialSampler` instead of a `RandomSampler` results in a 10x speedup. Could  you please re-open this issue?

I disagree that this is simply a PyTorch issue. Even if this was purely caused by a bug in PyTorch (which is by no means obvious or clear; see the corresponding PyTorch issue [here](https://github.com/pytorch/pytorch/issues/50689)), HF should simply not include this class in its codebase if it entails causing such a bug to users of the `Trainer` class. At the very least, there should be an official workaround.