Training with a large dataset
Author: hgjlee
Pull request: no

I'm currently following the tutorial on how to train a new language model ([here](https://huggingface.co/blog/how-to-train)) and facing some issues on Colab because of my large training corpus (+40 mil lines, 5 GB). 

I tried to use IterableDataset so I can load data on the fly, and I'm getting this error when I try to train with the script provided in the tutorial:
"ValueError: DataLoader with IterableDataset: expected unspecified sampler option, but got sampler=<torch.utils.data.sampler.RandomSampler object at 0x7f777bdbe710>" 

What's the best way to resolve this? 

Comments:
Hi, can you share more details, (all the details asked in the issue template for instance).
In particular, the exact command line you are using, the full error message, etc...

Hi! Thank you for your reply. My apologies for not using the template.

I'm following the tutorial on Colab and switched LineByLineTextDataset to the following custom dataset class to process each line on the fly rather than loading everything in memory:
```
from torch.utils.data import IterableDataset

class CustomIterableDataset(IterableDataset):
  def __init__(self, filename, tokenizer, block_size, len):
    self.filename = filename
    self.tokenizer = tokenizer
    self.block_size = block_size
    self.len = len 

  def preprocess(self, text):
    batch_encoding = self.tokenizer(text.strip("\n"), add_special_tokens=True, truncation=True, max_length=self.block_size)

    return torch.tensor(batch_encoding["input_ids"])

  def line_mapper(self, line):      
    return self.preprocess(line)

  def __iter__(self):
    file_itr = open(self.filename, encoding="utf-8")
    mapped_itr = map(self.line_mapper, file_itr)

    return mapped_itr

  def __len__(self):
    return self.len

dataset = CustomIterableDataset("large_corpus.txt", tokenizer=tokenizer, block_size=128, len=40228303)
```
I run the rest of the script until this part:
```
%%time
trainer.train()
```

The error looks like this:
```
ValueError                                Traceback (most recent call last)
<ipython-input-31-0c647bc3a8b8> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', 'trainer.train()')

5 frames
<decorator-gen-60> in time(self, line, cell, local_ns)

<timed eval> in <module>()

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __init__(self, dataset, batch_size, shuffle, sampler, batch_sampler, num_workers, collate_fn, pin_memory, drop_last, timeout, worker_init_fn, multiprocessing_context)
    177                 raise ValueError(
    178                     "DataLoader with IterableDataset: expected unspecified "
--> 179                     "sampler option, but got sampler={}".format(sampler))
    180             elif batch_sampler is not None:
    181                 # See NOTE [ Custom Samplers and IterableDataset ]

ValueError: DataLoader with IterableDataset: expected unspecified sampler option, but got sampler=<torch.utils.data.sampler.RandomSampler object at 0x7f1eaf8fa128>
```

Please let me know if there's anything else I can provide to help you understand this problem.

That's because unfortunately the trainer cannot be currently used with an `IterableDataset`, because the `get_train_dataloader` method creates a `DataLoader` with a sampler, while `IterableDataset` may not be used with a sampler. You could override the trainer and reimplement that method as follows:

```py
    def get_train_dataloader(self) -> DataLoader:
        if self.train_dataset is None:
            raise ValueError("Trainer: training requires a train_dataset.")
        if is_tpu_available():
            train_sampler = get_tpu_sampler(self.train_dataset)
        else:
            train_sampler = (
                RandomSampler(self.train_dataset)
                if self.args.local_rank == -1
                else DistributedSampler(self.train_dataset)
            )
        data_loader = DataLoader(
            self.train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler if not isinstance(self.train_dataset, IterableDataset) else None,
            collate_fn=self.data_collator.collate_batch,
        )
```

Note how we're not passing the sampler if it's an `IterableDataset`.

@LysandreJik Thank you so much for your help. I just started training with your suggestion. 

For future references, I had to make the following trivial adjustments to the snippet to make it with the current master version.

```
from torch.utils.data.dataset import IterableDataset

def get_train_dataloader(self) -> DataLoader:
        if self.train_dataset is None:
            raise ValueError("Trainer: training requires a train_dataset.")
        if is_torch_tpu_available():
            train_sampler = get_tpu_sampler(self.train_dataset)
        else:
            train_sampler = (
                RandomSampler(self.train_dataset)
                if self.args.local_rank == -1
                else DistributedSampler(self.train_dataset)
            )
            
        data_loader = DataLoader(
            self.train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler if not isinstance(self.train_dataset, IterableDataset) else None,
            collate_fn=self.data_collator,
        )

        return data_loader
```

oops yes forgot the return statement :sweat_smile: glad you got it to work!

Thanks! It was super helpful. Actually, I just had to change is_tpu_available() -> is_torch_tpu_available() and collate_fn=self.data_collator.collate_batch -> collate_fn=self.data_collator. I wasn't sure if these were from different versions. 

> I'm currently following the tutorial on how to train a new language model ([here](https://huggingface.co/blog/how-to-train)) and facing some issues on Colab because of my large training corpus (+40 mil lines, 5 GB).
> 
> I tried to use IterableDataset so I can load data on the fly, and I'm getting this error when I try to train with the script provided in the tutorial:
> "ValueError: DataLoader with IterableDataset: expected unspecified sampler option, but got sampler=<torch.utils.data.sampler.RandomSampler object at 0x7f777bdbe710>"
> 
> What's the best way to resolve this?

HI hgjlee,

Have you solve this problem?  Could you please share the solution that how to load a huge dataset for the pretraining?
Now my corpus has 10G in hundres of txt files. Now I have no idea to load these file for pretraining. 

Thanks in advance for your help.