Trainer: exception raised when calling len() on IterableDataset
Author: j-rossi-nl
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information
While pre-training a Longformer model from scratch, the text is delivered through an `IterableDataset` object. The code which is called by `Trainer.train()` still calls `len()` on this object, which raises an exception.
#5829 addressed the proper creation of the Dataloader.

The problem arises when using:
* [x] my own modified scripts: see code

The tasks I am working on is:
* [x] my own task or dataset: pre-train a LM from scratch

## To reproduce
Here is my entire code, but it can be reproduced with any `PreTrainedModel` by using an `IterableDataset`.

```python
import logging
import random

from dataclasses import dataclass, field

from transformers import LongformerConfig, LongformerForMaskedLM, LongformerTokenizerFast
from transformers import Trainer, TrainingArguments
from transformers import TextDataset, DataCollatorForLanguageModeling
from transformers import HfArgumentParser

from sklearn.model_selection import train_test_split

from pathlib import Path

from utils_pretrain import MultiTextDataset

logger = logging.getLogger(__name__)





@dataclass
class ModelArguments:
    """
    Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.
    """

    max_seq_len: int = field(
        metadata={"help": "Input Sequence Length"}
    )
    num_hidden_layers: int = field(
        metadata={'help': 'Number of transformer layers in Longformer'}
    )
    tok_dir: str = field(
        metadata={
            'help': 'Folder with tokenizer files'
        }
    )
    txt_dir: str = field(
        metadata={"help": "Folder with txt files for tokenizer training"}
    )
    filter_files: str = field(
        default='[a-c]*.txt',
        metadata={"help": "regex to select specific files"}
    )
    test_size: float = field(
        default=0.05,
        metadata={'help': 'proportion of the data that will be used for evaluation'} 
    )


def main():
    parser = HfArgumentParser((ModelArguments, TrainingArguments))
    model_args, train_args = parser.parse_args_into_dataclasses()
    model_args: ModelArguments

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.WARN,
    )
    logger.warning(
        "Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s",
        train_args.local_rank,
        train_args.device,
        train_args.n_gpu,
        bool(train_args.local_rank != -1),
        train_args.fp16,
    )
    logger.info("Training/evaluation parameters %s", train_args)

    MODEL_NAME = 'allenai/longformer-base-4096'
    tokenizer: LongformerTokenizerFast = LongformerTokenizerFast.from_pretrained(model_args.tok_dir)

    # Customize an existing config rather than create from scratch
    config: LongformerConfig = LongformerConfig.from_pretrained(MODEL_NAME)
    config.max_position_embeddings = model_args.max_seq_len + 2
    config.num_hidden_layers = model_args.num_hidden_layers
    config.attention_window = [512] * model_args.num_hidden_layers
    config.vocab_size = tokenizer.vocab_size
    model = LongformerForMaskedLM(config)

    data_files = list(Path(model_args.txt_dir).glob(model_args.filter_files))
    shuffled_files = random.sample(data_files, len(data_files))
    train_files, val_files = train_test_split(shuffled_files, test_size=model_args.test_size)

    train_ds, val_ds = list(
        map(
            lambda x: MultiTextDataset(
                files=x,
                tokenizer=tokenizer,
                block_size=model_args.max_seq_len
            ),
            [train_files, val_files]
        )
    )

    data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer,
        mlm=True,
        mlm_probability=0.15
    )

    train_args: TrainingArguments
    train_args.do_train = True
    train_args.evaluate_during_training = True

    trainer = Trainer(
        model=model,
        args=train_args,
        data_collator=data_collator,
        train_dataset=train_ds,
        eval_dataset=val_ds,
    )

    trainer.train(train_args.output_dir)
```

The class `MultiTextDataset` inherits `IterableDataset`. It has no `__len__` method, and the length would require the whole dataset to be parsed at once to be known.

Here is the exception and stack trace:
```
Traceback (most recent call last):
  File "longformer_pretrain.py", line 131, in <module>
    main()
  File "longformer_pretrain.py", line 122, in main
    trainer.train(train_args.output_dir)
  File "/home/jrossi/anaconda3/envs/COLIEE/lib/python3.7/site-packages/transformers/trainer.py", line 392, in train
    self.args.max_steps // (len(train_dataloader) // self.args.gradient_accumulation_steps) + 1
  File "/home/jrossi/anaconda3/envs/COLIEE/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 313, in __len__
    length = self._IterableDataset_len_called = len(self.dataset)
TypeError: object of type 'MultiTextDataset' has no len()
```

## Expected behavior

The call to `Trainer.train()` starts the training. A case has to be made in the code to accomodate the usage of `IterableDataset`, which means not assuming that `len()` can be called on the dataset at any point.

- If a number of epochs is given, one epoch corresponds to consuming the iterable dataset until StopIteration
- If a number of steps is given, training stops after performing MAX_STEPS or catching a StopIteration, whichever comes first
- During training, the progress bar should be either a % of epochs performed, or a % of steps performed
- (optional) If a number of epochs is given, register how many steps it took to consume the iterator so a better progress bar can be shown for the next epochs (each epoch will consume the same iterator once)

With regards to [Pytorch documentation](https://pytorch.org/docs/stable/data.html#), there is no certainty that `__len__` method will be implemented, even on `Dataset` objects. 
The distinction should be made between objects that implement `__len__` and those that do not implement it.
The current code __assumes__ that the `Dataset` objects given when creating a `Trainer` implement `len()`, but there is no guarantee of this.

```python
import collections
if isinstance(bar, collections.Sized): (...)
```

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-5.7.8-1.el7.elrepo.x86_64-x86_64-with-centos-7.8.2003-Core
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO (for the moment)

## Fix
I can contribute. I will suggest a PR to fix this.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.