Repository: huggingface/transformers

Issue:
AttributeError when using custom IterableDataset with set_epoch method
## Environment info
- `transformers` version: 4.8.2
- Platform: Linux-5.4.0-77-generic-x86_64-with-glibc2.31
- Python version: 3.9.4
- PyTorch version (GPU?): 1.9.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: kind of

### Who can help
@sgugger since it concerns trainer and/or documentation.

## Information

Model and task do not matter here.

I am using a custom `IterableDataset` for evaluation which I want to use for distributed training at some point. My development machine is CPU-only, so I am not using distributed training there. Following the [trainer documentation](https://huggingface.co/transformers/main_classes/trainer.html) I added a `set_epoch()` to this `IterableDataset`.
With transformers 4.8.2 and pytorch 1.9.0, evaluation reproducibly fails with an `AttributeError`.

## To reproduce

Steps to reproduce the behavior:

1. Run this minimum working example on CPU:
    ```python
    from datasets import load_dataset
    from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer,\
        TrainingArguments, IntervalStrategy
    
    from torch.utils.data.dataset import IterableDataset
    
    raw_datasets = load_dataset("imdb")
    tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
    model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)
    
    def tokenize_function(examples):
        return tokenizer(examples["text"], padding="max_length", truncation=True)
    
    tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
    small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
    small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(2))
    
    class CustomDataset(IterableDataset):
        def __iter__(self):
            yield from small_eval_dataset
    
        def __len__(self):
            return 2
    
        def set_epoch(self, epoch: int):
            pass
    
    custom_eval = CustomDataset()
    training_args = TrainingArguments(output_dir="test_trainer",
                                      evaluation_strategy=IntervalStrategy.STEPS,
                                      logging_steps=1,
                                      per_device_train_batch_size=1,
                                      per_device_eval_batch_size=1)
    trainer = Trainer(model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=custom_eval)
    trainer.train()
    ```
2. Stacktrace:
    ```
    Traceback (most recent call last):
      File "/home/mbugert/.../problem.py", line 42, in <module>
        trainer.train()
      File "/home/mbugert/.local/bin/pyenv/versions/huggingface-bug-3.9.4/lib/python3.9/site-packages/transformers/trainer.py", line 1325, in train
        self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
      File "/home/mbugert/.local/bin/pyenv/versions/huggingface-bug-3.9.4/lib/python3.9/site-packages/transformers/trainer.py", line 1426, in _maybe_log_save_evaluate
        metrics = self.evaluate()
      File "/home/mbugert/.local/bin/pyenv/versions/huggingface-bug-3.9.4/lib/python3.9/site-packages/transformers/trainer.py", line 2024, in evaluate
        output = eval_loop(
      File "/home/mbugert/.local/bin/pyenv/versions/huggingface-bug-3.9.4/lib/python3.9/site-packages/transformers/trainer.py", line 2245, in evaluation_loop
        num_samples = eval_dataset.num_examples
      File "/home/mbugert/.local/bin/pyenv/versions/huggingface-bug-3.9.4/lib/python3.9/site-packages/torch/utils/data/dataset.py", line 163, in __getattr__
        raise AttributeError
    AttributeError
   ```

## Expected behavior

Evaluation succeeds.

## Possible cause
I looked into the problem a bit myself but got stuck:
* The same code runs fine with transformers 4.6.1 and pytorch 1.8.1, so it's likely that the issue is related to the `Dataset` typing changes introduced in pytorch 1.9.0.
* The problem arises here: https://github.com/huggingface/transformers/blob/e52288a140489b486a7a436f8d25d7dea66d0a62/src/transformers/trainer.py#L2242-L2247
  `isinstance(eval_dataset, IterableDatasetShard)` returns `True` despite the facts that training isn't distributed and `eval_dataset` is of type `CustomDataset`.
* Debugging revealed that the `isinstance` call leads to `typing._ProtocolMeta.__instancecheck__` where some funky runtime typecheck is performed, which turns out `True` because `CustomDataset` has the all the attributes and methods which `IterableDatasetShard` has. Hence, if one comments out `set_epoch` in `CustomDataset` (or if one adds a `foo()` method to `IterableDatasetShard`), the code runs through.

At this point I wasn't sure anymore if this is a bug in transformers, a bug in pytorch or just user error because I didn't understand how `set_epoch` is supposed to be defined. Please let me know what you think. :smile: 


Discussion:

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -2245,7 +2245,9 @@ def evaluation_loop(
         # Number of samples
         if not isinstance(eval_dataset, IterableDataset):
             num_samples = len(eval_dataset)
-        elif isinstance(eval_dataset, IterableDatasetShard):
+        # The instance check is weird and does not actually check for the type, but whether the dataset has the right
+        # methods. Therefore we need to make sure it also has the attribute.
+        elif isinstance(eval_dataset, IterableDatasetShard) and hasattr(eval_dataset, "num_examples"):
             num_samples = eval_dataset.num_examples
         else:
             num_samples = observed_num_examples


Test patch: