Difference in support for IterableDataset in Trainer.fit and Trainer.test
❓ Questions and Help
What is your question?
When using WebDataset (which implements IterableDataset), Trainer.fit works without warnings, but Trainer.test will fail unless a nominal length is specified. Is this intended behaviour?
What have you tried?
When running Trainer.test without specifiying nominal length:
Traceback (most recent call last):
  File "sunset.py", line 222, in <module>
    main(parser.parse_args())
  File "sunset.py", line 126, in main
    main_test(args)
  File "sunset.py", line 210, in main_test
    trainer.test(model, test_dataloaders=test_loader, verbose=True)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 1346, in test
    if test_dataloaders and datamodule:
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 325, in __len__
    length = self._IterableDataset_len_called = len(self.dataset)
TypeError: 'NoneType' object cannot be interpreted as an integer

When running Trainer.test after specifying nominal length (and having multiple workers):
/usr/local/lib/python3.6/dist-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: Your `IterableDataset` has `__len__` defined. In combination with multi-processing data loading (e.g. batch size > 1), this can lead to unintended side effects since the samples will be duplicated.
  warnings.warn(*args, **kwargs)

What's your environment?

OS: Ubuntu 18.04 (inside Docker Container)
Packaging: pip
Version: 0.9.0 on Python 3.6.9