Repository: Lightning-AI/pytorch-lightning

Issue:
_has_len does not handle NotImplementedError (raised by torchtext)
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug
When using torchtext.data.Iterator with a batch_size_fn function the __len__  function raises a  NotImplementedError which is not caught by _has_len function.

A bug-fix is **very simple** by just returning False if a NotImplementedError is raised. This is unlikely to have any negative side effects since it corresponds with what _hads_len is expected to do. The fix allowed me to train my model using torch text.

I plan to submit a pull request with the fix above.

There are no additional dependencies required; however this problem occurred when using torchtext.

Example stack trace:
```
Traceback (most recent call last):
  File "/Users/thomas/scm/OakDataPrep/oakSkipThoughtTrainer.py", line 18, in <module>
    trainer.fit(model)
  File "/Users/thomas/virtualenv/Python3/PyTorch/env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 952, in fit
    self.run_pretrain_routine(model)
  File "/Users/thomas/virtualenv/Python3/PyTorch/env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1091, in run_pretrain_routine
    self.train()
  File "/Users/thomas/virtualenv/Python3/PyTorch/env/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 334, in train
    self.reset_train_dataloader(model)
  File "/Users/thomas/virtualenv/Python3/PyTorch/env/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py", line 201, in reset_train_dataloader
    if not _has_len(self.train_dataloader):
  File "/Users/thomas/virtualenv/Python3/PyTorch/env/lib/python3.7/site-packages/pytorch_lightning/trainer/data_loading.py", line 49, in _has_len
    if len(dataloader) == 0:
  File "/Users/thomas/virtualenv/Python3/PyTorch/env/lib/python3.7/site-packages/torchtext/data/iterator.py", line 136, in __len__
    raise NotImplementedError
NotImplementedError
```

### To Reproduce

Sorry I currently don't have a minimal example. The issue will always occur when torchtext.data.Iterator gets a batch_size_fn passed in. If the fix is not convincing I can take the time and construct a code example. Hope this is not necessary.

#### Code sample

I created my own Iterator for a Skip-Thought model, that dynamically batches sentences together. This might be unnecessary complex, or even not really useful however it revealed that issue described above when using torchtext. For context here is a code excerpt that creates the issue:
```
import torchtext
...
global max_src_in_batch, max_tgt_in_batch
def batch_size_fn(new, count, sofar):
    "Keep augmenting batch and calculate total number of tokens + padding."
    global max_src_in_batch, max_tgt_in_batch
    if count == 1:
        max_src_in_batch = 0
        max_tgt_in_batch = 0
    max_src_in_batch = max(max_src_in_batch,  len(new.current))
    max_tgt_in_batch = max(max_tgt_in_batch,  len(new.next) + 2)
    src_elements = count * max_src_in_batch
    tgt_elements = count * max_tgt_in_batch
    return max(src_elements, tgt_elements)

class MyIterator(torchtext.data.Iterator):

    def create_batches(self):
        if self.train:
            def pool(d, random_shuffler):
                for p in data.batch(d, self.batch_size * 100):
                    p_batch = data.batch(
                            sorted(p, key=self.sort_key),
                            self.batch_size, self.batch_size_fn)
                    for b in random_shuffler(list(p_batch)):
                        yield b
            self.batches = pool(self.data(), self.random_shuffler)

        else:
            self.batches = []
            for b in data.batch(self.data(), self.batch_size,
                                self.batch_size_fn):
                self.batches.append(sorted(b, key=self.sort_key))

...
class SkipThoughts(pl.LightningModule):
...
    @pl.data_loader
    def train_dataloader(self):
        train_iter = MyIterator(self.my_train_dataloader, batch_size=self.batch_size, repeat=False,
                                   sort_key=lambda x:
                                   data.interleave_keys(len(x.current),
                                                        data.interleave_keys(len(x.prev),
                                                                             len(x.next))),
                                   batch_size_fn=batch_size_fn, train=True,
                                   shuffle=True)
        return train_iter
```
But this happens whenever a batch_size_fn is used in torchtext. Because it is unknown how many batches the data set will have torchtext __len__ method returns a NotImplementedError. See code snipped below:
```
def __len__(self):
    if self.batch_size_fn is not None:
        raise NotImplementedError
    return math.ceil(len(self.dataset) / self.batch_size)
```
### Expected behavior

The function _has_len tests if len can is available and then returns True, otherwise False. It shoudl return False if NotImplementedError is raised.

### Environment

/Users/thomas/virtualenv/Python3/PyTorch/env/bin/python /Users/thomas/scm/OakDataPrep/collect_env_details.py
* CUDA:
	- GPU:
	- available:         False
	- version:           None
* Packages:
	- numpy:             1.18.2
	- pyTorch_debug:     False
	- pyTorch_version:   1.5.0
	- pytorch-lightning: 0.8.0
	- tensorboard:       2.2.0
	- tqdm:              4.45.0
* System:
	- OS:                Darwin
	- architecture:
		- 64bit
		- 
	- processor:         i386
	- python:            3.7.7
	- version:           Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64

Process finished with exit code 0

### Additional context

Issue occur with Pytorch-Lighning 0.8 and Torchtext 0.6

<!-- Add any other context about the problem here. -->



Discussion:

Resolving patch:
diff --git a/pytorch_lightning/trainer/data_loading.py b/pytorch_lightning/trainer/data_loading.py
--- a/pytorch_lightning/trainer/data_loading.py
+++ b/pytorch_lightning/trainer/data_loading.py
@@ -52,6 +52,8 @@ def _has_len(dataloader: DataLoader) -> bool:
         return True
     except TypeError:
         return False
+    except NotImplementedError:  # e.g. raised by torchtext if a batch_size_fn is used
+        return False
 
 
 class TrainerDataLoadingMixin(ABC):


Test patch:
diff --git a/tests/base/dataloaders.py b/tests/base/dataloaders.py
--- a/tests/base/dataloaders.py
+++ b/tests/base/dataloaders.py
@@ -21,3 +21,29 @@ def __next__(self):
         except StopIteration:
             self.iter = iter(self.dataloader)
             return next(self.iter)
+
+
+class CustomNotImplementedErrorDataloader:
+
+    def __init__(self, dataloader):
+        self.dataloader = dataloader
+        self.iter = iter(dataloader)
+        self.count = 0
+
+    def __len__(self):
+        """raise NotImplementedError"""
+        raise NotImplementedError
+
+    def __iter__(self):
+        self.count = 0
+        return self
+
+    def __next__(self):
+        if self.count >= 50:
+            raise StopIteration
+        self.count = self.count + 1
+        try:
+            return next(self.iter)
+        except StopIteration:
+            self.iter = iter(self.dataloader)
+            return next(self.iter)
diff --git a/tests/base/model_test_dataloaders.py b/tests/base/model_test_dataloaders.py
--- a/tests/base/model_test_dataloaders.py
+++ b/tests/base/model_test_dataloaders.py
@@ -1,6 +1,7 @@
 from abc import ABC, abstractmethod
 
 from tests.base.dataloaders import CustomInfDataloader
+from tests.base.dataloaders import CustomNotImplementedErrorDataloader
 
 
 class TestDataloaderVariations(ABC):
@@ -15,6 +16,9 @@ def test_dataloader(self):
     def test_dataloader__infinite(self):
         return CustomInfDataloader(self.dataloader(train=False))
 
+    def test_dataloader__not_implemented_error(self):
+        return CustomNotImplementedErrorDataloader(self.dataloader(train=False))
+
     def test_dataloader__empty(self):
         return None
 
diff --git a/tests/base/model_train_dataloaders.py b/tests/base/model_train_dataloaders.py
--- a/tests/base/model_train_dataloaders.py
+++ b/tests/base/model_train_dataloaders.py
@@ -1,6 +1,7 @@
 from abc import ABC, abstractmethod
 
 from tests.base.dataloaders import CustomInfDataloader
+from tests.base.dataloaders import CustomNotImplementedErrorDataloader
 
 
 class TrainDataloaderVariations(ABC):
@@ -15,6 +16,9 @@ def train_dataloader(self):
     def train_dataloader__infinite(self):
         return CustomInfDataloader(self.dataloader(train=True))
 
+    def train_dataloader__not_implemented_error(self):
+        return CustomNotImplementedErrorDataloader(self.dataloader(train=True))
+
     def train_dataloader__zero_length(self):
         dataloader = self.dataloader(train=True)
         dataloader.dataset.data = dataloader.dataset.data[:0]
diff --git a/tests/base/model_valid_dataloaders.py b/tests/base/model_valid_dataloaders.py
--- a/tests/base/model_valid_dataloaders.py
+++ b/tests/base/model_valid_dataloaders.py
@@ -1,6 +1,7 @@
 from abc import ABC, abstractmethod
 
 from tests.base.dataloaders import CustomInfDataloader
+from tests.base.dataloaders import CustomNotImplementedErrorDataloader
 
 
 class ValDataloaderVariations(ABC):
@@ -18,3 +19,6 @@ def val_dataloader__multiple(self):
 
     def val_dataloader__infinite(self):
         return CustomInfDataloader(self.dataloader(train=False))
+
+    def val_dataloader__not_implemented_error(self):
+        return CustomNotImplementedErrorDataloader(self.dataloader(train=False))
diff --git a/tests/trainer/test_dataloaders.py b/tests/trainer/test_dataloaders.py
--- a/tests/trainer/test_dataloaders.py
+++ b/tests/trainer/test_dataloaders.py
@@ -295,6 +295,18 @@ def test_train_inf_dataloader_error(tmpdir):
         trainer.fit(model)
 
 
+@pytest.mark.skip('TODO: speed up this test')
+def test_train_not_implemented_error_dataloader_error(tmpdir):
+    """Test not_implemented_error train data loader (e.g. IterableDataset)"""
+    model = EvalModelTemplate()
+    model.train_dataloader = model.train_dataloader__not_implemented_error
+
+    trainer = Trainer(default_root_dir=tmpdir, max_epochs=1, val_check_interval=0.5)
+
+    with pytest.raises(MisconfigurationException, match='not_implemented_error DataLoader'):
+        trainer.fit(model)
+
+
 @pytest.mark.skip('TODO: speed up this test')
 def test_val_inf_dataloader_error(tmpdir):
     """Test inf train data loader (e.g. IterableDataset)"""
@@ -307,6 +319,18 @@ def test_val_inf_dataloader_error(tmpdir):
         trainer.fit(model)
 
 
+@pytest.mark.skip('TODO: speed up this test')
+def test_val_not_implemented_error_dataloader_error(tmpdir):
+    """Test not_implemented_error train data loader (e.g. IterableDataset)"""
+    model = EvalModelTemplate()
+    model.val_dataloader = model.val_dataloader__not_implemented_error
+
+    trainer = Trainer(default_root_dir=tmpdir, max_epochs=1, limit_val_batches=0.5)
+
+    with pytest.raises(MisconfigurationException, match='not_implemented_error DataLoader'):
+        trainer.fit(model)
+
+
 @pytest.mark.skip('TODO: speed up this test')
 def test_test_inf_dataloader_error(tmpdir):
     """Test inf train data loader (e.g. IterableDataset)"""
@@ -319,6 +343,18 @@ def test_test_inf_dataloader_error(tmpdir):
         trainer.test(model)
 
 
+@pytest.mark.skip('TODO: speed up this test')
+def test_test_not_implemented_error_dataloader_error(tmpdir):
+    """Test not_implemented_error train data loader (e.g. IterableDataset)"""
+    model = EvalModelTemplate()
+    model.test_dataloader = model.test_dataloader__not_implemented_error
+
+    trainer = Trainer(default_root_dir=tmpdir, max_epochs=1, limit_test_batches=0.5)
+
+    with pytest.raises(MisconfigurationException, match='not_implemented_error DataLoader'):
+        trainer.test(model)
+
+
 @pytest.mark.parametrize('check_interval', [50, 1.0])
 @pytest.mark.skip('TODO: speed up this test')
 def test_inf_train_dataloader(tmpdir, check_interval):
@@ -337,6 +373,24 @@ def test_inf_train_dataloader(tmpdir, check_interval):
     assert result == 1
 
 
+@pytest.mark.parametrize('check_interval', [50, 1.0])
+@pytest.mark.skip('TODO: speed up this test')
+def test_not_implemented_error_train_dataloader(tmpdir, check_interval):
+    """Test not_implemented_error train data loader (e.g. IterableDataset)"""
+
+    model = EvalModelTemplate()
+    model.train_dataloader = model.train_dataloader__not_implemented_error
+
+    trainer = Trainer(
+        default_root_dir=tmpdir,
+        max_epochs=1,
+        val_check_interval=check_interval
+    )
+    result = trainer.fit(model)
+    # verify training completed
+    assert result == 1
+
+
 @pytest.mark.parametrize('check_interval', [1.0])
 @pytest.mark.skip('TODO: speed up this test')
 def test_inf_val_dataloader(tmpdir, check_interval):
@@ -357,6 +411,26 @@ def test_inf_val_dataloader(tmpdir, check_interval):
     assert result == 1
 
 
+@pytest.mark.parametrize('check_interval', [1.0])
+@pytest.mark.skip('TODO: speed up this test')
+def test_not_implemented_error_dataloader(tmpdir, check_interval):
+    """Test not_implemented_error data loader (e.g. IterableDataset)"""
+
+    model = EvalModelTemplate()
+    model.val_dataloader = model.val_dataloader__not_implemented_error
+
+    # logger file to get meta
+    trainer = Trainer(
+        default_root_dir=tmpdir,
+        max_epochs=1,
+        val_check_interval=check_interval,
+    )
+    result = trainer.fit(model)
+
+    # verify training completed
+    assert result == 1
+
+
 def test_error_on_zero_len_dataloader(tmpdir):
     """ Test that error is raised if a zero-length dataloader is defined """