Repository: Lightning-AI/pytorch-lightning
Pull request: Add warning for few workers
# Before submitting

- [x] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
- [x] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md), Pull Request section?
- [x] Did you make sure to update the docs?   
- [x] Did you write any new necessary tests?  
- [ ] If you made a notable change (that affects users), did you update the [CHANGELOG](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CHANGELOG.md)?

## What does this PR do?
Fixes #1322 

- adds a warning when a data loader with only a few (<= 2) workers is used. Could potentially use a higher number?  or just warn when num workers is zero?

## PR review    
Anyone in the community is free to review the PR once the tests have passed.     
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

## Did you have fun?
Make sure you had fun coding 🙃


Problem statement:
Training loop temporarily hangs after every 4 steps
I am porting some of my code to pytorch lightning, and everything seems to work fine. However, for some reason after every 4 training steps I see some temporary hanging (~1 second), which is severely slowing down my overall training time. Am I missing some obvious configuration?  This is my Trainer configuration:

```
    trainer = pl.Trainer(
        gpus=8
        num_nodes=1,
        distributed_backend='ddp',
        checkpoint_callback=False,
        max_epochs=50,
        max_steps=None,
        progress_bar_refresh_rate=1,
        check_val_every_n_epoch=1,
        val_check_interval=1.0,
        gradient_clip_val=0.0,
        log_save_interval=0,
        num_sanity_val_steps=0,
        amp_level='O0',
    )
```

Hints from issue discussion:
Hi! thanks for your contribution!, great first issue!
@PyTorchLightning/core-contributors 
Thanks for the issue! Would it be possible to post the code that reproduces this error? I've only seen this sort of behaviour before when the number of data loading workers is low - are you working with large data here (e.g. big images)?
I increased the number of workers and it works perfectly now, thank you very much! You can close this issue.
should we throw a warning when users use few workers?
If possible, sure! Seems like an obvious solution now, but it could save a couple of hours for other people. :)

Commit message: Update test

Patch to review:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c329a33e91407..5151c43121580 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -26,6 +26,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Added model configuration checking ([#1199](https://github.com/PyTorchLightning/pytorch-lightning/pull/1199))
 - Added support for optimizer frequencies through `LightningModule.configure_optimizers()` ([#1269](https://github.com/PyTorchLightning/pytorch-lightning/pull/1269))
 - Added option to run without an optimizer by returning `None` from `configure_optimizers`. ([#1279](https://github.com/PyTorchLightning/pytorch-lightning/pull/1279))
+- Added a warning when the number of data loader workers is small. ([#1378](https://github.com/PyTorchLightning/pytorch-lightning/pull/1378))
 
 ### Changed
 
diff --git a/pytorch_lightning/trainer/data_loading.py b/pytorch_lightning/trainer/data_loading.py
index fe1adf75c3fdc..51e2efaadcbc8 100644
--- a/pytorch_lightning/trainer/data_loading.py
+++ b/pytorch_lightning/trainer/data_loading.py
@@ -1,8 +1,9 @@
+import warnings
 from abc import ABC, abstractmethod
 from typing import Union, List, Tuple, Callable
 
 import torch.distributed as torch_distrib
-from torch.utils.data import SequentialSampler, DataLoader
+from torch.utils.data import DataLoader
 from torch.utils.data.distributed import DistributedSampler
 
 from pytorch_lightning.core import LightningModule
@@ -73,6 +74,14 @@ def _percent_range_check(self, name: str) -> None:
         if not 0. <= value <= 1.:
             raise ValueError(msg)
 
+    def _worker_check(self, dataloader: DataLoader, name: str) -> None:
+        msg = f'The dataloader, {name}, does not have many workers which may be a bottleneck. `' \
+              f'Consider increasing the value of the `num_workers` argument in the `' \
+              f'`DataLoader` init to improve performance.'
+
+        if isinstance(dataloader, DataLoader) and dataloader.num_workers <= 2:
+            warnings.warn(msg)
+
     def auto_add_sampler(self, dataloader: DataLoader, train: bool) -> DataLoader:
 
         # don't do anything if it's not a dataloader
@@ -112,11 +121,13 @@ def reset_train_dataloader(self, model: LightningModule) -> None:
             model: The current `LightningModule`
         """
         self.train_dataloader = self.request_dataloader(model.train_dataloader)
+
         self.num_training_batches = 0
 
         # automatically add samplers
         self.train_dataloader = self.auto_add_sampler(self.train_dataloader, train=True)
 
+        self._worker_check(self.train_dataloader, 'train dataloader')
         self._percent_range_check('train_percent_check')
 
         if not _has_len(self.train_dataloader):
@@ -176,10 +187,10 @@ def _reset_eval_dataloader(self, model: LightningModule,
         # determine number of batches
         # datasets could be none, 1 or 2+
         if len(dataloaders) != 0:
-            for dataloader in dataloaders:
+            for i, dataloader in enumerate(dataloaders):
+                self._worker_check(dataloader, f'{mode} dataloader {i}')
                 if not _has_len(dataloader):
                     num_batches = float('inf')
-                    break
 
             percent_check = getattr(self, f'{mode}_percent_check')
 
diff --git a/tests/trainer/test_dataloaders.py b/tests/trainer/test_dataloaders.py
index d0da0044f217e..408774430c398 100644
--- a/tests/trainer/test_dataloaders.py
+++ b/tests/trainer/test_dataloaders.py
@@ -15,6 +15,7 @@
     LightValStepFitMultipleDataloadersMixin,
     LightValStepFitSingleDataloaderMixin,
     LightTrainDataloader,
+    LightValidationDataloader,
     LightInfTrainDataloader,
     LightInfValDataloader,
     LightInfTestDataloader,
@@ -485,6 +486,47 @@ class CurrentTestModel(
         trainer.fit(model)
 
 
+def test_warning_with_few_workers(tmpdir):
+    """ Test that error is raised if dataloader with only a few workers is used """
+    tutils.reset_seed()
+
+    class CurrentTestModel(
+        LightTrainDataloader,
+        LightValStepFitSingleDataloaderMixin,
+        LightTestFitSingleTestDataloadersMixin,
+        LightEmptyTestStep,
+        TestModelBase,
+    ):
+        pass
+
+    hparams = tutils.get_default_hparams()
+    model = CurrentTestModel(hparams)
+
+    # logger file to get meta
+    trainer_options = dict(
+        default_save_path=tmpdir,
+        max_epochs=1,
+        val_percent_check=0.1,
+        train_percent_check=0.2
+    )
+
+    fit_options = dict(train_dataloader=model._dataloader(train=True),
+                       val_dataloaders=model._dataloader(train=False),
+                       test_dataloaders=model._dataloader(train=False))
+
+    trainer = Trainer(**trainer_options)
+
+    # fit model
+    with pytest.warns(UserWarning, match='train'):
+        trainer.fit(model, **fit_options)
+
+    with pytest.warns(UserWarning, match='val'):
+        trainer.fit(model, **fit_options)
+
+    with pytest.warns(UserWarning, match='test'):
+        trainer.test()
+
+
 @pytest.mark.skipif(torch.cuda.device_count() < 2, reason='Test requires multiple GPUs')
 def test_dataloader_reinit_for_subclass():
 


Code review comments:
diff_hunk: @@ -73,6 +74,14 @@ def _percent_range_check(self, name: str) -> None:
         if not 0. <= value <= 1.:
             raise ValueError(msg)
 
+    def _worker_check(self, dataloader: DataLoader, name: str) -> None:
+        msg = f'The dataloader, {name}, does not have many workers which may be a bottleneck. `' \
+              f'Consider increasing the value of the `num_workers` argument in the `' \
+              f'`DataLoader` init to improve performance.'
+
+        if isinstance(dataloader, DataLoader) and dataloader.num_workers <= 2:
+            warnings.warn(msg)
path: pytorch_lightning/trainer/data_loading.py
text: @user1:
```suggestion
        if isinstance(dataloader, DataLoader) and dataloader.num_workers <= 2:
            warnings.warn(f'The dataloader, {name}, does not have many workers which may be a bottleneck. `'
                          'Consider increasing the value of the `num_workers` argument in the `' \
                          `DataLoader` init to improve performance.')
```