Repository: Project-MONAI/MONAI

Issue:
MONAI dataloader does not exhibit random behaviour when using np.random
**Describe the bug**
There is a known issue with PyTorch's dataloader with >0 workers and using `np.random` to generate random numbers, described [here](https://tanelp.github.io/posts/a-bug-that-plagues-thousands-of-open-source-ml-projects/). This issue does not occur if you use `torch.randint` to generate the random numbers.

I had though MONAI's dataloader fixes this issue by setting a default `worker_init_fn`, but it seems like this isn't the case - the correct behaviour is observed when using `torch.randint` but not with `np.random`.

I understand the simple solution is to always use torch's rng, but I suspect lots of users won't know about this and will be caught out by this issue.

**To Reproduce**
Steps to reproduce the behavior:
Run this code, switching between torch/np in `__getitem__`:
```python
import numpy as np
import torch
from monai.data import Dataset, DataLoader
#from torch.utils.data import DataLoader

class RandomDataset(Dataset):
    def __getitem__(self, index):
        #return np.random.randint(0, 1000, (1,))
        return torch.randint(0, 1000, (1,))

    def __len__(self):
        return 16


dataset = RandomDataset([])
dataloader = DataLoader(dataset, batch_size=2, num_workers=4)
for epoch in range(2):
    for i, batch in enumerate(dataloader):
        print(epoch, i, batch.data.numpy().flatten().tolist())
```

**Expected behavior**
Data should be random between batches, and between epochs. This is seen using `torch`:
```python
0 0 [613, 264]
0 1 [642, 526]
0 2 [265, 338]
0 3 [988, 574]
0 4 [138, 602]
0 5 [577, 24]
0 6 [172, 986]
0 7 [902, 680]
1 0 [610, 146]
1 1 [898, 486]
1 2 [178, 408]
1 3 [679, 366]
1 4 [302, 361]
1 5 [698, 83]
1 6 [65, 102]
1 7 [615, 643]
```

but not when using `numpy`:
```python
0 0 [819, 130]
0 1 [819, 130]
0 2 [819, 130]
0 3 [819, 130]
0 4 [24, 498]
0 5 [24, 498]
0 6 [24, 498]
0 7 [24, 498]
1 0 [819, 130]
1 1 [819, 130]
1 2 [819, 130]
1 3 [819, 130]
1 4 [24, 498]
1 5 [24, 498]
1 6 [24, 498]
1 7 [24, 498]
```

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Environment**

Ensuring you use the relevant python executable, please paste the output of:

```
python -c 'import monai; monai.config.print_debug_info()'
================================
Printing MONAI config...
================================
MONAI version: 0.5.1+1.g36d855c6.dirty
Numpy version: 1.19.5
Pytorch version: 1.8.1+cu102
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: 36d855c690f03a44554f7017417cf7fdb12b8477

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 3.2.1
scikit-image version: NOT INSTALLED or UNKNOWN VERSION.
Pillow version: 8.2.0
Tensorboard version: 2.5.0
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.9.1+cu102
ITK version: NOT INSTALLED or UNKNOWN VERSION.
tqdm version: 4.60.0
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
`psutil` required for `print_system_info`

================================
Printing GPU config...
================================
Num GPUs: 3
Has CUDA: True
CUDA version: 10.2
cuDNN enabled: True
cuDNN version: 7605
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70']
GPU 0 Name: TITAN RTX
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 72
GPU 0 Total memory (GB): 23.7
GPU 0 CUDA capability (maj.min): 7.5
GPU 1 Name: Quadro RTX 8000
GPU 1 Is integrated: False
GPU 1 Is multi GPU board: False
GPU 1 Multi processor count: 72
GPU 1 Total memory (GB): 47.5
GPU 1 CUDA capability (maj.min): 7.5
GPU 2 Name: GeForce GT 1030
GPU 2 Is integrated: False
GPU 2 Is multi GPU board: False
GPU 2 Multi processor count: 3
GPU 2 Total memory (GB): 2.0
GPU 2 CUDA capability (maj.min): 6.1
```

**Additional context**
Already discussed with @ericspod who I believe has also discussed with @rijobro - thought it would be useful to have the discussion here.


Discussion:
The expectation is that the random behaviour that you would get with `torch.randint` should be matched if substituted with `np.random.randint`. This isn't the case since [set_rnd](https://github.com/Project-MONAI/MONAI/blob/5eb919869b4211b8f8146a89132127c5fb6bec89/monai/data/utils.py#L400) doesn't set numpy's random state. 

This is a problem if you're interating through data in your dataloaders in the same deterministic order for each epoch. If instead you use a randomized sampler then the chances the same random state would coincide with the same data item in a process is very low.
Also, correct me if I am wrong, but isn't this critical since MONAI's transformations are currently numpy based?
the monai loader with the `set_rnd` worker_init_fn will set all the `Randomizable` components properly https://github.com/Project-MONAI/MONAI/blob/5eb919869b4211b8f8146a89132127c5fb6bec89/monai/transforms/transform.py#L74-L78
it'll not directly modify the global random state for the `np.random` methods. I feel this is a good behaviour but happy to know any other options
@wyli maybe i've misunderstood you, but are you saying that simply inheriting `RandomDataset` from `Randomizable` should solve the problem? Because I just tried that and the behaviour persisted.
a quick example would be:
```py
import numpy as np
import torch
from monai.data import Dataset, DataLoader
from monai.transforms import Randomizable

class RandomDataset(Dataset, Randomizable):
    def __getitem__(self, index):
        return self.R.randint(0, 1000, (1,))
        # return torch.randint(0, 1000, (1,))

    def __len__(self):
        return 16


dataset = RandomDataset([])
dataloader = DataLoader(dataset, batch_size=2, num_workers=4)
for epoch in range(2):
    for i, batch in enumerate(dataloader):
        print(epoch, i, batch.data.numpy().flatten().tolist())
```
Thanks @wyli, good to know this doesn't affect MONAI's transforms.

Can I ask why MONAI relies on np for random numbers, and not torch? I notice other packages, such as torchvision and torchio, use torch's rng which means they get the correct random behaviour even with torch's default dataloader. As it stands in MONAI, for users to get the correct random behaviour they need to both use MONAI's dataloader, instead of torch's, and ensure they use self.R or torch's rng rathen than np.random in any code they write themselves. The docs don't make it clear to users they need to do things this way - for example, in the [dataloader docs](https://docs.monai.io/en/latest/data.html#dataloader) it isn't made clear that using this over torch's dataloader is necessary to get the right random behaviour. If MONAI were to rely on torch's rng internally (ie self.R used torch) then the dataloader requirement would be dropped and users would only need to use torch's rng in their code (or self.R) to get the expected behaviour, which would increase the likelihood users do the right thing by default. 
one of the motivations is to provide the flexibility of component-specific determinism without affecting the global states. the pytorch 'equivalent' would be torch.Generator but I have an impression that the API is not stable/widely supported
Definitely sounds like if we don't change the current implementation we need to **at least** improve the documentation to make it clearer to future users.
Many other libraries and examples are numpy based so we want to support that ecosystem, although some of our transforms do operate with Pytorch components such as the affine ones. We do have a mismatch between APIs if we introduce more that use Pytorch and not use their Generator class since we're setting Pytorch seed globally but Numpy seed locally with the Randomizable transform. 

Definitely we should be documenting this in the appropriate classes but people will still get caught out by this if they use numpy random functions directly. 
OK, agreed the use of np's rng makes sense for now. Here are a a couple of suggestions for places we can update the docs to help users make the right choices:

[dataloader:](https://docs.monai.io/en/latest/data.html#monai.data.DataLoader) We should say why users should prefer the MONAI dataloader over torch's. I've preferred torch's own in the past because the benefits of using MONAI's were not apparent to me.

[Randomizable](https://docs.monai.io/en/latest/_modules/monai/transforms/transform.html#Randomizable) The docs state that self.R should be used instead of np.random but again I think explaining why this is necessary would help users.
Hi @marksgraham we'd welcome a PR if you wanted to submit your suggested changes!
Hey @rijobro, sure will do. Will hopefully get round to this next week

Resolving patch:
diff --git a/monai/data/dataloader.py b/monai/data/dataloader.py
--- a/monai/data/dataloader.py
+++ b/monai/data/dataloader.py
@@ -19,20 +19,54 @@
 
 
 class DataLoader(_TorchDataLoader):
-    """Generates images/labels for train/validation/testing from dataset.
-    It inherits from PyTorch DataLoader and adds default callbacks for `collate`
-    and `worker_fn` if user doesn't set them.
+    """
+    Provides an iterable over the given `dataset`.  It inherits the PyTorch
+    DataLoader and adds enhanced `collate_fn` and `worker_fn` by default.
+
+    Although this class could be configured to be the same as
+    `torch.utils.data.DataLoader`, its default configuration is
+    recommended, mainly for the following extra features:
 
-    More information about PyTorch DataLoader, please check:
+        - It handles MONAI randomizable objects with appropriate random state
+          managements for deterministic behaviour.
+        - It is aware of the patch-based transform (such as
+          :py:class:`monai.transforms.RandSpatialCropSamplesDict`) samples for
+          preprocessing with enhanced data collating behaviour.
+          See: :py:class:`monai.transforms.Compose`.
+
+    For more details about :py:class:`torch.utils.data.DataLoader`, please see:
     https://github.com/pytorch/pytorch/blob/master/torch/utils/data/dataloader.py
 
+    For example, to construct a randomized dataset and iterate with the data loader:
+
+    .. code-block:: python
+
+        import torch
+
+        from monai.data import DataLoader
+        from monai.transforms import Randomizable
+
+
+        class RandomDataset(torch.utils.data.Dataset, Randomizable):
+            def __getitem__(self, index):
+                return self.R.randint(0, 1000, (1,))
+
+            def __len__(self):
+                return 16
+
+
+        dataset = RandomDataset()
+        dataloader = DataLoader(dataset, batch_size=2, num_workers=4)
+        for epoch in range(2):
+            for i, batch in enumerate(dataloader):
+                print(epoch, i, batch.data.numpy().flatten().tolist())
+
     Args:
         dataset: dataset from which to load the data.
         num_workers: how many subprocesses to use for data
             loading. ``0`` means that the data will be loaded in the main process.
             (default: ``0``)
         kwargs: other parameters for PyTorch DataLoader.
-
     """
 
     def __init__(self, dataset: Dataset, num_workers: int = 0, **kwargs) -> None:
diff --git a/monai/transforms/croppad/batch.py b/monai/transforms/croppad/batch.py
--- a/monai/transforms/croppad/batch.py
+++ b/monai/transforms/croppad/batch.py
@@ -58,7 +58,6 @@ class PadListDataCollate(InvertibleTransform):
     pass the inverse through multiprocessing.
 
     Args:
-        batch: batch of data to pad-collate
         method: padding method (see :py:class:`monai.transforms.SpatialPad`)
         mode: padding mode (see :py:class:`monai.transforms.SpatialPad`)
     """
@@ -72,6 +71,10 @@ def __init__(
         self.mode = mode
 
     def __call__(self, batch: Any):
+        """
+        Args:
+            batch: batch of data to pad-collate
+        """
         # data is either list of dicts or list of lists
         is_list_of_dicts = isinstance(batch[0], dict)
         # loop over items inside of each element in a batch
diff --git a/monai/transforms/transform.py b/monai/transforms/transform.py
--- a/monai/transforms/transform.py
+++ b/monai/transforms/transform.py
@@ -73,8 +73,12 @@ def _log_stats(data, prefix: Optional[str] = "Data"):
 
 class Randomizable(ABC):
     """
-    An interface for handling random state locally, currently based on a class variable `R`,
-    which is an instance of `np.random.RandomState`.
+    An interface for handling random state locally, currently based on a class
+    variable `R`, which is an instance of `np.random.RandomState`.  This
+    provides the flexibility of component-specific determinism without
+    affecting the global states.  It is recommended to use this API with
+    :py:class:`monai.data.DataLoader` for deterministic behaviour of the
+    preprocessing pipelines.
     """
 
     R: np.random.RandomState = np.random.RandomState()


Test patch:
diff --git a/tests/test_dataloader.py b/tests/test_dataloader.py
--- a/tests/test_dataloader.py
+++ b/tests/test_dataloader.py
@@ -17,7 +17,8 @@
 from parameterized import parameterized
 
 from monai.data import CacheDataset, DataLoader, Dataset
-from monai.transforms import Compose, DataStatsd, SimulateDelayd
+from monai.transforms import Compose, DataStatsd, Randomizable, SimulateDelayd
+from monai.utils import set_determinism
 
 TEST_CASE_1 = [
     [
@@ -64,5 +65,34 @@ def test_exception(self, datalist):
                 pass
 
 
+class _RandomDataset(torch.utils.data.Dataset, Randomizable):
+    def __getitem__(self, index):
+        return self.R.randint(0, 1000, (1,))
+
+    def __len__(self):
+        return 8
+
+
+class TestLoaderRandom(unittest.TestCase):
+    """
+    Testing data loader working with the randomizable interface
+    """
+
+    def setUp(self):
+        set_determinism(0)
+
+    def tearDown(self):
+        set_determinism(None)
+
+    def test_randomize(self):
+        dataset = _RandomDataset()
+        dataloader = DataLoader(dataset, batch_size=2, num_workers=3)
+        output = []
+        for _ in range(2):
+            for batch in dataloader:
+                output.extend(batch.data.numpy().flatten().tolist())
+        self.assertListEqual(output, [594, 170, 524, 778, 370, 906, 292, 589, 762, 763, 156, 886, 42, 405, 221, 166])
+
+
 if __name__ == "__main__":
     unittest.main()