Repository: Project-MONAI/MONAI

Issue:
`CacheDataset` gets stuck when initializing cache content
**Is your feature request related to a problem? Please describe.**
Feedback from Andriy and Daguang when using NVIDIA Clara MMARs:
one peculiar issue if we mount dataset from NGC, then training gets stuck in 90% of times (both train.sh and train_multi_gpu.sh)
I believe it has something to do with SmartCache (since it’s the only MMAR that uses it) and ngc-dataset interoperability. 
It says : “Loading Data: 0%” and just gets stuck, sometimes it goes to “Loading Data: 6%” and then gets stuck.
Is it something we should be worried about?
 
PS: If I copy data to local DGX folder first, then training runs fine.


Discussion:
After changing the "num_init_workers" of `SmartCacheDataset` to 0 or 1, everything goes fine.
I think we may need to consider changing the default value of this parameter from max CPU threads to 1 or 0, too many threads may not help improve the performance in many cases.
@wyli @ericspod Do you have any concerns?

Thanks in advance.

Thanks, perhaps could @yiheng-wang-nv help create a PR and address this one?
Hi @Nic-Ma , I'm not sure if changing the default value of `num_init_workers` can totally solve the issue.

The title of this ticket is for `CacheDataset `, thus if using `CacheDataset` and `num_workers = None`, will the same issue be raised?
In addition, will the `num_replace_workers` also have some impacts when replacing the cache?
Hi @yiheng-wang-nv ,

I would suggest change the default value of `num_workers` of all the subclasses of `CacheDataset` to 1.
Please also change the logic of `num_replace_workers`.

Thanks.


Resolving patch:
diff --git a/monai/apps/datasets.py b/monai/apps/datasets.py
--- a/monai/apps/datasets.py
+++ b/monai/apps/datasets.py
@@ -50,7 +50,8 @@ class MedNISTDataset(Randomizable, CacheDataset):
         cache_rate: percentage of cached data in total, default is 1.0 (cache all).
             will take the minimum of (cache_num, data_length x cache_rate, data_length).
         num_workers: the number of worker threads to use.
-            if 0 a single thread will be used. Default is 0.
+            If num_workers is None then the number returned by os.cpu_count() is used.
+            If a value less than 1 is speficied, 1 will be used instead.
         progress: whether to display a progress bar when downloading dataset and computing the transform cache content.
         copy_cache: whether to `deepcopy` the cache content before applying the random transforms,
             default to `True`. if the random transforms don't modify the cached content
@@ -82,7 +83,7 @@ def __init__(
         test_frac: float = 0.1,
         cache_num: int = sys.maxsize,
         cache_rate: float = 1.0,
-        num_workers: int = 0,
+        num_workers: Optional[int] = 1,
         progress: bool = True,
         copy_cache: bool = True,
         as_contiguous: bool = True,
@@ -202,7 +203,8 @@ class DecathlonDataset(Randomizable, CacheDataset):
         cache_rate: percentage of cached data in total, default is 1.0 (cache all).
             will take the minimum of (cache_num, data_length x cache_rate, data_length).
         num_workers: the number of worker threads to use.
-            if 0 a single thread will be used. Default is 0.
+            If num_workers is None then the number returned by os.cpu_count() is used.
+            If a value less than 1 is speficied, 1 will be used instead.
         progress: whether to display a progress bar when downloading dataset and computing the transform cache content.
         copy_cache: whether to `deepcopy` the cache content before applying the random transforms,
             default to `True`. if the random transforms don't modify the cached content
@@ -274,7 +276,7 @@ def __init__(
         val_frac: float = 0.2,
         cache_num: int = sys.maxsize,
         cache_rate: float = 1.0,
-        num_workers: int = 0,
+        num_workers: int = 1,
         progress: bool = True,
         copy_cache: bool = True,
         as_contiguous: bool = True,
diff --git a/monai/apps/pathology/data/datasets.py b/monai/apps/pathology/data/datasets.py
--- a/monai/apps/pathology/data/datasets.py
+++ b/monai/apps/pathology/data/datasets.py
@@ -119,8 +119,10 @@ class SmartCachePatchWSIDataset(SmartCacheDataset):
             will take the minimum of (cache_num, data_length x cache_rate, data_length).
         num_init_workers: the number of worker threads to initialize the cache for first epoch.
             If num_init_workers is None then the number returned by os.cpu_count() is used.
+            If a value less than 1 is speficied, 1 will be used instead.
         num_replace_workers: the number of worker threads to prepare the replacement cache for every epoch.
             If num_replace_workers is None then the number returned by os.cpu_count() is used.
+            If a value less than 1 is speficied, 1 will be used instead.
         progress: whether to display a progress bar when caching for the first epoch.
         copy_cache: whether to `deepcopy` the cache content before applying the random transforms,
             default to `True`. if the random transforms don't modify the cache content
@@ -142,8 +144,8 @@ def __init__(
         replace_rate: float = 0.5,
         cache_num: int = sys.maxsize,
         cache_rate: float = 1.0,
-        num_init_workers: Optional[int] = None,
-        num_replace_workers: Optional[int] = None,
+        num_init_workers: Optional[int] = 1,
+        num_replace_workers: Optional[int] = 1,
         progress: bool = True,
         copy_cache: bool = True,
         as_contiguous: bool = True,
diff --git a/monai/data/dataset.py b/monai/data/dataset.py
--- a/monai/data/dataset.py
+++ b/monai/data/dataset.py
@@ -672,7 +672,7 @@ def __init__(
         transform: Union[Sequence[Callable], Callable],
         cache_num: int = sys.maxsize,
         cache_rate: float = 1.0,
-        num_workers: Optional[int] = None,
+        num_workers: Optional[int] = 1,
         progress: bool = True,
         copy_cache: bool = True,
         as_contiguous: bool = True,
@@ -687,8 +687,9 @@ def __init__(
                 will take the minimum of (cache_num, data_length x cache_rate, data_length).
             cache_rate: percentage of cached data in total, default is 1.0 (cache all).
                 will take the minimum of (cache_num, data_length x cache_rate, data_length).
-            num_workers: the number of worker processes to use.
+            num_workers: the number of worker threads to use.
                 If num_workers is None then the number returned by os.cpu_count() is used.
+                If a value less than 1 is speficied, 1 will be used instead.
             progress: whether to display a progress bar.
             copy_cache: whether to `deepcopy` the cache content before applying the random transforms,
                 default to `True`. if the random transforms don't modify the cached content
@@ -862,8 +863,10 @@ class SmartCacheDataset(Randomizable, CacheDataset):
             will take the minimum of (cache_num, data_length x cache_rate, data_length).
         num_init_workers: the number of worker threads to initialize the cache for first epoch.
             If num_init_workers is None then the number returned by os.cpu_count() is used.
+            If a value less than 1 is speficied, 1 will be used instead.
         num_replace_workers: the number of worker threads to prepare the replacement cache for every epoch.
             If num_replace_workers is None then the number returned by os.cpu_count() is used.
+            If a value less than 1 is speficied, 1 will be used instead.
         progress: whether to display a progress bar when caching for the first epoch.
         shuffle: whether to shuffle the whole data list before preparing the cache content for first epoch.
             it will not modify the original input data sequence in-place.
@@ -884,8 +887,8 @@ def __init__(
         replace_rate: float,
         cache_num: int = sys.maxsize,
         cache_rate: float = 1.0,
-        num_init_workers: Optional[int] = None,
-        num_replace_workers: Optional[int] = None,
+        num_init_workers: Optional[int] = 1,
+        num_replace_workers: Optional[int] = 1,
         progress: bool = True,
         shuffle: bool = True,
         seed: int = 0,


Test patch: