Repository: awslabs/gluonts

Issue:
Multiprocessing hangs when num_workers > len(dataset)
## Description
I'm trying to serialize a predictor trained on multiple cores. When calling the `serialize` method nothing happens.
Running the same code, but without specifying `num_workers`, it works as expected.

## To Reproduce

```python
from pathlib import Path
from typing import Optional

from gluonts.dataset.multivariate_grouper import MultivariateGrouper
from gluonts.dataset.common import TrainDatasets
from gluonts.model.gpvar import GPVAREstimator
from gluonts.dataset.repository.datasets import get_dataset
from gluonts.mx.trainer import Trainer


def load_multivariate_dataset(dataset_name: str, target_dim: Optional[int] = None):
    ds = get_dataset(dataset_name)

    if target_dim is None:
        target_dim = len(ds.train)

    grouper = MultivariateGrouper(max_target_dim=target_dim)

    meta = ds.metadata
    meta.feat_static_cat[0].cardinality = target_dim

    return (TrainDatasets(
        metadata=meta,
        train=grouper(ds.train),
        test=grouper(ds.test)
    ), target_dim)


ds, target_dim = load_multivariate_dataset("exchange_rate")
metadata = ds.metadata

estimator = GPVAREstimator(
    prediction_length=metadata.prediction_length,
    freq=metadata.freq,
    target_dim=target_dim,
    trainer=Trainer(
        epochs=2,
        num_batches_per_epoch=10,
        batch_size=8,
    ),
)

predictor = estimator.train(training_data=ds.train, num_workers=2)

predictor.serialize(Path("/tmp"))

```

## Error message or code output
Nothing happens.


## Environment
- Operating system: Mac OSX 10.15.7
- Python version: 3.6.12
- GluonTS version: 0.6.0
- MXNet version: 1.7.0post1




Discussion:
@kaijennissen I'm not able to reproduce this: the following snippet, adapted from yours, doesn't raise an error

(I modified the path where the model is serialized, and added a test in the end to check if the reconstructed model is the same)

*Edit:* I'm using `master` with `mxnet==1.7.0post1`

```python
from pathlib import Path
from typing import Optional
import tempfile

from gluonts.dataset.multivariate_grouper import MultivariateGrouper
from gluonts.dataset.common import TrainDatasets
from gluonts.model.predictor import Predictor
from gluonts.model.gpvar import GPVAREstimator
from gluonts.dataset.repository.datasets import get_dataset
from gluonts.mx.trainer import Trainer


def load_multivariate_dataset(dataset_name: str, target_dim: Optional[int] = None):
    ds = get_dataset(dataset_name)

    if target_dim is None:
        target_dim = len(ds.train)

    grouper = MultivariateGrouper(max_target_dim=target_dim)

    meta = ds.metadata
    meta.feat_static_cat[0].cardinality = target_dim

    return (TrainDatasets(
        metadata=meta,
        train=grouper(ds.train),
        test=grouper(ds.test)
    ), target_dim)


ds, target_dim = load_multivariate_dataset("exchange_rate")
metadata = ds.metadata

estimator = GPVAREstimator(
    prediction_length=metadata.prediction_length,
    freq=metadata.freq,
    target_dim=target_dim,
    trainer=Trainer(
        epochs=2,
        num_batches_per_epoch=10,
        batch_size=8,
    ),
)

predictor = estimator.train(training_data=ds.train, num_workers=2)

with tempfile.TemporaryDirectory() as path_str:
    print(f"saving model in {path_str}")
    predictor.serialize(Path(path_str))
    predictor_copy = Predictor.deserialize(Path(path_str))

assert predictor == predictor_copy
```
I just tried it on gpu with

```python
import multiprocessing
multiprocessing.set_start_method("spawn", force=True)
```
at the beginning of the script. The training takes approx. 4 minutes to start.
On CPU it works, but the CPU usage of Python goes up to 99% after the code finishes until I restart the kernel. I think there are some processes which do not get terminated. I think this is related to #941.
> On CPU it works, but the CPU usage of Python goes up to 99% after the code finishes until I restart the kernel. I think there are some processes which do not get terminated. I think this is related to #941.

You' re right.
I've modified the last part and it appears that the problem is not the serialization but that some processes are not terminated.
It also occurs inside a `docker` container, with `Gitpod` (using `master)` and on a EC2 DLAMI instance (using the `mxnet_latest_p37` conda env and `pip install gluonts==0.6`).

```python

print("Finished serialization!")
assert predictor == predictor_copy
```

```
learning rate from ``lr_scheduler`` has been overwritten by ``learning_rate`` in optimizer.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:25<00:00,  2.52s/it, epoch=1/2, avg_epoch_loss=1.31]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:18<00:00,  1.90s/it, epoch=2/2, avg_epoch_loss=-.802]
saving model in /tmp/tmpz684afay
WARNING:root:Serializing RepresentableBlockPredictor instances does not save the prediction network structure in a backwards-compatible manner. Be careful not to use this method in production.
Finished serialization!
```

Looks like the `MultivariateGrouper` doesn't play well with multiprocessing: the following example works fine with `use_grouper = False` (where we construct a multivariate dataset manually), but hangs with `use_grouper = True` (where we group an originally univariate dataset to get a multivariate one). 

```python
import pandas as pd
import numpy as np

from gluonts.dataset.multivariate_grouper import MultivariateGrouper
from gluonts.model.gpvar import GPVAREstimator
from gluonts.mx.trainer import Trainer

use_grouper = False

if use_grouper:
    dataset = [
        {"start": pd.Timestamp("2020-01-01 00:00:00", freq="1H"), "target": np.array([1.0]*1000)}
        for _ in range(8)
    ]
    grouper = MultivariateGrouper(max_target_dim=8)
    multivariate_dataset = grouper(dataset)
else:
    multivariate_dataset = [
        {"start": pd.Timestamp("2020-01-01 00:00:00", freq="1H"), "target": np.array([[1.0]*1000] * 8)}
    ]

estimator = GPVAREstimator(
    prediction_length=24,
    freq="1H",
    target_dim=8,
    trainer=Trainer(
        epochs=2,
        num_batches_per_epoch=10,
        batch_size=8,
    ),
)

predictor = estimator.train(training_data=multivariate_dataset, num_workers=2)
```
@kaijennissen @PascalIversen the problem appears to be `num_workers > len(dataset)`, I reduced the issue to the following minimal example (univariate with DeepAR, but I also checked multivariate with GPVAR):

```python
from gluonts.dataset.common import ListDataset
from gluonts.model.deepar import DeepAREstimator
from gluonts.mx.trainer import Trainer

univariate_dataset = ListDataset(
    data_iter=[
        {"start": "2020-01-01 00:00:00", "target": [1.0] * 1000}
    ],
    freq="1H",
)

estimator = DeepAREstimator(
    prediction_length=24,
    freq="1H",
    trainer=Trainer(
        epochs=2,
        num_batches_per_epoch=10,
        batch_size=8,
    ),
)

predictor = estimator.train(training_data=univariate_dataset, num_workers=2)
```
@lostella Okay. To summarize this. 
For a univariate datasets with multiple entries, one can speed up training by setting `num_workers>1`.
The only way to reduce training time in case of a multivariate dataset (which will be of length 1) is to use a `GPU`.
Correct?
This works :
```python

from gluonts.model.deepar import DeepAREstimator
from gluonts.mx.trainer import Trainer
import pandas as pd
data_iter=[
    {"start": pd.Timestamp("1990-01-01 00:00:00", freq="1H"), "target": [1.0] * 1000}
]
estimator = DeepAREstimator(
    prediction_length=24,
    freq="1H",
    trainer=Trainer(
        epochs=2,
        num_batches_per_epoch=10,
        batch_size=8,
    ),
)

predictor = estimator.train(training_data=data_iter, num_workers=2)
```
Maybe the problem is the ListDataset?



This works for as-well (only difference is the `list()` in the last line)

```python
from pathlib import Path
from typing import Optional

from gluonts.dataset.multivariate_grouper import MultivariateGrouper
from gluonts.dataset.common import TrainDatasets
from gluonts.model.gpvar import GPVAREstimator
from gluonts.dataset.repository.datasets import get_dataset
from gluonts.mx.trainer import Trainer


def load_multivariate_dataset(dataset_name: str, target_dim: Optional[int] = None):
    ds = get_dataset(dataset_name)

    if target_dim is None:
        target_dim = len(ds.train)

    grouper = MultivariateGrouper(max_target_dim=target_dim)

    meta = ds.metadata
    meta.feat_static_cat[0].cardinality = target_dim

    return (TrainDatasets(
        metadata=meta,
        train=grouper(ds.train),
        test=grouper(ds.test)
    ), target_dim)


ds, target_dim = load_multivariate_dataset("exchange_rate")
metadata = ds.metadata

estimator = GPVAREstimator(
    prediction_length=metadata.prediction_length,
    freq=metadata.freq,
    target_dim=target_dim,
    trainer=Trainer(
        epochs=2,
        num_batches_per_epoch=10,
        batch_size=8,
    ),
)

predictor = estimator.train(training_data=list(ds.train), num_workers=2)
```

However the speedup on CPU compared to `num_workers=None` is negligible. My intuition is that data-loading is not really the bottleneck on CPU.
> For a univariate datasets with multiple entries, one can speed up training by setting num_workers>1.

That's correct. Whether you will observe any speedup depends on what the bottleneck is: if data loading is the bottleneck (because the lazy transformations involved there) then multiple workers will help.
There might be other ways of speeding things up in that case, like caching some of the transformations (instead of doing them again and again) but that's not exposed yet as an option.

> The only way to reduce training time in case of a multivariate dataset (which will be of length 1) is to use a GPU.

That's definitely one thing to try, but it may not help with all models. CNN-based models will benefit from it, and also dense layers should operate faster on GPU, but with RNN-based models one may not observe significant speedup. I'm not sure what's the case for GPVAR.

All of this said, we should keep this issue open and fix the way data loader workers terminate in this (not so) edge case. Thanks for opening this @kaijennissen!!
@PascalIversen yes, the issue is in the `ListDataset` (and might be in `FileDataset` too) and concerns what happens if some worker doesn't get assigned any slice of the dataset, which is the case when num_workers > 1 and len(dataset) == 1

I'm not sure what happens with multiple workers when a `list` is used directly, but I suspect that all workers just iterate the entire list, so you can have as many workers as you like and they won't be bothered. *Edit:* of course this is also not the intended behavior, but outside of the scope here.


Resolving patch:
diff --git a/src/gluonts/itertools.py b/src/gluonts/itertools.py
--- a/src/gluonts/itertools.py
+++ b/src/gluonts/itertools.py
@@ -21,8 +21,13 @@
 def cyclic(it):
     """Like `itertools.cycle`, but does not store the data."""
 
+    at_least_one = False
     while True:
-        yield from it
+        for el in it:
+            at_least_one = True
+            yield el
+        if not at_least_one:
+            break
 
 
 def batcher(iterable: Iterable[T], batch_size: int) -> Iterator[List[T]]:


Test patch:
diff --git a/test/test_itertools.py b/test/test_itertools.py
--- a/test/test_itertools.py
+++ b/test/test_itertools.py
@@ -11,12 +11,22 @@
 # express or implied. See the License for the specific language governing
 # permissions and limitations under the License.
 
-from typing import Iterable
+from typing import Iterable, List
+import itertools
 
 import pytest
 
 from gluonts.dataset.artificial import constant_dataset
-from gluonts.itertools import pseudo_shuffled
+from gluonts.itertools import pseudo_shuffled, cyclic
+
+
+@pytest.mark.parametrize(
+    "data, n, expected", [([1, 2, 3], 7, [1, 2, 3, 1, 2, 3, 1]), ([], 4, [])]
+)
+def test_cyclic(data: Iterable, n: int, expected: List) -> None:
+    cyclic_data = cyclic(data)
+    actual = list(itertools.islice(cyclic_data, n))
+    assert actual == expected
 
 
 @pytest.mark.parametrize("data", [range(20), constant_dataset()[1],])