Repository: chainer/chainer

Issue:
Process does not exit when CUDARuntimeError occurs
Hi,

My problem is: If I misuse MultiprocessParallelUpdater such that an exception is thrown, the python process does not exit, but just hangs.

The code to reproduce is at the bottom. It causes `cupy.cuda.runtime.CUDARuntimeError: cudaErrorInitializationError: initialization error`, but the python process does not exit.

I'd like the process to exit, but I am not sure how -- I can't use `python -m mpi4py` or call `mpi4py.MPI.COMM_WORLD.abort()` as in https://github.com/chainer/chainermn/issues/236 since this code does not use mpi4py.

Thank you.

* Conditions
  - Chainer version 4.0.0
  - CuPy version 4.0.0
  - OS/Platform ubuntu
  - CUDA/cuDNN version 9.0
* Code to reproduce

```
import chainer
import chainer.functions as F
import chainer.links as L

class MLP(chainer.Chain):

    def __init__(self, n_units, n_out):
        super(MLP, self).__init__()
        with self.init_scope():
            # the size of the inputs to each layer will be inferred
            self.l1 = L.Linear(None, n_units)  # n_in -> n_units
            self.l2 = L.Linear(None, n_units)  # n_units -> n_units
            self.l3 = L.Linear(None, n_out)  # n_units -> n_out

    def __call__(self, x):
        h1 = F.relu(self.l1(x))
        h2 = F.relu(self.l2(h1))
        return self.l3(h2)

def train():
    train, test = chainer.datasets.get_mnist()

    batch_size = 64
    learning_rate = 0.05

    model = L.Classifier(MLP(1000, 10))

    optimizer = chainer.optimizers.MomentumSGD(learning_rate)
    optimizer.setup(model)
    optimizer.add_hook(chainer.optimizer.WeightDecay(5e-4))

    # Set up a trainer
    num_gpus = 2
    devices = range(num_gpus)

    # this is just to force the error
    chainer.cuda.get_device_from_id(0).use()

    train_iters = [chainer.iterators.MultiprocessIterator(i, batch_size, n_processes=num_gpus) \
                   for i in chainer.datasets.split_dataset_n_random(train, len(devices))]

    updater = training.updaters.MultiprocessParallelUpdater(train_iters, optimizer, devices=range(num_gpus))
    updater.setup_workers()


if __name__=="__main__":
    train()
```

Stacktrace:
```
algo-1_1  | Process _Worker-1:
algo-1_1  | Traceback (most recent call last):
algo-1_1  |   File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
algo-1_1  |     self.run()
algo-1_1  |   File "/usr/local/lib/python3.5/dist-packages/chainer/training/updaters/multiprocess_parallel_updater.py", line 45, in run
algo-1_1  |     dev.use()
algo-1_1  |   File "cupy/cuda/device.pyx", line 101, in cupy.cuda.device.Device.use
algo-1_1  |   File "cupy/cuda/device.pyx", line 107, in cupy.cuda.device.Device.use
algo-1_1  |   File "cupy/cuda/runtime.pyx", line 184, in cupy.cuda.runtime.setDevice
algo-1_1  |   File "cupy/cuda/runtime.pyx", line 136, in cupy.cuda.runtime.check_status
algo-1_1  | cupy.cuda.runtime.CUDARuntimeError: cudaErrorInitializationError: initialization error

```

I had a problem similar to this one, but with mpi4py / chainermn, so the answer might be similar: https://github.com/chainer/chainermn/issues/236


Discussion:
I reproduced this problem with this Dockerfile and scripts: https://gist.github.com/mitmul/571a155d651df3b0dbe6ec129c846cb1
If we can know if the CUDA context has been initialized, this may be able to be resolved, I suppose. How can I know it?

Resolving patch:
diff --git a/chainer/training/updaters/multiprocess_parallel_updater.py b/chainer/training/updaters/multiprocess_parallel_updater.py
--- a/chainer/training/updaters/multiprocess_parallel_updater.py
+++ b/chainer/training/updaters/multiprocess_parallel_updater.py
@@ -124,6 +124,18 @@ def __init__(self, iterators, optimizer, converter=convert.concat_examples,
                 'requires NCCL.\n'
                 'Please reinstall chainer after you install NCCL.\n'
                 '(see https://github.com/chainer/chainer#installation).')
+        try:
+            cuda.cupy.cuda.driver.ctxGetCurrent()
+            _cuda_initialized = True
+        except cuda.cupy.cuda.driver.CUDADriverError:
+            # The context is not initialized, it will be fine.
+            _cuda_initialized = False
+        if _cuda_initialized:
+            raise ValueError(
+                'The CUDA context has been already initialized. '
+                'MultiprocessParallelUpdater assumes the context is '
+                'uninitialized. Please do not call CUDA API before '
+                'MultiprocessParallelUpdater creates processes.')
 
         assert len(iterators) == len(devices)
         for iterator in iterators[1:]:


Test patch: