Repository: cupy/cupy

Issue:
TypeError for OutOfMemoryError
Seen while using chainer while multiprocessing and using the GPU:
```
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 463, in _handle_results
    task = get()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "cupy/cuda/memory.pyx", line 37, in cupy.cuda.memory.OutOfMemoryError.__init__
TypeError: __init__() takes exactly 3 positional arguments (2 given)
```
Seems like it tried to raise an OutOfMemoryError but failed to do so.

```
CuPy Version          : 6.1.0
CUDA Root             : /usr/local/cuda
CUDA Build Version    : 10010
CUDA Driver Version   : 10010
CUDA Runtime Version  : 10010
cuDNN Build Version   : 7500
cuDNN Version         : 7500
NCCL Build Version    : 2402
NCCL Runtime Version  : 2402
```


Discussion:
Thanks for raising the issue! We recently received the similar report and looking into it. https://stackoverflow.com/questions/56753355/in-windows-envs-cupy-get-error-in-multithread-pool-if-gpu-already-used-even-if

* What platform are you on? (Ubuntu, Windows, etc)
* Did you installed CuPy using wheel (cupy-cudaXXX), or built from source? In the latter case, what is your Cython version?
Ubuntu 18.04, installed using wheel (cupy-cuda101). NVIDIA GTX 1060 with 3GB memory. Like the stackoverflow issue, I am using 'spawn' launch method, though I think I saw the same error when using fork; I thought using spawn would help resolve it.

Output of `nvidia-smi`:
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  On   | 00000000:01:00.0 Off |                  N/A |
|  0%   43C    P2    27W / 120W |   1401MiB /  3018MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       896      G   /usr/lib/xorg/Xorg                           120MiB |
|    0     25629      C   /usr/bin/python3                             531MiB |
|    0     25631      C   /usr/bin/python3                             745MiB |
+-----------------------------------------------------------------------------+
```
That is *not* while running the reproduction case, and therefore I just noticed that somehow some python processes stuck around claiming memory after the fact. Soooo, huh. Just gonna `killall python3` real quick.

If it helps, I am calling into [waifu2x-chainer](https://github.com/tsurumeso/waifu2x-chainer) and doing a lot of image scaling and processing.

I don't have a *minimized* reproduction case, but I can reproduce the error pretty consistently; and it is code that I planned on making public soon. So if you need a reproduction case let me know and I can push it up. That is if you don't mind dealing with some specific project upscaling Doom sprites.

I also saw that if you catch the exception in the process and print it, it seems fine, insofar as it prints the right exception anyway:
`out of memory to allocate 50466816 bytes (total 223895040 bytes)`
The exception problem only occurs if you let it bubble up out of the process.

There are really two issues:
1. An out of memory exception occurs at all. But that might just be because I am flying too close to the sun trying to batch process too many things at once. Even so I had hoped it would block until there was memory available or something.
2. The exception itself is not handled well when multiprocessing.
@kcghost Thanks for the information! I could reproduce the error; the root cause was that some exceptions defined in CuPy does not support pickle protocol and thus exceptions raised in a subprocess (which is pickled in the subprocess) fail to be un-pickled in the parent process.

Resolving patch:
diff --git a/cupy/cuda/compiler.py b/cupy/cuda/compiler.py
--- a/cupy/cuda/compiler.py
+++ b/cupy/cuda/compiler.py
@@ -193,6 +193,10 @@ def __init__(self, msg, source, name, options):
         self.source = source
         self.name = name
         self.options = options
+        super(CompileException, self).__init__()
+
+    def __reduce__(self):
+        return (type(self), (self._msg, self.source, self.name, self.options))
 
     def __repr__(self):
         return str(self)


Test patch:
diff --git a/tests/cupy_tests/cuda_tests/test_compiler.py b/tests/cupy_tests/cuda_tests/test_compiler.py
--- a/tests/cupy_tests/cuda_tests/test_compiler.py
+++ b/tests/cupy_tests/cuda_tests/test_compiler.py
@@ -1,3 +1,4 @@
+import pickle
 import unittest
 
 import mock
@@ -92,3 +93,12 @@ def test_symbol(self):
 
     def test_space(self):
         self.assertFalse(compiler.is_valid_kernel_name('invalid name'))
+
+
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = compiler.CompileException('msg', 'fn.cu', 'fn', ('-ftz=true',))
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_cublas.py b/tests/cupy_tests/cuda_tests/test_cublas.py
new file mode 100644
--- /dev/null
+++ b/tests/cupy_tests/cuda_tests/test_cublas.py
@@ -0,0 +1,13 @@
+import pickle
+import unittest
+
+from cupy.cuda import cublas
+
+
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = cublas.CUBLASError(1)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_cudnn.py b/tests/cupy_tests/cuda_tests/test_cudnn.py
new file mode 100644
--- /dev/null
+++ b/tests/cupy_tests/cuda_tests/test_cudnn.py
@@ -0,0 +1,18 @@
+import pickle
+import unittest
+
+try:
+    from cupy.cuda import cudnn
+    cudnn_available = True
+except Exception:
+    cudnn_available = False
+
+
+@unittest.skipUnless(cudnn_available, 'cuDNN is unavailable')
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = cudnn.CuDNNError(1)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_cufft.py b/tests/cupy_tests/cuda_tests/test_cufft.py
new file mode 100644
--- /dev/null
+++ b/tests/cupy_tests/cuda_tests/test_cufft.py
@@ -0,0 +1,13 @@
+import pickle
+import unittest
+
+from cupy.cuda import cufft
+
+
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = cufft.CuFFTError(1)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_curand.py b/tests/cupy_tests/cuda_tests/test_curand.py
--- a/tests/cupy_tests/cuda_tests/test_curand.py
+++ b/tests/cupy_tests/cuda_tests/test_curand.py
@@ -1,3 +1,4 @@
+import pickle
 import unittest
 
 import numpy
@@ -35,3 +36,12 @@ def test_invalid_argument_log_normal_double(self):
         with self.assertRaises(ValueError):
             curand.generateLogNormalDouble(
                 self.generator, out.data.ptr, 1, 0.0, 1.0)
+
+
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = curand.CURANDError(100)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_cusolver.py b/tests/cupy_tests/cuda_tests/test_cusolver.py
--- a/tests/cupy_tests/cuda_tests/test_cusolver.py
+++ b/tests/cupy_tests/cuda_tests/test_cusolver.py
@@ -1,3 +1,4 @@
+import pickle
 import unittest
 
 from cupy import cuda
@@ -8,3 +9,13 @@ class TestCusolver(unittest.TestCase):
     def test_cusolver_enabled(self):
         self.assertEqual(cuda.runtime.runtimeGetVersion() >= 8000,
                          cuda.cusolver_enabled)
+
+
+@unittest.skipUnless(cuda.cusolver_enabled, 'cuSOLVER is unavailable')
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = cuda.cusolver.CUSOLVERError(1)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_cusparse.py b/tests/cupy_tests/cuda_tests/test_cusparse.py
new file mode 100644
--- /dev/null
+++ b/tests/cupy_tests/cuda_tests/test_cusparse.py
@@ -0,0 +1,13 @@
+import pickle
+import unittest
+
+from cupy.cuda import cusparse
+
+
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = cusparse.CuSparseError(1)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_cutensor.py b/tests/cupy_tests/cuda_tests/test_cutensor.py
new file mode 100644
--- /dev/null
+++ b/tests/cupy_tests/cuda_tests/test_cutensor.py
@@ -0,0 +1,18 @@
+import pickle
+import unittest
+
+try:
+    from cupy.cuda import cutensor
+    cutensor_available = True
+except Exception:
+    cutensor_available = False
+
+
+@unittest.skipUnless(cutensor_available, 'cuTensor is unavailable')
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = cutensor.CuTensorError(1)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_driver.py b/tests/cupy_tests/cuda_tests/test_driver.py
--- a/tests/cupy_tests/cuda_tests/test_driver.py
+++ b/tests/cupy_tests/cuda_tests/test_driver.py
@@ -1,3 +1,4 @@
+import pickle
 import threading
 import unittest
 
@@ -34,3 +35,12 @@ def f(self):
         # After the context is created, it should return the valid
         # context pointer.
         self.assertNotEqual(0, self._result1)
+
+
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = driver.CUDADriverError(1)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_memory.py b/tests/cupy_tests/cuda_tests/test_memory.py
--- a/tests/cupy_tests/cuda_tests/test_memory.py
+++ b/tests/cupy_tests/cuda_tests/test_memory.py
@@ -1,5 +1,6 @@
 import ctypes
 import gc
+import pickle
 import sys
 import threading
 import unittest
@@ -699,3 +700,12 @@ def test(self):
             lock.acquire()
         assert gc.isenabled()
         self.assertRaises(Exception, lock.release)
+
+
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = memory.OutOfMemoryError(124, 1024, 1024)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_nccl.py b/tests/cupy_tests/cuda_tests/test_nccl.py
--- a/tests/cupy_tests/cuda_tests/test_nccl.py
+++ b/tests/cupy_tests/cuda_tests/test_nccl.py
@@ -1,3 +1,4 @@
+import pickle
 import unittest
 
 from cupy import cuda
@@ -30,3 +31,13 @@ def test_check_async_error(self):
         comm = cuda.nccl.NcclCommunicator(1, id, 0)
         comm.check_async_error()
         comm.destroy()
+
+
+@unittest.skipUnless(cuda.nccl_enabled, 'nccl is not installed')
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = cuda.nccl.NcclError(1)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_nvrtc.py b/tests/cupy_tests/cuda_tests/test_nvrtc.py
new file mode 100644
--- /dev/null
+++ b/tests/cupy_tests/cuda_tests/test_nvrtc.py
@@ -0,0 +1,13 @@
+import pickle
+import unittest
+
+from cupy.cuda import nvrtc
+
+
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = nvrtc.NVRTCError(1)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)
diff --git a/tests/cupy_tests/cuda_tests/test_runtime.pyx b/tests/cupy_tests/cuda_tests/test_runtime.pyx
new file mode 100644
--- /dev/null
+++ b/tests/cupy_tests/cuda_tests/test_runtime.pyx
@@ -0,0 +1,13 @@
+import pickle
+import unittest
+
+from cupy.cuda import runtime
+
+
+class TestExceptionPicklable(unittest.TestCase):
+
+    def test(self):
+        e1 = runtime.CUDARuntimeError(1)
+        e2 = pickle.loads(pickle.dumps(e1))
+        assert e1.args == e2.args
+        assert str(e1) == str(e2)