Repository: cupy/cupy

Issue:
v7.4.0 cupy/cuda/driver.pyx error line 118
Hi,

I'm working in conda envs with conda installs. Hit a snag upgrading from cupy 6.0.0 to 7.4.0 with rapidsai.

The MRE runs in cupy 6.0.0 and crashes in 7.4.0 with this error:

```
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable
```

In Jupyter Notebook the MRE errors one line later:

```
CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
```

The complete stack trace is in the attached notebook along with additional system and device specs.

Great tool box, thanks.

Tom

---

[cupy740_crash_mre.ipynb.pdf](https://github.com/cupy/cupy/files/4602529/cupy740_crash_mre.ipynb.pdf)

```
# MRE
import numpy as np
import cupy as cp

MB = 1024**2
cp.cuda.Device(3).use()

free, total = cp.cuda.Device(3).mem_info
print(f"MB free {free / MB :.0f} total {total / MB :.0f}")

# ok on these ...
# n, p, g = 250, 5, 10
# n, p, g = 2500, 25, 1000

# errors on these
n, p, g = 25000, 250, 10000

yg = np.random.rand(n, g).astype("float32")
X = np.random.rand(n, p).astype("float32")


ygd = cp.asarray(yg)
Xd = cp.asarray(X)
print(f"MB matrices: {(ygd.nbytes + Xd.nbytes) / MB :.0f}")
assert ygd.nbytes + Xd.nbytes < free 

Qd, Rd = cp.linalg.qr(Xd)
bhatsd = cp.linalg.solve(Rd, Qd.T @ ygd)
yhatsd = Xd @ bhatsd  # jupyter gets past this line

# ed = yhatsd - ygd   # jupyter errors on this line
```


[sandbox]$ conda activate cupy
(cupy) [sandbox]$ python --version; python -c "import cupy; cupy.show_config()"; python cupy740_crash_mre.py
Python 3.7.7
CuPy Version          : 6.0.0
CUDA Root             : /usr/local/cuda-8.0
CUDA Build Version    : 10000
CUDA Driver Version   : 10020
CUDA Runtime Version  : 10000
cuDNN Build Version   : 7301
cuDNN Version         : 7605
NCCL Build Version    : 1000
NCCL Runtime Version  : (unknown)
MB free 12039 total 12196
MB matrices: 978
(cupy) [sandbox]$ conda deactivate
[sandbox]$ conda activate rapidsai37
(rapidsai37) [sandbox]$ python --version; python -c "import cupy; cupy.show_config()"; python cupy740_crash_mre.py
Python 3.7.6
CuPy Version          : 7.4.0
CUDA Root             : /home/turbach/.conda/envs/rapidsai37
CUDA Build Version    : 10020
CUDA Driver Version   : 10020
CUDA Runtime Version  : 10020
cuBLAS Version        : 10202
cuFFT Version         : 10102
cuRAND Version        : 10102
cuSOLVER Version      : (10, 3, 0)
cuSPARSE Version      : 10301
NVRTC Version         : (10, 2)
cuDNN Build Version   : 7605
cuDNN Version         : 7605
NCCL Build Version    : 2406
NCCL Runtime Version  : 2507
MB free 12027 total 12196
MB matrices: 978
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable
(rapidsai37) [sandbox]$ 



Discussion:
Seems like it is an error doing the cleanup?

We have hit some of these before, they should not affect your computations but we should fix this too.
It's really odd...I can't reproduce this with v7.4.0 from conda-forge. @emcastillo can you?

By the way, just curious (I don't think this is relevant):
```
CuPy Version : 6.0.0
CUDA Root : /usr/local/cuda-8.0
CUDA Build Version : 10000
CUDA Driver Version : 10020
CUDA Runtime Version : 10000
```
How is this combination possible? You run with CUDA 8.0, but the runtime detection is 10.0?!
OK looks like something is odd with CUDA 10.1/10.2. I can reproduce this with the following conda environment:
```bash
conda create -n CF_cupy_test python=3.7 cupy cudatoolkit=10.2 -c conda-forge
conda activate CF_cupy_test
python cupy740_crash_mre.py
```
but not with `cudatoolkit=10.0`.
Seems to be on `cupy.cuda.function.Module.__dealloc__`?
maybe it is like the bug we had before on cleanup
Yes, I think it's a cleanup bug. Just don't know what exactly triggered it, and why no one had complained until now 😂 

Perhaps we need to list a few more common infrastructural objects in `cupy.cuda` and apply the fix to each of them in case we miss something?
I will look at it later and try to fix it the same way we did before.
> Yes, I think it's a cleanup bug. 

I take it back, sorry @emcastillo. This is the full output I got (this time with `cudatoolkit=10.1`):
```
$ python test_module_cleanup_bug.py 
CuPy Version          : 7.4.0
CUDA Root             : /home/leofang/miniconda3/envs/CF_cupy_test
CUDA Build Version    : 10010
CUDA Driver Version   : 10020
CUDA Runtime Version  : 10010
cuBLAS Version        : 10201
cuFFT Version         : 10101
cuRAND Version        : 10101
cuSOLVER Version      : (10, 2, 0)
cuSPARSE Version      : 10300
NVRTC Version         : (10, 1)
cuDNN Build Version   : 7605
cuDNN Version         : 7605
NCCL Build Version    : 2406
NCCL Runtime Version  : 2604
None
MB free 10526 total 11019
MB matrices: 978
Traceback (most recent call last):
  File "test_module_cleanup_bug.py", line 33, in <module>
    ed = yhatsd - ygd   # jupyter errors on this line
  File "cupy/core/core.pyx", line 974, in cupy.core.core.ndarray.__sub__
  File "cupy/core/_kernel.pyx", line 951, in cupy.core._kernel.ufunc.__call__
  File "cupy/core/_kernel.pyx", line 974, in cupy.core._kernel.ufunc._get_ufunc_kernel
  File "cupy/core/_kernel.pyx", line 714, in cupy.core._kernel._get_ufunc_kernel
  File "cupy/core/_kernel.pyx", line 61, in cupy.core._kernel._get_simple_elementwise_kernel
  File "cupy/core/carray.pxi", line 194, in cupy.core.core.compile_with_cache
  File "/home/leofang/miniconda3/envs/CF_cupy_test/lib/python3.7/site-packages/cupy/cuda/compiler.py", line 287, in compile_with_cache
    extra_source, backend)
  File "/home/leofang/miniconda3/envs/CF_cupy_test/lib/python3.7/site-packages/cupy/cuda/compiler.py", line 372, in _compile_with_cache_cuda
    mod.load(cubin)
  File "cupy/cuda/function.pyx", line 197, in cupy.cuda.function.Module.load
  File "cupy/cuda/function.pyx", line 199, in cupy.cuda.function.Module.load
  File "cupy/cuda/driver.pyx", line 240, in cupy.cuda.driver.moduleLoadData
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
cupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
  File "cupy/cuda/driver.pyx", line 247, in cupy.cuda.driver.moduleUnload
  File "cupy/cuda/driver.pyx", line 118, in cupy.cuda.driver.check_status
TypeError: 'NoneType' object is not callable
```
Note that the earliest error is `mod.load`, so it seems not due to cleanup but something else...?
OK so CUDA 9.2 & 10.0 do not trigger this, but 10.1 & 10.2 do. Perhaps we hit another "massive array" bug in cuSOLVER (as in #3127)?! 
could be ...
I can't reproduce wtih 10.2
```
CuPy Version          : 8.0.0b2
CUDA Root             : /usr/local/cuda
CUDA Build Version    : 10020
CUDA Driver Version   : 10020
CUDA Runtime Version  : 10020
cuBLAS Version        : 10202
cuFFT Version         : 10102
cuRAND Version        : 10102
cuSOLVER Version      : (10, 3, 0)
cuSPARSE Version      : 10301
NVRTC Version         : (10, 2)
cuDNN Build Version   : 7500
cuDNN Version         : 7500
NCCL Build Version    : None
NCCL Runtime Version  : None
```
@emcastillo Have you uncommented the last line `ed = yhatsd - ygd`?
@turbach what's the GPU model you are using?

Resolving patch:
diff --git a/cupy/linalg/solve.py b/cupy/linalg/solve.py
--- a/cupy/linalg/solve.py
+++ b/cupy/linalg/solve.py
@@ -80,24 +80,28 @@ def _solve(a, b, cublas_handle, cusolver_handle):
         geqrf = cusolver.sgeqrf
         geqrf_bufferSize = cusolver.sgeqrf_bufferSize
         ormqr = cusolver.sormqr
+        ormqr_bufferSize = cusolver.sormqr_bufferSize
         trans = cublas.CUBLAS_OP_T
         trsm = cublas.strsm
     elif dtype == 'd':
         geqrf = cusolver.dgeqrf
         geqrf_bufferSize = cusolver.dgeqrf_bufferSize
         ormqr = cusolver.dormqr
+        ormqr_bufferSize = cusolver.dormqr_bufferSize
         trans = cublas.CUBLAS_OP_T
         trsm = cublas.dtrsm
     elif dtype == 'F':
         geqrf = cusolver.cgeqrf
         geqrf_bufferSize = cusolver.cgeqrf_bufferSize
         ormqr = cusolver.cormqr
+        ormqr_bufferSize = cusolver.cunmqr_bufferSize
         trans = cublas.CUBLAS_OP_C
         trsm = cublas.ctrsm
     elif dtype == 'D':
         geqrf = cusolver.zgeqrf
         geqrf_bufferSize = cusolver.zgeqrf_bufferSize
         ormqr = cusolver.zormqr
+        ormqr_bufferSize = cusolver.zunmqr_bufferSize
         trans = cublas.CUBLAS_OP_C
         trsm = cublas.ztrsm
     else:
@@ -112,8 +116,13 @@ def _solve(a, b, cublas_handle, cusolver_handle):
         buffersize, dev_info.data.ptr)
     cupy.linalg.util._check_cusolver_dev_info_if_synchronization_allowed(
         geqrf, dev_info)
-
+    # Explicitly free the space allocated by geqrf
+    del workspace
     # 2. ormqr (Q^T * B)
+    buffersize = ormqr_bufferSize(
+        cusolver_handle, cublas.CUBLAS_SIDE_LEFT, trans, m, k, m, a.data.ptr,
+        m, tau.data.ptr, b.data.ptr, m)
+    workspace = cupy.empty(buffersize, dtype=dtype)
     ormqr(
         cusolver_handle, cublas.CUBLAS_SIDE_LEFT, trans, m, k, m, a.data.ptr,
         m, tau.data.ptr, b.data.ptr, m, workspace.data.ptr, buffersize,
@@ -121,6 +130,8 @@ def _solve(a, b, cublas_handle, cusolver_handle):
     cupy.linalg.util._check_cusolver_dev_info_if_synchronization_allowed(
         ormqr, dev_info)
 
+    # Explicitly free the space allocated by ormqr
+    del workspace
     # 3. trsm (X = R^{-1} * (Q^T * B))
     trsm(
         cublas_handle, cublas.CUBLAS_SIDE_LEFT, cublas.CUBLAS_FILL_MODE_UPPER,


Test patch: