Repository: cupy/cupy

Issue:
`SuperLU.solve` leaks memory for `trans="T"`
### Description

I'm trying to use the sparse linear solver to repeatedly solve a problem where the LHS matrix is constantly changing (and I need both it and its transpose). However, doing this seems to blow up my CUDA memory fairly quickly. If I refactor my code and change `splu(A).solve(b, trans="T")` into `splu(A.T).solve(b)` the memory stays constant throughout the program.

### To Reproduce

```py

A: cp.sparse.coo_matrix =  ... # some big sparse matrix
b = cp.asarray(torch.zeros(A.shape[0]))
for i in range(2000):
    lu = linalg.splu(A)
    x = lu.solve(b, trans="T")
```

Observe the memory through `nvidia-smi -l` until eventually getting the error `CuSparseError: CUSPARSE_STATUS_ALLOC_FAILED`.

On the other hand, this works fine:
```py

A: cp.sparse.coo_matrix =  ... # some big sparse matrix
b = cp.asarray(torch.zeros(A.shape[0]))
for i in range(2000):
    lu = linalg.splu(A.T)
    x = lu.solve(b)
```


### Installation

Conda-Forge (`conda install ...`)

### Environment

```
OS                           : Linux-5.4.0-122-generic-x86_64-with-glibc2.31
Python Version               : 3.10.5
CuPy Version                 : 11.0.0
CuPy Platform                : NVIDIA CUDA
NumPy Version                : 1.23.1
SciPy Version                : 1.9.0
Cython Build Version         : 0.29.30
Cython Runtime Version       : None
CUDA Root                    : /home/anadodik/miniconda3/envs/donut
nvcc PATH                    : None
CUDA Build Version           : 11020
CUDA Driver Version          : 11070
CUDA Runtime Version         : 11060
cuBLAS Version               : (available)
cuFFT Version                : 10600
cuRAND Version               : 10209
cuSOLVER Version             : (11, 3, 2)
cuSPARSE Version             : (available)
NVRTC Version                : (11, 6)
Thrust Version               : 101000
CUB Build Version            : 101000
Jitify Build Version         : 3c4a4ba
cuDNN Build Version          : 8401
cuDNN Version                : 8401
NCCL Build Version           : 21212
NCCL Runtime Version         : 21304
cuTENSOR Version             : 10500
cuSPARSELt Build Version     : None
Device 0 Name                : NVIDIA TITAN Xp
Device 0 Compute Capability  : 61
Device 0 PCI Bus ID          : 0000:04:00.0
```


### Additional Information

_No response_


Discussion:
Could you print out the memory usage in each iteration, just for us to get some idea?
```python
pool = cp.get_default_memory_pool()
...

for i in range(2000):
    ...
    print(pool.used_bytes())
    print(pool.total_bytes())
This sort of patterns seems to repeat indefinitely:
```
9854976
33645056
9854976
33645056
9854976
33645056
9854976
33645056
…
```

The memory used in `nvidia-smi` is going up. Does this indicate that there is some allocation happening outside of the pool or that something isn’t getting freed correctly?
I did another test where I moved `lu = linalg.splu(A)` before the loop, and the memory still blew up, indicating that `lu.solve(b, trans=“T”)` really is the culprit.
seems to be a leak in nvidia libraries and not CuPy
Hi @anadodik, thanks for testing, but I cannot reproduce it with the same CuPy and CUDA versions. Could you give me the following information?
- A's shape
- A and b's dtype
- A's nnz
Hi @leofang, I just got a chance to look at it:
 
```
print(A.shape, b.shape, A.nnz)
(11381, 11381) (11381,) 79549
```
Thanks, @anadodik. I tried, but for some reason with your size (11381, nnz=79549) I got either "RuntimeError: Factor is exactly singular" or "RuntimeError: failed to factorize matrix at line 413 in file scipy/sparse/linalg/dsolve/SuperLU/SRC/spanel_bmod.c", both raised from SciPy. May I ask which SciPy version are you using? 

btw I've tried a larger size (12000) and larger nnz (density=0.1) and it works fine, no error or leaking.
Hi @leofang, I’m on scipy 1.9.0. I guess I don’t know exactly how you are generating the matrix, these errors might be from that. Could I somehow send you the exact data I am using?
Yes a script would be the best, next is a dataset that I can load.
Great, I found an email address on your profile to send you the data. Here is the entirety of the code I’m running to reproduce the bug:

```
import cupy as cp
import cupyx.scipy.sparse.linalg as linalg
import numpy as np
import scipy.sparse

A_scipy = scipy.sparse.load_npz("A.npz")
b_scipy = np.load("b.npz")["b"]

A = cp.sparse.coo_matrix(A_scipy)
b = cp.asarray(b_scipy)

print(A.shape, b.shape, A.nnz)
pool = cp.get_default_memory_pool()

lu_t = linalg.splu(A)
for i in range(2000):
    x = lu_t.solve(b, trans="T")
    print(pool.used_bytes())
    print(pool.total_bytes())
```

The error is eventually:
```

Traceback (most recent call last):
  File "/home/anadodik/miniconda3/envs/donut/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/anadodik/miniconda3/envs/donut/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/anadodik/donut/tests/test_lu_memory.py", line 49, in <module>
    x = lu_t.solve(b, trans="T")
  File "/home/anadodik/miniconda3/envs/donut/lib/python3.10/site-packages/cupyx/scipy/sparse/linalg/_solve.py", line 558, in solve
    cusparse.csrsm2(self.U, x, lower=False, transa=trans)
  File "/home/anadodik/miniconda3/envs/donut/lib/python3.10/site-packages/cupy/cusparse.py", line 1600, in csrsm2
    analysis(handle, algo, transa, transb, m, nrhs, a.nnz, alpha.ctypes.data,
  File "cupy_backends/cuda/libs/cusparse.pyx", line 2198, in cupy_backends.cuda.libs.cusparse.scsrsm2_analysis
  File "cupy_backends/cuda/libs/cusparse.pyx", line 2210, in cupy_backends.cuda.libs.cusparse.scsrsm2_analysis
  File "cupy_backends/cuda/libs/cusparse.pyx", line 1596, in cupy_backends.cuda.libs.cusparse.check_status
cupy_backends.cuda.libs.cusparse.CuSparseError: CUSPARSE_STATUS_ALLOC_FAILED
```
Hi @leofang, just wondering if you've had a chance to look into this?
I may encounter a similar problem here:

``` python
from torch.utils.dlpack import to_dlpack
from torch.utils.dlpack import from_dlpack
from torch_sparse import spmm, transpose
import cupy
import torch

class deformation_gradient(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input, lu_solver, idxs, vals, shape):

        batch_size = input.shape[0]
        input = input.reshape(input.shape[0], -1, 3)
        input = input.transpose(1, 0)
        input = input.reshape(input.shape[0], -1) # since lu.solve only support dim (F, D), we concate all additional dimentions to the last dim

        ctx.solver = lu_solver
        ctx.idxs = idxs
        ctx.vals = vals
        ctx.shape = shape
        ctx.set_materialize_grads(False)
        # ctx.grad = from_dlpack(ctx.solver.solve(ctx.rhs.toarray()).toDlpack())
        # cupy_input = cupy.from_dlpack(to_dlpack(input))
        b = spmm(idxs, vals, m=shape[0], n=shape[1], matrix=input)
        b = cupy.from_dlpack(to_dlpack(b))
        cupy_output = ctx.solver.solve(b)
        output = from_dlpack(cupy_output.toDlpack())
        output = output.reshape(-1, batch_size, 3)
        output = output.transpose(0, 1)
        # print(f'forward time: {time.time() - t:.4f}s')
        return output
``` 

This is a custom function for pytorch . Here the `lu_solver` is a cupy superLU object, `spmm` is a sparse matrix multiplication function provided by `pytorch_sparse`.  The problematic code is `cupy_output = ctx.solver.solve(b)`. With that the GPU memory increases occasionally after several epochs of training. 

Resolving patch:
diff --git a/cupy/cusparse.py b/cupy/cusparse.py
--- a/cupy/cusparse.py
+++ b/cupy/cusparse.py
@@ -9,6 +9,7 @@
 from cupy_backends.cuda.libs import cusparse as _cusparse
 from cupy._core import _dtype
 from cupy.cuda import device as _device
+from cupy.cuda import stream as _stream
 from cupy import _util
 import cupyx.scipy.sparse
 
@@ -1605,6 +1606,10 @@ def csrsm2(a, b, alpha=1.0, lower=True, unit_diag=False, transa=False,
           a_desc.descriptor, a.data.data.ptr, a.indptr.data.ptr,
           a.indices.data.ptr, b.data.ptr, ldb, info, policy, ws.data.ptr)
 
+    # without sync we'd get either segfault or cuda context error
+    _stream.get_current_stream().synchronize()
+    _cusparse.destroyCsrsm2Info(info)
+
 
 def csrilu02(a, level_info=False):
     """Computes incomplete LU decomposition for a sparse square matrix.


Test patch: