Repository: cupy/cupy

Issue:
Additional CUDA libraries troubleshoot on Windows
I am sorry for this basic question, but I read the docs and installed cupy via pip (I used wheel) and the latest version of CUDA. Everything works well, but I am not sure how to install the additional CUDA libraries (cuTENSOR, NCCL and cuDNN). The latter two, the nvidia CUDA installation said were installed as well. But they don't seem to be found by cupy (see below)? Furthermore, I downloaded cuTENSOR, but I couldn't find how to install this as well on Windows.

Thanks!

* Conditions (you can just paste the output of `python -c 'import cupy; cupy.show_config()'`)
OS                           : Windows-10-10.0.19041-SP0
CuPy Version                 : 8.6.0
NumPy Version                : 1.19.2
SciPy Version                : 1.6.2
Cython Build Version         : 0.29.22
CUDA Root                    : C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2
CUDA Build Version           : 11020
CUDA Driver Version          : 11020
CUDA Runtime Version         : 11020
cuBLAS Version               : 11401
cuFFT Version                : 10401
cuRAND Version               : 10203
cuSOLVER Version             : (11, 1, 0)
cuSPARSE Version             : 11401
NVRTC Version                : (11, 2)
Thrust Version               : 101000
CUB Build Version            : 101000
cuDNN Build Version          : None
cuDNN Version                : None
NCCL Build Version           : None
NCCL Runtime Version         : None
cuTENSOR Version             : None
Device 0 Name                : GeForce GTX 980 Ti
Device 0 Compute Capability  : 52



Discussion:
Those libraries are used in compile time, you need to have them installed before compiling the cupy wheel in case you only did `pip install cupy`, and exposed them through the CPATH, LD_LIBRARY_PATH, [CUTENSOR_PATH](https://docs.cupy.dev/en/stable/reference/environment.html#for-installation) and other relevant env vars.

Alternatively, if you used a binary package such as `cupy-cuda112` you can use the `cupyx/tools/install_library.py` utility to install these in an easy way, as the wheels have been precompiled against specific versions of these libraries.
https://github.com/cupy/cupy/blob/master/cupyx/tools/install_library.py
In CuPy v8 wheels,
* (For wheels for CUDA 10.1 or later) cuDNN is not bundled as a wheel so you need to install it manually using the tool as commented by Emilio. `import cupy.cuda.cudnn` will show you the detailed command line.
* cuTENSOR build is disabled in wheels. You need to build from source instead of using a wheel. cuTENSOR support is enabled by default in CuPy v9+ wheels.

Also note that NCCL is unavailable on Windows.
Thanks for your responses!
First, I have installed cudnn via the detailed command line. This did work. And the `np.cuda.cudnn.available` results in True (after I load it with `import cupy.cuda.cudnn`). However, if I try to run `python -c "import cupy; cupy.show_config()"` I get the following error: 
`Could not load library cudnn_ops_infer64_8.dll. Error code 126
Please make sure cudnn_ops_infer64_8.dll is in your library path!`

Lastly, I tried installing cuTENSOR via the install script, but that is not possible (it only allows cudnn). So, I installed the pre release version of CuPy 9.0.0 and then I was able to install cuTENSOR via the install_library script. However, I am still getting the above error when trying to load the config. Also, the `available` property seems to return True for both cuTENSOR and cuDNN, which is great. Is there another way to check whether the libraries are correctly installed (maybe via a real test)? At last, when trying to run `cupy.show_config()` on Jupyter Notebook, the block of code never seems to finish, while that wasn't the case with the command in CMD (that one returned the error above).
@WoutervDijk I could reproduce that on my Windows environment (not reproducible with Linux). It seems cuDNN v8 cannot be loaded on Windows via our wheel preloading system (for both CuPy v8 and v9). I'll take care of this later.
I think you can workaround this issue by manually downloading & extracting cuDNN v8.1 from NVIDIA and add cuDNN's `bin` directory to the `PATH` environment variable.

Resolving patch:
diff --git a/cupy/_environment.py b/cupy/_environment.py
--- a/cupy/_environment.py
+++ b/cupy/_environment.py
@@ -315,8 +315,14 @@ def _preload_libraries():
                 _log('Rejected candidate (not found): {}'.format(libpath))
                 continue
 
-            _log('Trying to load {}'.format(libpath))
             try:
+                if sys.platform == 'win32':
+                    # This is needed to load cuDNN v8 on Windows.
+                    libpath_dir = os.path.dirname(libpath)
+                    _log(f'Adding to PATH: {libpath_dir}')
+                    os.environ['PATH'] = (libpath_dir + os.pathsep +
+                                          os.environ.get('PATH', ''))
+                _log(f'Trying to load {libpath}')
                 # Keep reference to the preloaded module.
                 _preload_libs[lib] = (libpath, ctypes.CDLL(libpath))
                 _log('Loaded')


Test patch: