Repository: numba/numba

Issue:
Docs should be updated for CUDA 12
The [current installation docs](https://numba.pydata.org/numba-doc/latest/cuda/overview.html#software) only mention `cudatoolkit` should be installed. However, CUDA 12 changes the way packages are organized and Numba should now require `cuda-nvcc=12.x` and I believe `cuda-version=12.x` too. Particularly, without `cuda-nvcc=12.x` installed, one might encounter errors such as:

```python
Traceback (most recent call last):
  File "<string>", line 7, in <module>
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/runtime.py", line 111, in get_version
    self.cudaRuntimeGetVersion(ctypes.byref(rtver))
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/runtime.py", line 65, in __getattr__
    self._initialize()
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/runtime.py", line 51, in _initialize
    self.lib = open_cudalib('cudart')
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/libs.py", line 63, in open_cudalib
    path = get_cudalib(lib)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/libs.py", line 55, in get_cudalib
    libdir = get_cuda_paths()[dir_type].info
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cuda_paths.py", line 223, in get_cuda_paths
    'nvvm': _get_nvvm_path(),
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cuda_paths.py", line 201, in _get_nvvm_path
    candidates = find_lib('nvvm', path)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/misc/findlib.py", line 44, in find_lib
    return find_file(regex, libdir)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/misc/findlib.py", line 56, in find_file
    entries = os.listdir(ldir)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/envs/test/nvvm/lib64'
```

cc @jakirkham @gmarkall 


Discussion:
I think we want `cuda-nvcc` and `cuda-version=12.*` for CUDA 12 support. What's the second package you mentioned @pentschev? It said `cuda-cuda` but I think that might've been a typo?
You're right @bdice , thanks for picking that up. I was delusional. 😆 

Fixed the description.
@pentschev I'm a bit confused as to how you got that error. If you do `conda install cudatoolkit` it seems that you get 11.8 (because that's the last version before the structure completely changed).

Numba needs (excluding the UMD shared object which is not the responsibility of conda):

* `libnvvm.so`
* `libnvrtc.so`
* `libcudart.so`
* `libcudadevrt.a`
* `libdevice.10.bc`

It seems that Numba can detect them all if I install `cuda-nvcc` and `cuda-nvrtc` (output from the libs test with #9034 and the NVRTC change from #8893, for completeness):

```
$ python -c "from numba import cuda; cuda.cudadrv.libs.test()"
Finding driver from candidates:
	libcuda.so
	libcuda.so.1
	/usr/lib/libcuda.so
	/usr/lib/libcuda.so.1
	/usr/lib64/libcuda.so
	/usr/lib64/libcuda.so.1
Using loader <class 'ctypes.CDLL'>
	Trying to load driver...	ok
		Loaded from libcuda.so
	Mapped libcuda.so paths:
		/usr/lib/x86_64-linux-gnu/libcuda.so.530.30.02
Finding nvvm from Conda environment (NVIDIA package)
	Located at /home/gmarkall/mambaforge/envs/tktest/nvvm/lib64/libnvvm.so.4.0.0
	Trying to open library...	ok
Finding nvrtc from Conda environment (NVIDIA package)
	Located at /home/gmarkall/mambaforge/envs/tktest/lib/libnvrtc.so.12.0.76
	Trying to open library...	ok
Finding cudart from Conda environment (NVIDIA package)
	Located at /home/gmarkall/mambaforge/envs/tktest/lib/libcudart.so.12.0.107
	Trying to open library...	ok
Finding cudadevrt from Conda environment (NVIDIA package)
	Located at /home/gmarkall/mambaforge/envs/tktest/lib/libcudadevrt.a
	Checking library...	ok
Finding libdevice from Conda environment (NVIDIA package)
	Located at /home/gmarkall/mambaforge/envs/tktest/nvvm/libdevice/libdevice.10.bc
	Checking library...	ok
```

It seemed like `cuda-nvcc` pulled in `cuda-version` 12 already, and if we can avoid tying the docs to a specific version (i.e. so we don't have to change it when 13 comes out) that would be good. I'm thinking that we should recommend:

- For CUDA 11.x users: `conda install cudatoolkit`
- For everybody else (CUDA 12 and above): `conda install cuda-nvcc cuda-nvrtc`

If that sounds good I'll open a PR for this - can you confirm this would be a good plan @pentschev / @bdice?
> @pentschev I'm a bit confused as to how you got that error.

I just saw https://github.com/rapidsai/dask-cuda/pull/1201, and now I think maybe the error came when the CUDA toolkit wasn't installed at all?
> I'm thinking that we should recommend:
> 
> * For CUDA 11.x users: `conda install cudatoolkit`
> * For everybody else (CUDA 12 and above): `conda install cuda-nvcc cuda-nvrtc`

Sounds good to me!
After further exploration, we have found all the files above can be gotten with just 2 dependencies on CUDA 12:

* `cuda-nvcc-impl`
* `cuda-nvrtc`

Currently this is based on the `conda-forge` packages (though the `nvidia` packages are planned to reflect this in the future)
I am trying to call a ".cu" function (following https://numba.readthedocs.io/en/latest/cuda/cuda_ffi.html#linking-and-calling-functions). However, I get the error:
```
  File "cuda/nvrtc.pyx", line 213, in cuda.nvrtc.nvrtcCreateProgram
  File "cuda/cnvrtc.pyx", line 23, in cuda.cnvrtc.nvrtcCreateProgram
  File "cuda/_cuda/cnvrtc.pyx", line 130, in cuda._cuda.cnvrtc._nvrtcCreateProgram
  File "cuda/_cuda/cnvrtc.pyx", line 45, in cuda._cuda.cnvrtc.cuPythonInit
RuntimeError: Failed to dlopen libnvrtc.so.12
```
But I have cuda 11.8 installed, and cudatoolkit installed. Is this a bug or a misconfiguration on my part?
Perhaps  you have `cuda-python` 12 installed, which is looking for NVRTC from CUDA 12. I think you could try either upgrading your CUDA toolkit to 12 or downgrading `cuda-python` to 11.
BTW if you don't want to rely on cuda-python, you could apply https://github.com/numba/numba/pull/9086 locally, which should make it into 0.58.
> Perhaps you have `cuda-python` 12 installed, which is looking for NVRTC from CUDA 12. I think you could try either upgrading your CUDA toolkit to 12 or downgrading `cuda-python` to 11.

Bingo - thanks!
> After further exploration, we have found all the files above can be gotten with just 2 dependencies on CUDA 12:
> 
>     * `cuda-nvcc-impl`
> 
>     * `cuda-nvrtc`
> 
> 
> Currently this is based on the `conda-forge` packages (though the `nvidia` packages are planned to reflect this in the future)

@jakirkham We don't seem to have packages for Windows on the `conda-forge` channel`, but they are on the `nvidia` channel - should we be suggesting people use that instead?
No as the `conda-forge` packages have a number of fixes that have not made it into the `nvidia` channel packages. Wouldn't be comfortable making that recommendation until the `nvidia` channel has addressed those items

There are a number of Windows packages on the `conda-forge` channel. This includes `cuda-nvrtc`

Currently missing are `cuda-cudart` ( https://github.com/conda-forge/cuda-cudart-feedstock/issues/1 ) and as a consequence `cuda-nvcc-impl` ( https://github.com/conda-forge/cuda-nvcc-impl-feedstock/issues/3 )

Am wondering how much of `cuda-nvcc-impl` is needed here. For instance we discussed splitting out `nvvm` previously  ( https://github.com/conda-forge/cuda-nvcc-impl-feedstock/issues/4 ). Would that be sufficient for our needs here? What else would we want from `cuda-nvcc-impl` beyond `nvvm`?
> Currently missing are `cuda-cudart` ( https://github.com/conda-forge/cuda-cudart-feedstock/issues/1 ) and as a consequence `cuda-nvcc-impl` ( https://github.com/conda-forge/cuda-nvcc-impl-feedstock/issues/3 )

Both of these have since been added on Windows. We could use some help testing them though. Would appreciate hearing any feedback
@gmarkall would you (or someone) be able to test the Windows CUDA 12 packages and let us know whether those work for Numba?
Opened a separate issue on testing the conda-forge Numba package on Windows CUDA 12

Let's follow up there: https://github.com/conda-forge/numba-feedstock/issues/130
Based on testing, it looks like we may want to update the search path logic for Windows CUDA 12: https://github.com/numba/numba/issues/9278

Otherwise appears they are working 🎉

If you are interested in testing, please follow-up in issue: https://github.com/conda-forge/numba-feedstock/issues/130
The search path logic has since been updated: https://github.com/numba/numba/pull/9279

Testing with the nightly packages confirms they are working: https://github.com/conda-forge/numba-feedstock/issues/130

Think we are in a good place to update the docs at this point
> Think we are in a good place to update the docs at this point

@jakirkham @bdice could you confirm if what was suggested by @gmarkall in https://github.com/numba/numba/issues/9045#issuecomment-1611163628 is still correct? Summary below:

> For CUDA 11.x users: `conda install cudatoolkit`
> For everybody else (CUDA 12 and above): `conda install cuda-nvcc cuda-nvrtc`
@bdice Yes that is still correct. You could make it more minimal by switching cuda-nvcc to cuda-nvcc-impl as suggested in https://github.com/numba/numba/issues/9045#issuecomment-1636709607 but the impl package might be more of an implementation detail… I don’t have strong feelings either way.
Can we please make this change once and for all, and close this issue? 👍

Thanks for all the hard work, everyone. 

Resolving patch:
diff --git a/numba/cuda/cudadrv/libs.py b/numba/cuda/cudadrv/libs.py
--- a/numba/cuda/cudadrv/libs.py
+++ b/numba/cuda/cudadrv/libs.py
@@ -1,8 +1,9 @@
-"""Cudatoolkit libraries lookup utilities.
+"""CUDA Toolkit libraries lookup utilities.
 
-Cudatoolkit libraries can be available via either:
+CUDA Toolkit libraries can be available via either:
 
-- the `cudatoolkit` conda package,
+- the `cuda-nvcc` and `cuda-nvrtc` conda packages for CUDA 12,
+- the `cudatoolkit` conda package for CUDA 11,
 - a user supplied location from CUDA_HOME,
 - a system wide location,
 - package-specific locations (e.g. the Debian NVIDIA packages),
diff --git a/numba/cuda/cudadrv/nvvm.py b/numba/cuda/cudadrv/nvvm.py
--- a/numba/cuda/cudadrv/nvvm.py
+++ b/numba/cuda/cudadrv/nvvm.py
@@ -330,6 +330,7 @@ def get_log(self):
     (12, 1): ((5, 0), (9, 0)),
     (12, 2): ((5, 0), (9, 0)),
     (12, 3): ((5, 0), (9, 0)),
+    (12, 4): ((5, 0), (9, 0)),
 }
 
 
@@ -416,10 +417,14 @@ def get_arch_option(major, minor):
 
 
 MISSING_LIBDEVICE_FILE_MSG = '''Missing libdevice file.
-Please ensure you have package cudatoolkit >= 11.0
-Install package by:
+Please ensure you have a CUDA Toolkit 11.2 or higher.
+For CUDA 12, ``cuda-nvcc`` and ``cuda-nvrtc`` are required:
 
-    conda install cudatoolkit
+    $ conda install -c conda-forge cuda-nvcc cuda-nvrtc "cuda-version>=12.0"
+
+For CUDA 11, ``cudatoolkit`` is required:
+
+    $ conda install -c conda-forge cudatoolkit "cuda-version>=11.2,<12.0"
 '''
 
 


Test patch: