Repository: numba/numba

Issue:
Regression finding libNVVM in 0.46.0
Since 0.46.0 I get the following error. 0.45.0 was fine (both packages from conda-forge):
```
Traceback (most recent call last):
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/cudadrv/nvvm.py", line 111, in __new__
    inst.driver = open_cudalib('nvvm', ccc=True)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/cudadrv/libs.py", line 47, in open_cudalib
    raise OSError('library %s not found' % lib)
OSError: library nvvm not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/antoine/arrow/dev/python/pyarrow/tests/test_cuda_numba_interop.py", line 232, in test_pyarrow_jit
    increment_by_one[blockspergrid, threadsperblock](darr)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/compiler.py", line 799, in __call__
    kernel = self.specialize(*args)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/compiler.py", line 810, in specialize
    kernel = self.compile(argtypes)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/compiler.py", line 829, in compile
    kernel.bind()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/compiler.py", line 543, in bind
    self._func.get()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/compiler.py", line 421, in get
    ptx = self.ptx.get()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/compiler.py", line 392, in get
    **self._extra_options)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/cudadrv/nvvm.py", line 468, in llvm_to_ptx
    cu = CompilationUnit()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/cudadrv/nvvm.py", line 146, in __init__
    self.driver = NVVM()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/cudadrv/nvvm.py", line 116, in __new__
    raise NvvmSupportError(errmsg % e)
numba.cuda.cudadrv.error.NvvmSupportError: libNVVM cannot be found. Do `conda install cudatoolkit`:
library nvvm not found
```

I am on Ubuntu 18.04 with the `nvidia-cuda-dev` package installed.
```
$ locate nvvm
[ ... ]
/usr/include/nvvm.h
/usr/lib/cuda/nvvm
/usr/lib/cuda/nvvm/libdevice
/usr/lib/x86_64-linux-gnu/libnvvm.so
/usr/lib/x86_64-linux-gnu/libnvvm.so.3
/usr/lib/x86_64-linux-gnu/libnvvm.so.3.2.0
[ ... ]
```



Discussion:
Can you confirm that it worked in 0.45.0? IIRC, last change in the cuda library finding logic is in 0.44.  Since then, you can use the following to test the library loading:

```python
from numba.cuda import cudadrv
cudadrv.libs.test()
```

On my ubuntu 18.04 + `nvidia-cuda-dev` setup, i can confirm that numba can't find nvvm since 0.44.
In the past, the convention is that cuda on linux is located at `/usr/local/cuda` and the `libnvvm` library is in a subdirectory named `nvvm/lib64`. The `nvidia-cuda-dev` package is completely different and puts the libraries in `/usr/lib/x86_64-linux-gnu/` without any subdirectories for `libnvvm`. It can now be loaded with just `ctypes.CDLL('libnvvm.so')`, but that has never worked before.

@gmarkall, do you know if there is a new recommended load path?
Yes, it works with 0.45.0. I am using the following variables (apparently they are deprecated?):
```
export NUMBAPRO_LIBDEVICE=/usr/lib/cuda/nvvm/libdevice
export NUMBAPRO_NVVM=/usr/lib/x86_64-linux-gnu/libnvvm.so
```

```python
>>> import numba                                                                                                                                                            
>>> numba.__version__                                                                                                                                                       
'0.45.0'
>>> from numba.cuda import cudadrv                                                                                                                                          
>>> cudadrv.libs.test()                                                                                                                                                     
/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/envvars.py:16: NumbaDeprecationWarning: 
Environment variables with the 'NUMBAPRO' prefix are deprecated, found use of NUMBAPRO_NVVM=/usr/lib/x86_64-linux-gnu/libnvvm.so.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-numbapro-environment-variables
  warnings.warn(errors.NumbaDeprecationWarning(msg))
/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numba/cuda/envvars.py:16: NumbaDeprecationWarning: 
Environment variables with the 'NUMBAPRO' prefix are deprecated, found use of NUMBAPRO_LIBDEVICE=/usr/lib/cuda/nvvm/libdevice.

For more information visit http://numba.pydata.org/numba-doc/latest/reference/deprecation.html#deprecation-of-numbapro-environment-variables
  warnings.warn(errors.NumbaDeprecationWarning(msg))
Finding cublas from <unavailable>
	ERROR: can't locate lib
Finding cusparse from <unavailable>
	ERROR: can't locate lib
Finding cufft from <unavailable>
	ERROR: can't locate lib
Finding curand from <unavailable>
	ERROR: can't locate lib
Finding nvvm from NUMBAPRO_NVVM
	located at /usr/lib/x86_64-linux-gnu/libnvvm.so
Finding libdevice from NUMBAPRO_LIBDEVICE
	searching for compute_20...	ok
	searching for compute_30...	ok
	searching for compute_35...	ok
	searching for compute_50...	ok
False
>>>                                                                                                                                                                         
```

Just catching up on recent history with this deprecation (I don't have an Ubuntu 18.04 system handy to try with) - @pitrou does setting `CUDA_HOME` (http://numba.pydata.org/numba-doc/latest/cuda/overview.html#setting-cuda-installation-path) allow you to provide the correct location of libNVVM? If not, perhaps we need to add a `NUMBA_NVVM` env var that provides the same function as the old `NUMBAPRO_NVVM`?
`CUDA_HOME` does not work, no. Perhaps Numba should simply detect the Ubuntu layout?
I think we can do that by falling back to the system loader; i.e. `ctypes.CDLL('libnvvm.so')`. It looks like the debian packages are putting all cuda libraries in the default search both.  However, just having libnvvm is not enough. We need to find where the `libdevice` bitcode files are located.
I also noticed that the CUDA linux installation guide (https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) has no mentioning of the debian package.
> I also noticed that the CUDA linux installation guide (https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html) has no mentioning of the debian package.

There are debian packages that you will reach from the link to the downloads - e.g. https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=debnetwork - however, I don't think the `nvidia-cuda-dev` package from Debian NVIDIA Maintainers (which I think @pitrou is using) is one of those. IIRC the packages from developer.nvidia.com do install into /usr/local/cuda.

>  However, just having libnvvm is not enough. We need to find where the libdevice bitcode files are located.

Right now I have the `nvidia-cuda-dev` package installed on 18.04 and I can't find the libdevice bitcode files anywhere - I only see:

```
$ locate libdevice
/usr/lib/llvm-10/lib/clang/10.0.0/include/__clang_cuda_libdevice_declares.h
```

Are they included in `nvidia-cuda-dev`?

> I think we can do that by falling back to the system loader; i.e. ctypes.CDLL('libnvvm.so'). It looks like the debian packages are putting all cuda libraries in the default search both.

Falling back to the system loader seems reasonable, and what I'd expect / hope for as a "normal" user - in general, having shared libs on the library search path results in them being found once other options have been exhausted.
> > However, just having libnvvm is not enough. We need to find where the libdevice bitcode files are located.
> 
> Right now I have the `nvidia-cuda-dev` package installed on 18.04 and I can't find the libdevice bitcode files anywhere...

I now noticed they're in the `nvidia-cuda-toolkit` package, which places them at `/usr/lib/nvidia-cuda-toolkit/libdevice`. Maybe that would need adding as another location to search for it, in addition to falling back to the system loader for shared objects.
Just to clarify, I'd be happy to make a PR that:

1. Falls back to the system loader for shared objects if they're not otherwise found.
2. Also looks for libdevice in `/usr/lib/nvidia-cuda-toolkit`.

if there's agreement that this is an acceptable way to solve the problem - @sklam would you be happy with this?
@gmarkall, that should solve the problem.

Resolving patch:
diff --git a/numba/cuda/cuda_paths.py b/numba/cuda/cuda_paths.py
--- a/numba/cuda/cuda_paths.py
+++ b/numba/cuda/cuda_paths.py
@@ -14,13 +14,13 @@
 def _find_valid_path(options):
     """Find valid path from *options*, which is a list of 2-tuple of
     (name, path).  Return first pair where *path* is not None.
-    If no valid path is found, return ('<unavailable>', None)
+    If no valid path is found, return ('<unknown>', None)
     """
     for by, data in options:
         if data is not None:
             return by, data
     else:
-        return '<unavailable>', None
+        return '<unknown>', None
 
 
 def _get_libdevice_path_decision():
@@ -30,6 +30,7 @@ def _get_libdevice_path_decision():
         ('Conda environment', get_conda_ctk()),
         ('CUDA_HOME', get_cuda_home('nvvm', 'libdevice')),
         ('System', get_system_ctk('nvvm', 'libdevice')),
+        ('Debian package', get_debian_pkg_libdevice()),
     ]
     by, libdir = _find_valid_path(options)
     return by, libdir
@@ -167,3 +168,14 @@ def get_cuda_paths():
         # Cache result
         get_cuda_paths._cached_result = d
         return d
+
+
+def get_debian_pkg_libdevice():
+    """
+    Return the Debian NVIDIA Maintainers-packaged libdevice location, if it
+    exists.
+    """
+    pkg_libdevice_location = '/usr/lib/nvidia-cuda-toolkit/libdevice'
+    if not os.path.exists(pkg_libdevice_location):
+        return None
+    return pkg_libdevice_location
diff --git a/numba/cuda/cudadrv/libs.py b/numba/cuda/cudadrv/libs.py
--- a/numba/cuda/cudadrv/libs.py
+++ b/numba/cuda/cudadrv/libs.py
@@ -1,23 +1,28 @@
 """Cudatoolkit libraries lookup utilities.
 
-Cudatoolkit libraries can be available via the `cudatoolkit` conda package or a
-user supplied location from CUDA_HOME or a system wide location.
+Cudatoolkit libraries can be available via either:
+
+- the `cudatoolkit` conda package,
+- a user supplied location from CUDA_HOME,
+- a system wide location,
+- package-specific locations (e.g. the Debian NVIDIA packages),
+- or can be discovered by the system loader.
 """
+
 from __future__ import print_function
 import os
 import sys
 import ctypes
-import platform
 
 from numba.findlib import find_lib
 from numba.cuda.cuda_paths import get_cuda_paths
 
 if sys.platform == 'win32':
-    _dllopener = ctypes.WinDLL
+    _dllnamepattern = '%s.dll'
 elif sys.platform == 'darwin':
-    _dllopener = ctypes.CDLL
+    _dllnamepattern = 'lib%s.dylib'
 else:
-    _dllopener = ctypes.CDLL
+    _dllnamepattern = 'lib%s.so'
 
 
 def get_libdevice(arch):
@@ -32,22 +37,24 @@ def open_libdevice(arch):
 
 
 def get_cudalib(lib, platform=None):
+    """
+    Find the path of a CUDA library based on a search of known locations. If
+    the search fails, return a generic filename for the library (e.g.
+    'libnvvm.so' for 'nvvm') so that we may attempt to load it using the system
+    loader's search mechanism.
+    """
     if lib == 'nvvm':
-        return get_cuda_paths()['nvvm'].info
+        return get_cuda_paths()['nvvm'].info or _dllnamepattern % 'nvvm'
     else:
         libdir = get_cuda_paths()['cudalib_dir'].info
 
     candidates = find_lib(lib, libdir, platform)
-    return max(candidates) if candidates else None
+    return max(candidates) if candidates else _dllnamepattern % lib
 
 
-def open_cudalib(lib, ccc=False):
+def open_cudalib(lib):
     path = get_cudalib(lib)
-    if path is None:
-        raise OSError('library %s not found' % lib)
-    if ccc:
-        return ctypes.CDLL(path)
-    return _dllopener(path)
+    return ctypes.CDLL(path)
 
 
 def _get_source_variable(lib):
@@ -67,24 +74,19 @@ def test(_platform=None, print_paths=True):
     for lib in libs:
         path = get_cudalib(lib, _platform)
         print('Finding {} from {}'.format(lib, _get_source_variable(lib)))
-        if path:
-            if print_paths:
-                print('\tlocated at', path)
-            else:
-                print('\tnamed ', os.path.basename(path))
+        if print_paths:
+            print('\tlocated at', path)
         else:
-            print('\tERROR: can\'t locate lib')
-            failed = True
+            print('\tnamed ', os.path.basename(path))
 
-        if not failed and _platform in (None, sys.platform):
+        if _platform in (None, sys.platform):
             try:
                 print('\ttrying to open library', end='...')
-                open_cudalib(lib, ccc=True)
+                open_cudalib(lib)
                 print('\tok')
             except OSError as e:
                 print('\tERROR: failed to open %s:\n%s' % (lib, e))
-                # NOTE: ignore failure of dlopen on cuBlas on OSX 10.5
-                failed = True if not _if_osx_10_5() else False
+                failed = True
 
     archs = 'compute_20', 'compute_30', 'compute_35', 'compute_50'
     where = _get_source_variable('libdevice')
@@ -98,11 +100,3 @@ def test(_platform=None, print_paths=True):
             print('\tERROR: can\'t open libdevice for %s' % arch)
             failed = True
     return not failed
-
-
-def _if_osx_10_5():
-    if sys.platform == 'darwin':
-        vers = tuple(map(int, platform.mac_ver()[0].split('.')))
-        if vers < (10, 6):
-            return True
-    return False
diff --git a/numba/cuda/cudadrv/nvvm.py b/numba/cuda/cudadrv/nvvm.py
--- a/numba/cuda/cudadrv/nvvm.py
+++ b/numba/cuda/cudadrv/nvvm.py
@@ -108,7 +108,7 @@ def __new__(cls):
             if cls.__INSTANCE is None:
                 cls.__INSTANCE = inst = object.__new__(cls)
                 try:
-                    inst.driver = open_cudalib('nvvm', ccc=True)
+                    inst.driver = open_cudalib('nvvm')
                 except OSError as e:
                     cls.__INSTANCE = None
                     errmsg = ("libNVVM cannot be found. Do `conda install "


Test patch:
diff --git a/numba/cuda/tests/nocuda/test_library_lookup.py b/numba/cuda/tests/nocuda/test_library_lookup.py
--- a/numba/cuda/tests/nocuda/test_library_lookup.py
+++ b/numba/cuda/tests/nocuda/test_library_lookup.py
@@ -101,7 +101,7 @@ def test_libdevice_path_decision(self):
         if has_cuda:
             self.assertEqual(by, 'Conda environment')
         else:
-            self.assertEqual(by, "<unavailable>")
+            self.assertEqual(by, "<unknown>")
             self.assertIsNone(info)
         self.assertFalse(warns)
         # Check that CUDA_HOME works by removing conda-env
@@ -119,7 +119,7 @@ def test_libdevice_path_decision(self):
         if get_system_ctk() is None:
             # Fake remove conda environment so no cudatoolkit is available
             by, info, warns = self.remote_do(self.do_clear_envs)
-            self.assertEqual(by, '<unavailable>')
+            self.assertEqual(by, '<unknown>')
             self.assertIsNone(info)
             self.assertFalse(warns)
         else:
@@ -157,7 +157,7 @@ def test_nvvm_path_decision(self):
         if has_cuda:
             self.assertEqual(by, 'Conda environment')
         else:
-            self.assertEqual(by, "<unavailable>")
+            self.assertEqual(by, "<unknown>")
             self.assertIsNone(info)
         self.assertFalse(warns)
         # Check that CUDA_HOME works by removing conda-env
@@ -184,7 +184,7 @@ def test_nvvm_path_decision(self):
         if get_system_ctk() is None:
             # Fake remove conda environment so no cudatoolkit is available
             by, info, warns = self.remote_do(self.do_clear_envs)
-            self.assertEqual(by, '<unavailable>')
+            self.assertEqual(by, '<unknown>')
             self.assertIsNone(info)
             self.assertFalse(warns)
         else:
@@ -227,7 +227,7 @@ def test_cudalib_path_decision(self):
         if has_cuda:
             self.assertEqual(by, 'Conda environment')
         else:
-            self.assertEqual(by, "<unavailable>")
+            self.assertEqual(by, "<unknown>")
             self.assertIsNone(info)
         self.assertFalse(warns)
 
@@ -249,7 +249,7 @@ def test_cudalib_path_decision(self):
         if get_system_ctk() is None:
             # Fake remove conda environment so no cudatoolkit is available
             by, info, warns = self.remote_do(self.do_clear_envs)
-            self.assertEqual(by, "<unavailable>")
+            self.assertEqual(by, "<unknown>")
             self.assertIsNone(info)
             self.assertFalse(warns)
         else: