Repository: numba/numba

Issue:
cuda segfault nvvm with opt+debug
<!--

Thanks for opening an issue! To help the Numba team handle your information
efficiently, please first ensure that there is no other issue present that
already describes the issue you have
(search at https://github.com/numba/numba/issues?&q=is%3Aissue).

-->

## Reporting a bug

<!--

Before submitting a bug report please ensure that you can check off these boxes:

-->

- [x] I have tried using the latest released version of Numba (most recent is
 visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
- [x] I have included a self contained code sample to reproduce the problem.
  i.e. it's possible to run as 'python bug.py'.

This segfaults:
```python
from numba import cuda

@cuda.jit(debug=True)
def f(b, n):
    result = 0
    for i in range(n):
        result += b

f[1, 1](1, 2)
```
Under `cuda-gdb` the backtrace points to the `nvvmCompilerProgram` call in `libnvvm.so`. 

Setting `opt=False` or `debug=False` avoids the segfault.

CC @gmarkall 


Discussion:
Thanks for opening this ticket @stuartarchibald. 

I apologise if this is not the right place to post this, but I thought it might be related and helpful to the issue:

I am experiencing some more strange issues with while/for loops in numba: matrix indexing in a while loop also creates segfaults. I have attached a minimal example here that produces the error:

```python
from numba import cuda
import numpy as np

# 3 kernels, 2 fail on segfault
# fmat       indexes a matrix b[i,j] with 2 variables i,j; fails on segfault
# flinTwo   indexes an array b[i+j] with 2 variables i,j; passes
# flinthree indexes an array b[i+m+j] with 3 variables i,j,m; fails on segfault

@cuda.jit(debug=True)
def fmat(b, n):
    # this kernel fails... segfault
    result = 0
    j = 1
    i = 0
    while (j < b.shape[0]): # don't overflow
        while (i < n) and (i < b.shape[1]): # don't overflow
            result = b[j,i]
            i += 1
        j += 1
        i = 0

@cuda.jit(debug=True)
def flinTwo(b, m, n):
    # this kernel is happy, no faults!
    result = 0
    j = 0
    i = 0
    while (j < 2):
        while (i < n):
            if (j+i+m) < len(b): # don't overflow
                result = b[j+i]
            i += 1
        j += 1
        i = 0

@cuda.jit(debug=True)
def flinThree(b, m, n):
    # this kernel is happy, no faults!
    result = 0
    j = 0
    i = 0
    while (j < 2):
        while (i < n):
            if (j+i+m) < len(b): # don't overflow
                result = b[j+i+m]
            i += 1
        j += 1
        i = 0


a = np.asarray([[0,1,2,3,4,5],[0,1,2,3,4,5]])
d = a.flatten()
a_d = cuda.to_device(a)
d_d = cuda.to_device(d)

flinTwo[1,1](d_d, 0, 5)
print('flinTwo completes, no failures')

# this segfaults 
flinThree[1, 1](d_d, 0, 5)

# this segfaults too
fmat[1,1](a_d, 5)
```

I assume this issue is related to the issue on which the ticket was opened, so I post it here. Any fix or workaround would be appreciated!

Turning on debug without turning off optimization is not supported by NVVM - noted in the description of the `-g` option in https://docs.nvidia.com/cuda/libnvvm-api/group__compilation.html#group__compilation_1g76ac1e23f5d0e2240e78be0e63450346.

To compile with debug, `opt=False` also needs adding:

```python
from numba import cuda

@cuda.jit(debug=True, opt=False)
def f(b, n):
    result = 0
    for i in range(n):
        result += b

f[1, 1](1, 2)
```

and

```python
from numba import cuda
import numpy as np

# 3 kernels, 2 fail on segfault
# fmat       indexes a matrix b[i,j] with 2 variables i,j; fails on segfault
# flinTwo   indexes an array b[i+j] with 2 variables i,j; passes
# flinthree indexes an array b[i+m+j] with 3 variables i,j,m; fails on segfault

@cuda.jit(debug=True, opt=False)
def fmat(b, n):
    # this kernel fails... segfault
    result = 0
    j = 1
    i = 0
    while (j < b.shape[0]): # don't overflow
        while (i < n) and (i < b.shape[1]): # don't overflow
            result = b[j,i]
            i += 1
        j += 1
        i = 0

@cuda.jit(debug=True, opt=False)
def flinTwo(b, m, n):
    # this kernel is happy, no faults!
    result = 0
    j = 0
    i = 0
    while (j < 2):
        while (i < n):
            if (j+i+m) < len(b): # don't overflow
                result = b[j+i]
            i += 1
        j += 1
        i = 0

@cuda.jit(debug=True, opt=False)
def flinThree(b, m, n):
    # this kernel is happy, no faults!
    result = 0
    j = 0
    i = 0
    while (j < 2):
        while (i < n):
            if (j+i+m) < len(b): # don't overflow
                result = b[j+i+m]
            i += 1
        j += 1
        i = 0


a = np.asarray([[0,1,2,3,4,5],[0,1,2,3,4,5]])
d = a.flatten()
a_d = cuda.to_device(a)
d_d = cuda.to_device(d)

flinTwo[1,1](d_d, 0, 5)
print('flinTwo completes, no failures')

# this segfaults 
flinThree[1, 1](d_d, 0, 5)

# this segfaults too
fmat[1,1](a_d, 5)
```

both execute for me with no segfault or other issue.

For now, (prior to other fixes / reorganization of the `debug` kwarg to disentangle exception checking and debuginfo generation), we should emit a warning to inform the user they ought to also use `opt=False`.

Resolving patch:
diff --git a/numba/cuda/compiler.py b/numba/cuda/compiler.py
--- a/numba/cuda/compiler.py
+++ b/numba/cuda/compiler.py
@@ -18,7 +18,8 @@
 from numba.core.compiler_machinery import (LoweringPass, PassManager,
                                            register_pass)
 from numba.core.dispatcher import OmittedArg
-from numba.core.errors import NumbaDeprecationWarning
+from numba.core.errors import (NumbaDeprecationWarning,
+                               NumbaInvalidConfigWarning)
 from numba.core.typed_passes import IRLegalization, NativeLowering
 from numba.core.typing.typeof import Purpose, typeof
 from warnings import warn
@@ -212,6 +213,12 @@ def compile_ptx(pyfunc, args, debug=False, lineinfo=False, device=False,
     :return: (ptx, resty): The PTX code and inferred return type
     :rtype: tuple
     """
+    if debug and opt:
+        msg = ("debug=True with opt=True (the default) "
+               "is not supported by CUDA. This may result in a crash"
+               " - set debug=False or opt=False.")
+        warn(NumbaInvalidConfigWarning(msg))
+
     nvvm_options = {
         'debug': debug,
         'lineinfo': lineinfo,
diff --git a/numba/cuda/decorators.py b/numba/cuda/decorators.py
--- a/numba/cuda/decorators.py
+++ b/numba/cuda/decorators.py
@@ -1,6 +1,7 @@
 from warnings import warn
 from numba.core import types, config, sigutils
-from numba.core.errors import DeprecationError, NumbaDeprecationWarning
+from numba.core.errors import (DeprecationError, NumbaDeprecationWarning,
+                               NumbaInvalidConfigWarning)
 from .compiler import (compile_device, declare_device_function, Dispatcher,
                        compile_device_dispatcher)
 from .simulator.kernel import FakeCUDAKernel
@@ -44,8 +45,9 @@ def jit(func_or_sig=None, device=False, inline=False, link=[], debug=None,
     :type link: list
     :param debug: If True, check for exceptions thrown when executing the
        kernel. Since this degrades performance, this should only be used for
-       debugging purposes.  Defaults to False.  (The default value can be
-       overridden by setting environment variable ``NUMBA_CUDA_DEBUGINFO=1``.)
+       debugging purposes. If set to True, then ``opt`` should be set to False.
+       Defaults to False.  (The default value can be overridden by setting
+       environment variable ``NUMBA_CUDA_DEBUGINFO=1``.)
     :param fastmath: When True, enables fastmath optimizations as outlined in
        the :ref:`CUDA Fast Math documentation <cuda-fast-math>`.
     :param max_registers: Request that the kernel is limited to using at most
@@ -81,6 +83,12 @@ def jit(func_or_sig=None, device=False, inline=False, link=[], debug=None,
     debug = config.CUDA_DEBUGINFO_DEFAULT if debug is None else debug
     fastmath = kws.get('fastmath', False)
 
+    if debug and opt:
+        msg = ("debug=True with opt=True (the default) "
+               "is not supported by CUDA. This may result in a crash"
+               " - set debug=False or opt=False.")
+        warn(NumbaInvalidConfigWarning(msg))
+
     if sigutils.is_signature(func_or_sig):
         if config.ENABLE_CUDASIM:
             def jitwrapper(func):


Test patch:
diff --git a/numba/cuda/tests/cudapy/test_debuginfo.py b/numba/cuda/tests/cudapy/test_debuginfo.py
--- a/numba/cuda/tests/cudapy/test_debuginfo.py
+++ b/numba/cuda/tests/cudapy/test_debuginfo.py
@@ -38,7 +38,7 @@ def foo(x):
         self._check(foo, sig=(types.int32[:],), expect=False)
 
     def test_debuginfo_in_asm(self):
-        @cuda.jit(debug=True)
+        @cuda.jit(debug=True, opt=False)
         def foo(x):
             x[0] = 1
 
@@ -47,7 +47,7 @@ def foo(x):
     def test_environment_override(self):
         with override_config('CUDA_DEBUGINFO_DEFAULT', 1):
             # Using default value
-            @cuda.jit
+            @cuda.jit(opt=False)
             def foo(x):
                 x[0] = 1
 
@@ -116,7 +116,7 @@ def test_debug_function_calls_internal_impl(self):
         # This is condensed from this reproducer in Issue 5311:
         # https://github.com/numba/numba/issues/5311#issuecomment-674206587
 
-        @cuda.jit((types.int32[:], types.int32[:]), debug=True)
+        @cuda.jit((types.int32[:], types.int32[:]), debug=True, opt=False)
         def f(inp, outp):
             outp[0] = 1 if inp[0] in (2, 3) else 3
 
@@ -136,15 +136,15 @@ def kernel(arr):
                 arr[i] = threadid()
 
     def _test_chained_device_function(self, kernel_debug, f1_debug, f2_debug):
-        @cuda.jit(device=True, debug=f2_debug)
+        @cuda.jit(device=True, debug=f2_debug, opt=False)
         def f2(x):
             return x + 1
 
-        @cuda.jit(device=True, debug=f1_debug)
+        @cuda.jit(device=True, debug=f1_debug, opt=False)
         def f1(x, y):
             return x - f2(y)
 
-        @cuda.jit((types.int32, types.int32), debug=kernel_debug)
+        @cuda.jit((types.int32, types.int32), debug=kernel_debug, opt=False)
         def kernel(x, y):
             f1(x, y)
 
@@ -168,15 +168,15 @@ def test_chained_device_function(self):
     def _test_chained_device_function_two_calls(self, kernel_debug, f1_debug,
                                                 f2_debug):
 
-        @cuda.jit(device=True, debug=f2_debug)
+        @cuda.jit(device=True, debug=f2_debug, opt=False)
         def f2(x):
             return x + 1
 
-        @cuda.jit(device=True, debug=f1_debug)
+        @cuda.jit(device=True, debug=f1_debug, opt=False)
         def f1(x, y):
             return x - f2(y)
 
-        @cuda.jit(debug=kernel_debug)
+        @cuda.jit(debug=kernel_debug, opt=False)
         def kernel(x, y):
             f1(x, y)
             f2(x)
@@ -233,7 +233,7 @@ def test_chained_device_three_functions(self):
         # to ensure that the recursion visits all the way down the call tree
         # when fixing linkage of functions for debug.
         def three_device_fns(kernel_debug, leaf_debug):
-            @cuda.jit(device=True, debug=leaf_debug)
+            @cuda.jit(device=True, debug=leaf_debug, opt=False)
             def f3(x):
                 return x * x
 
@@ -245,7 +245,7 @@ def f2(x):
             def f1(x, y):
                 return x - f2(y)
 
-            @cuda.jit(debug=kernel_debug)
+            @cuda.jit(debug=kernel_debug, opt=False)
             def kernel(x, y):
                 f1(x, y)
 
diff --git a/numba/cuda/tests/cudapy/test_warning.py b/numba/cuda/tests/cudapy/test_warning.py
--- a/numba/cuda/tests/cudapy/test_warning.py
+++ b/numba/cuda/tests/cudapy/test_warning.py
@@ -1,6 +1,6 @@
 import numpy as np
 from numba import cuda
-from numba.cuda.testing import CUDATestCase, skip_on_cudasim
+from numba.cuda.testing import unittest, CUDATestCase, skip_on_cudasim
 from numba.tests.support import linux_only, override_config
 from numba.core.errors import NumbaPerformanceWarning
 import warnings
@@ -107,3 +107,33 @@ def foo(r, x):
                 foo[1, N](ary, N)
 
         self.assertEqual(len(w), 0)
+
+    def test_warn_on_debug_and_opt(self):
+        with warnings.catch_warnings(record=True) as w:
+            cuda.jit(debug=True, opt=True)
+
+        self.assertEqual(len(w), 1)
+        self.assertIn('not supported by CUDA', str(w[0].message))
+
+    def test_warn_on_debug_and_opt_default(self):
+        with warnings.catch_warnings(record=True) as w:
+            cuda.jit(debug=True)
+
+        self.assertEqual(len(w), 1)
+        self.assertIn('not supported by CUDA', str(w[0].message))
+
+    def test_no_warn_on_debug_and_no_opt(self):
+        with warnings.catch_warnings(record=True) as w:
+            cuda.jit(debug=True, opt=False)
+
+        self.assertEqual(len(w), 0)
+
+    def test_no_warn_with_no_debug_and_opt_kwargs(self):
+        with warnings.catch_warnings(record=True) as w:
+            cuda.jit()
+
+        self.assertEqual(len(w), 0)
+
+
+if __name__ == '__main__':
+    unittest.main()