Repository: numba/numba

Issue:
cuda.syncthreads()  - most important primitive - not working in some cases
Threads are not waiting as they are supposed to on syncthreads, resulting in reading global memory (also shared memory) before it has been written to.

The attached code snippet is abstracted from several code files to make it as short/simple to read as possible, while still retaining the broken behavior. The reduced/nonsensical statements are only there to reproduce the syncthreads problem, you can ignore everything but the reading and writing of the number 112211, the other statements shouldn't impact that.

Basically, threads 14-20 are skipping some logic, but are supposed to wait for the threat that does execute the logic and writes a marker into parameter global memory (originally shared memory, but the issue is the same).

The output should say
[112211 112211],
but it only gives
[112211 0],
because the thread that's writing the second entry is not waiting at the syncthread(), as he is supposed to, so the first entry is still zero when he accesses it to copy it into the second position.

It's easy to check two ways that that's what's happening: you can replace the line 'if t < 14 :' with 'if True :', so thread 17 executes the same block and can't run ahead whether or not syncthreads works (bad), then proper result gets output. Second, simply run the script using the cuda simulator, it will properly synch and therefore output the right result.

NUMBA_ENABLE_CUDASIM=1 gdb -q  python
run snippet.py

(I saw some further truly bizarre behavior on my actual code, where it always failed the first time the kernel was created and sent to cuda drivers, but if I ran it again from ipython prompt, it would work the second time)

I have tested this on p2 and g2 instances of Amazon/EC2, using various AMIs and anaconda/numba versions, python2.7 and 3.5, all with the same result. The latest I tested on is Ubuntu 16.04, fully updated numba cuda via conda update, on python 2.7, and with and without anaconda accelerate installed.


Test code with failing syncthreads:
https://github.com/reneschaub/opt_einsum/blob/master/snippet.py

cuda environment info:
In [3]: numba.cuda.cudadrv.libs.test()
Finding cublas
located at /home/ubuntu/anaconda2/lib/libcublas.so.8.0.88
trying to open library...	ok
Finding cusparse
located at /home/ubuntu/anaconda2/lib/libcusparse.so.8.0.61
trying to open library...	ok
Finding cufft
located at /home/ubuntu/anaconda2/lib/libcufft.so.8.0.61
trying to open library...	ok
Finding curand
located at /home/ubuntu/anaconda2/lib/libcurand.so.8.0.61
trying to open library...	ok
Finding nvvm
located at /home/ubuntu/anaconda2/lib/libnvvm.so.3.1.0
trying to open library...	ok
finding libdevice for compute_20...	ok
finding libdevice for compute_30...	ok
finding libdevice for compute_35...	ok
finding libdevice for compute_50...	ok
Out[3]: True
In [4]: numba.cuda.api.detect()
Found 1 CUDA devices
id 0 Tesla K80 [SUPPORTED]
compute capability: 3.7
pci device id: 30
pci bus id: 0
Summary:
1/1 devices are supported
Out[4]: True




Discussion:
Thanks for reporting this issue, I can replicate the problem locally and I'm looking into this.  
Here's a minimal reproducer of the problem: https://gist.github.com/sklam/e6b1decd42dea67822bad8a65dff8967

At the moment, I believe the issue is caused by the division (or the modulo op in the original code) that has an implicit divide-by-zero check that looks like:

```python
if divisor == 0:
   return DIV_BY_ZERO_ERROR_SIGNAL
```

This causes the control-flow to have a non-unifying path after the division.  The warp can diverge.  In my experiment, the `pad[0] = 11211` storing thread always ends up  running after `pad[1] = pad[0]` thread.  By CUDA ISA, the syncthread barrier (`bar.sync 0`) is executed by a warp and not by individual threads.  This means that the barrier is satisfied as long as one thread in the warp (of 32-threads) has reached the barrier.  Given how we only have 1 warp and that the warp have diverged, the syncthread has no effect at all.


I have a CUDA-C reproducer:

```C
#include <stdio.h>


__global__
void perm(int pad[]) {
    int t = threadIdx.x;
    int dxN;

    if ( t >= 0 ) {
        dxN = pad[0];
    }

    if ( t < 14 ) {
        if ( t >= 0 ) {
            // The following branch is the reason
            // comment it out to get the correct behavior
            if (dxN + 1 == 0) goto ERROR;
            pad[2] = 5 / (dxN + 1);
        }
    }

    if ( t == 1 ) {
        pad[0] = 112211;
    }

    __syncthreads();

    if ( t == 17 ) {
        pad[1] = pad[0];
    }

ERROR:
    pad[2] = 321321321;
}


int main() {

    int h_pad[3];
    int *dev_pad = 0;
    cudaMalloc(&dev_pad, sizeof(h_pad));
    cudaMemset(dev_pad, 0, sizeof(h_pad));

    perm<<< 1, 20 >>>(dev_pad);

    cudaMemcpy(h_pad, dev_pad, sizeof(h_pad), cudaMemcpyDeviceToHost);


    printf("pad[0] = %d    pad[1] = %d\n", h_pad[0], h_pad[1]);
    return 0;

}
```

The above code will have the same issue.  Removing the `if (dxN + 1 == 0) goto ERROR;` line will "fix" it.

Resolving patch:
diff --git a/numba/cuda/compiler.py b/numba/cuda/compiler.py
--- a/numba/cuda/compiler.py
+++ b/numba/cuda/compiler.py
@@ -75,7 +75,8 @@ def compile_kernel(pyfunc, args, link, debug=False, inline=False,
     cres = compile_cuda(pyfunc, types.void, args, debug=debug, inline=inline)
     fname = cres.fndesc.llvm_func_name
     lib, kernel = cres.target_context.prepare_cuda_kernel(cres.library, fname,
-                                                          cres.signature.args)
+                                                          cres.signature.args,
+                                                          debug=debug)
 
     cukern = CUDAKernel(llvm_module=lib._final_module,
                         name=kernel.name,
diff --git a/numba/cuda/target.py b/numba/cuda/target.py
--- a/numba/cuda/target.py
+++ b/numba/cuda/target.py
@@ -91,7 +91,7 @@ def call_conv(self):
     def mangler(self, name, argtypes):
         return itanium_mangler.mangle(name, argtypes)
 
-    def prepare_cuda_kernel(self, codelib, fname, argtypes):
+    def prepare_cuda_kernel(self, codelib, fname, argtypes, debug):
         """
         Adapt a code library ``codelib`` with the numba compiled CUDA kernel
         with name ``fname`` and arguments ``argtypes`` for NVVM.
@@ -102,11 +102,12 @@ def prepare_cuda_kernel(self, codelib, fname, argtypes):
         """
         library = self.codegen().create_library('')
         library.add_linking_library(codelib)
-        wrapper = self.generate_kernel_wrapper(library, fname, argtypes)
+        wrapper = self.generate_kernel_wrapper(library, fname, argtypes,
+                                               debug=debug)
         nvvm.fix_data_layout(library._final_module)
         return library, wrapper
 
-    def generate_kernel_wrapper(self, library, fname, argtypes):
+    def generate_kernel_wrapper(self, library, fname, argtypes, debug):
         """
         Generate the kernel wrapper in the given ``library``.
         The function being wrapped have the name ``fname`` and argument types
@@ -142,35 +143,37 @@ def define_error_gv(postfix):
         status, _ = self.call_conv.call_function(
             builder, func, types.void, argtypes, callargs)
 
-        # Check error status
-        with cgutils.if_likely(builder, status.is_ok):
-            builder.ret_void()
 
-        with builder.if_then(builder.not_(status.is_python_exc)):
-            # User exception raised
-            old = Constant.null(gv_exc.type.pointee)
+        if debug:
+            # Check error status
+            with cgutils.if_likely(builder, status.is_ok):
+                builder.ret_void()
 
-            # Use atomic cmpxchg to prevent rewriting the error status
-            # Only the first error is recorded
+            with builder.if_then(builder.not_(status.is_python_exc)):
+                # User exception raised
+                old = Constant.null(gv_exc.type.pointee)
 
-            casfnty = lc.Type.function(old.type, [gv_exc.type, old.type,
-                                                  old.type])
+                # Use atomic cmpxchg to prevent rewriting the error status
+                # Only the first error is recorded
 
-            casfn = wrapper_module.add_function(casfnty,
-                                                name="___numba_cas_hack")
-            xchg = builder.call(casfn, [gv_exc, old, status.code])
-            changed = builder.icmp(ICMP_EQ, xchg, old)
+                casfnty = lc.Type.function(old.type, [gv_exc.type, old.type,
+                                                    old.type])
 
-            # If the xchange is successful, save the thread ID.
-            sreg = nvvmutils.SRegBuilder(builder)
-            with builder.if_then(changed):
-                for dim, ptr, in zip("xyz", gv_tid):
-                    val = sreg.tid(dim)
-                    builder.store(val, ptr)
+                casfn = wrapper_module.add_function(casfnty,
+                                                    name="___numba_cas_hack")
+                xchg = builder.call(casfn, [gv_exc, old, status.code])
+                changed = builder.icmp(ICMP_EQ, xchg, old)
 
-                for dim, ptr, in zip("xyz", gv_ctaid):
-                    val = sreg.ctaid(dim)
-                    builder.store(val, ptr)
+                # If the xchange is successful, save the thread ID.
+                sreg = nvvmutils.SRegBuilder(builder)
+                with builder.if_then(changed):
+                    for dim, ptr, in zip("xyz", gv_tid):
+                        val = sreg.tid(dim)
+                        builder.store(val, ptr)
+
+                    for dim, ptr, in zip("xyz", gv_ctaid):
+                        val = sreg.ctaid(dim)
+                        builder.store(val, ptr)
 
         builder.ret_void()
 


Test patch:
diff --git a/numba/cuda/tests/cudapy/test_exception.py b/numba/cuda/tests/cudapy/test_exception.py
--- a/numba/cuda/tests/cudapy/test_exception.py
+++ b/numba/cuda/tests/cudapy/test_exception.py
@@ -1,9 +1,9 @@
-from __future__ import print_function, absolute_import
+from __future__ import print_function, absolute_import, division
 
 import numpy as np
 
 from numba import config, cuda, jit
-from numba.cuda.testing import unittest, SerialMixin
+from numba.cuda.testing import unittest, SerialMixin, skip_on_cudasim
 
 
 def foo(ary):
@@ -39,6 +39,69 @@ def foo(do_raise):
         with self.assertRaises(ValueError):
             foo[1, 1](True)
 
+    def case_raise_causing_warp_diverge(self, with_debug_mode):
+        """Testing issue #2655.
+
+        Exception raising code can cause the compiler to miss location
+        of unifying branch target and resulting in unexpected warp
+        divergence.
+        """
+        @cuda.jit(debug=with_debug_mode)
+        def problematic(x, y):
+            tid = cuda.threadIdx.x
+            ntid = cuda.blockDim.x
+
+            if tid > 12:
+                for i in range(ntid):
+                    y[i] += x[i] // y[i]
+
+            cuda.syncthreads()
+            if tid < 17:
+                for i in range(ntid):
+                    x[i] += x[i] // y[i]
+
+        @cuda.jit
+        def oracle(x, y):
+            tid = cuda.threadIdx.x
+            ntid = cuda.blockDim.x
+
+            if tid > 12:
+                for i in range(ntid):
+                    if y[i] != 0:
+                        y[i] += x[i] // y[i]
+
+            cuda.syncthreads()
+            if tid < 17:
+                for i in range(ntid):
+                    if y[i] != 0:
+                        x[i] += x[i] // y[i]
+
+        n = 32
+        got_x = 1. / (np.arange(n) + 0.01)
+        got_y = 1. / (np.arange(n) + 0.01)
+        problematic[1, n](got_x, got_y)
+
+        expect_x = 1. / (np.arange(n) + 0.01)
+        expect_y = 1. / (np.arange(n) + 0.01)
+        oracle[1, n](expect_x, expect_y)
+
+        np.testing.assert_almost_equal(expect_x, got_x)
+        np.testing.assert_almost_equal(expect_y, got_y)
+
+    def test_raise_causing_warp_diverge(self):
+        """Test case for issue #2655.
+        """
+        self.case_raise_causing_warp_diverge(with_debug_mode=False)
+
+    @skip_on_cudasim("failing case doesn't happen in CUDASIM")
+    @unittest.expectedFailure
+    def test_raise_causing_warp_diverge_failing(self):
+        """Test case for issue #2655.
+
+        This test that the issue still exists in debug mode.
+        """
+        self.case_raise_causing_warp_diverge(with_debug_mode=True)
+
 
 if __name__ == '__main__':
     unittest.main()