Custom CUDA Kernel returns Zero or Illegal Memory Access Error

jannk: <p>Hi everyone,<br>
following the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">C++/CUDA extension tutorial on the pytorch website</a> and having a look at the linked source code I have created my own CUDA kernel which does not do something useful, but is done as a learning project.<br>
My issue is that in this simple example I either get the created zero matrix as a result in Python or, if after the kernel call cudaDeviceSynchronize(); is added, an illegal memory access error (Code 700).</p>
<p>The kernel code is the following, it copies the data from a const 3d input matrix into a 3d output matrix:</p>
<pre><code class="lang-auto">template &lt;typename scalar_t&gt;
__global__ void reduce_cuda_kernel(
    const torch::PackedTensorAccessor32&lt;scalar_t,3,torch::RestrictPtrTraits&gt; matrix,
    torch::PackedTensorAccessor32&lt;scalar_t,3,torch::RestrictPtrTraits&gt; output
)
{
    const int y = blockIdx.y * blockDim.y + threadIdx.y;
    const int x = blockIdx.x * blockDim.x + threadIdx.x;
    
    if ((y &lt; output.size(0)) &amp;&amp; (x &lt; output.size(1)))
    {
        printf("Hello from CUDA If.\n");
        for(int i=0;i&lt;matrix.size(2);i++)
        {
            output[y][x][i] = matrix[y][x][i];
        }
    }
}
</code></pre>
<p>The kernel is launched in the following code, which creates a zero tensor from the input matrix “matrix” with zero_like and launches the kernel with calculated amount of blocks, each with pre set 32x32 thread grids:</p>
<pre><code class="lang-auto">torch::Tensor reduce_cuda(torch::Tensor matrix)
{
    // Define output like our input matrix
    auto output = torch::zeros_like(matrix);
    // Fixed block size of 32x32 threads
    const dim3 threads(32, 32, 1);
    // Calculate grid size based on matrix
    const auto H = matrix.size(0);
    const auto W = matrix.size(1);
    const dim3 blocks(ceilf(H/32.0), ceilf(W/32.0), 1);
    
    // Call the kernel
    AT_DISPATCH_FLOATING_TYPES(output.scalar_type(), "reduce_cuda", ([&amp;] {
            reduce_cuda_kernel&lt;scalar_t&gt;&lt;&lt;&lt;blocks, threads&gt;&gt;&gt;(
            matrix.packed_accessor32&lt;scalar_t,3,torch::RestrictPtrTraits&gt;(),
            output.packed_accessor32&lt;scalar_t,3,torch::RestrictPtrTraits&gt;());
    }));
    cudaDeviceSynchronize();
    printf("Cuda Error: %d \n", cudaGetLastError());
    return output;
}
</code></pre>
<p>The entire project and the setup.py install output are <a href="https://gist.github.com/jankolf/cd52c6d0ba9d8e427792d15dc5392c38" rel="noopener nofollow ugc">available here in this Github Gist</a>.</p>
<p>As described above, if I remove cudaDeviceSynchronize(); from reduce_cuda(…) in reduce_cuda_kernel.cu I get in Python the zero matrix created with</p>
<pre><code class="lang-auto">auto output = torch::zeros_like(matrix);
</code></pre>
<p>as a result from the kernel call. The matrix has the same shape, dtype and is on the same CUDA device, but it is just zero. No values are copied towards the output, but CUDA does not throw an error (Error Code 0 is printed) and I get all “Hello from CUDA If” printouts.<br>
If I add cudaDeviceSynchronize();, an error 700 is thrown with illegal memory access when I try to print out the result in Python.<br>
I do not find any error, as other code, e.g. the mentioned example above, are doing more or less the same thing when dispatching the kernel.<br>
Any help is appreciated, thanks in advance.</p>
<p>Greetings,<br>
Jan</p>

eqy: <p>I believe that the true behavior is what happens with the device synchronize—I suspect that the same failure would surface if you tried to e.g., print the output the tensor as it would also synchronize to copy the values back to the host. Another check you could do is run your executable with <code>compute-sanitizer</code> e.g., <code>compute-sanitizer ./myexec</code> which could hopefully give you a slightly more pinpointed location of where the IMA happens.</p>
<p>To start, I’d try a few sanity checks like removing memory accesses for either <code>matrix</code>, <code>output</code>, or both, and consider passing the tensors directly rather than a packed accessor to see if any of those changes the behavior. It might also help to see the failure occurs beyond a specific index, or if even accessing the very start of each tensor triggers the IMA.</p>