CUDA Synchronize using Aten.h for a PyTorch CUDA C++ extension

username: jvilchis
cooked: <p>Hi all,</p>
<p>I am trying to implement a mixed C++ CUDA Torch extension. My understanding is that I can only use ATen.h for this.  (e.g. Torch.h cannot be used). The issue is that I wan to <strong>synchronize CUDA kernels from CPU,</strong> so I am trying to do the equivalent to <strong>cudaDeviceSynchronize() in CUDA</strong> , I  cannot find any similar function within at::cuda class.   Could anyone help me out please ?  Am I missing something ?</p>

username: ptrblck
cooked: <p>I think you could try the following:</p>
<pre><code class="lang-auto">CUDAStream stream = getCurrentCUDAStream();
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>

username: jvilchis
cooked: <p>Thanks Ptrblck !,  I’wll try it and post my findings.</p>

username: jvilchis
cooked: <p>I am running my code using CUDAStream to synchronize my CUDA threads from CPU.  Since this is a CUDA C++ PyTorch extension, I imported the module to python and the call my extension. I am having this error at the line this is implemented:  AT_CUDA_CHECK(cudaStreamSynchronize(stream));</p>
<p>RuntimeError: CUDA error: CUDA driver version is insufficient for CUDA runtime version…</p>
<p>Any advice will be highly appreciate it.</p>

username: ptrblck
cooked: <p>Do you have multiple CUDA versions installed as described <a href="https://github.com/pytorch/pytorch/issues/10971" rel="nofollow noopener">here</a>?</p>

username: jvilchis
cooked: <p>I think there was an issue with the version of pytorch I had installed which was compiled for a different version of CUDA I have installed.  Having a clean pytroch installation with the appropriate CUDA version fixed the original problem.  Thanks a lot for that !.</p>
<p>Now, I am facing other issue.</p>
<p>When running my extension on python , the next error is thrown out:</p>
<p>RuntimeError: CUDA error: an illegal memory access was encountered (bspline_cuda at bspline_cuda_kernel.cu:252)</p>
<p>This is the line where there is this check:</p>
<p><strong>AT_CUDA_CHECK(cudaStreamSynchronize(stream));</strong></p>
<p>The code segment that is giving me issues is this:</p>
<pre><code class="lang-auto">vector&lt;at::Tensor&gt; bspline_cuda(at::Tensor x, at::Tensor y)
{

    const int n= NCP;
    const int m_mm = NCP;
    const int n_mm = NCP;
    const int k_mm=1;

    dim3 dimGrid((k_mm + BLOCK_SIZE - 1) / BLOCK_SIZE, (m_mm + BLOCK_SIZE - 1) / BLOCK_SIZE);
    dim3 dimBlock(BLOCK_SIZE, BLOCK_SIZE);

    auto a  = at::zeros_like(x);
    auto b  = at::zeros_like(x);
    auto c  = at::zeros_like(x);
    auto d  = at::zeros_like(x);
    
    auto delta  = at::zeros(NCP-1,  at::kFloat);
    auto Delta  = at::zeros(NCP-1,  at::kFloat);
    auto M      = at::zeros({NCP,NCP},  at::kFloat);
    auto D      = at::zeros(NCP,  at::kFloat);

    auto TensorM_inv = at::zeros({NCP,NCP});

    for (int j =0; j&lt;n; j++)
        for (int i=0;i&lt;n; i++)
            TensorM_inv[i][j] = M_inv[i][j];

 AT_DISPATCH_FLOATING_TYPES( a.type(), 
    "NaturalCubicSplinePart1",
       ([&amp;] 
        {
        NaturalCubicSplinePart1&lt;scalar_t&gt;&lt;&lt;&lt; N/TPB, TPB &gt;&gt;&gt;(
                   a.data&lt;scalar_t&gt;(), 
                   delta.data&lt;scalar_t&gt;(), 
                    Delta.data&lt;scalar_t&gt;() , 
                     x.data&lt;scalar_t&gt;(), 
                  y.data&lt;scalar_t&gt;() 
                     );
                }
              )
        );

at::cuda::CUDAStream stream = at::cuda::getCurrentCUDAStream();
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>
<p>Thanks a lot in advance for the help.</p>

username: jvilchis
cooked: <p>Hi all, finally I got what was wrong with my code:</p>
<p>a) I had a very old NVIDIA driver and CUDA 8.0. So, I update my system to have a clean installation:</p>
<p>NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0</p>
<p>b) * AT_CUDA_CHECK(cudaStreamSynchronize(stream));* was throwing out an error because I was not careful with the types of my tensors.  To solve this problem I made sure all types matched. Below the section of my code with types correctly set.</p>
<pre><code class="lang-auto">vector&lt;at::Tensor&gt; bspline_cuda(at::Tensor x, at::Tensor y)
{

    const int n= NCP;
    const int m_mm = NCP;
    const int n_mm = NCP;
    const int k_mm=1;

    dim3 dimGrid((k_mm + BLOCK_SIZE - 1) / BLOCK_SIZE, (m_mm + BLOCK_SIZE - 1) / BLOCK_SIZE);
    dim3 dimBlock(BLOCK_SIZE, BLOCK_SIZE);

    auto a  = at::zeros_like(x);
    auto b  = at::zeros_like(x);
    auto c  = at::zeros_like(x);
    auto d  = at::zeros_like(x);
    
    auto delta  = at::zeros(NCP-1,  x.type());
    auto Delta  = at::zeros(NCP-1,  x.type());
    auto M      = at::zeros({NCP,NCP},  x.type());
    auto D      = at::zeros(NCP,  x.type());

    auto TensorM_inv = at::zeros({NCP,NCP}, x.type());

    for (int j =0; j&lt;n; j++)
        for (int i=0;i&lt;n; i++)
            TensorM_inv[i][j] = M_inv[i][j];


AT_DISPATCH_ALL_TYPES( x.type(), 
                            "NaturalCubicSplinePart1",
                            ([&amp;] 
                                {
                                 NaturalCubicSplinePart1&lt;scalar_t&gt;&lt;&lt;&lt; N/TPB, TPB &gt;&gt;&gt;(
                                                                 a.data&lt;scalar_t&gt;(), 
                                                                 delta.data&lt;scalar_t&gt;(), 
                                                                 Delta.data&lt;scalar_t&gt;() , 
                                                                 x.data&lt;scalar_t&gt;(), 
                                                                 y.data&lt;scalar_t&gt;() 
                                                                 );
                                }
                            )
                        );


at::cuda::CUDAStream stream = at::cuda::getCurrentCUDAStream();
AT_CUDA_CHECK(cudaStreamSynchronize(stream));</code></pre>
<p>Thanks a lot, my issue is fixed. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=7" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: jy_mo
cooked: <p>Hello, how to use these two, and where should they be placed？</p>

username: ptrblck
cooked: <p>You can call these two lines of code in your C++ code when you want to synchronize the stream.</p>

username: jy_mo
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3dbed4163d57d51bcef654ff4e6213ba2f3fc96.png" data-download-href="https://discuss.pytorch.org/uploads/default/a3dbed4163d57d51bcef654ff4e6213ba2f3fc96" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3dbed4163d57d51bcef654ff4e6213ba2f3fc96_2_690x106.png" alt="image" data-base62-sha1="nnz19D2jZRmkLjufL3FumyPU73g" width="690" height="106" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/3/a3dbed4163d57d51bcef654ff4e6213ba2f3fc96_2_690x106.png, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3dbed4163d57d51bcef654ff4e6213ba2f3fc96.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/3/a3dbed4163d57d51bcef654ff4e6213ba2f3fc96.png 2x" data-dominant-color="F1F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">911×140 26.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I added it this way and it doesn’t work, it takes the same time as if I didn’t add it.</p>

username: jy_mo
cooked: <ul>
<li>Transferring tensors from CUDA to CPU takes a lot of time</li>
</ul>

username: ptrblck
cooked: <p>Could you post a minimal code snippet which shows the slow data transfer?</p>

username: guuzaa
cooked: <p>Hi Jorge,<br>
I don’t know how to install  c++ extension’s header file. Can you guys tell me? Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: eduardo4jesus
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="40048" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I think you could try the following:</p>
<pre><code class="lang-auto">CUDAStream stream = getCurrentCUDAStream();
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>
</blockquote>
</aside>
<p>I am having a hard time compiling these lines.</p>
<p>I have in my code</p>
<pre><code class="lang-auto">at::cuda::CUDAStream stream = at::cuda::getCurrentCUDAStream();
AT_CUDA_CHECK(cudaStreamSynchronize(stream));
</code></pre>
<hr>
<p>The compiler could not identify <code>at::cuda</code>, neither <code>AT_CUDA_CHECK</code>. Then, I tried to use the include</p>
<p><code>#include &lt;ATen/cuda/CUDAContext.h&gt;</code></p>
<p>But the compiler also complained</p>
<p><code>fatal error: cublas_v2.h: No such file or directory</code></p>
<hr>
<p>Does anyone know what am I missing?</p>
<p>Thanks in advance.</p>

username: eduardo4jesus
cooked: <p>In my case, the following snippet worked for me.</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;

#include &lt;chrono&gt;
#include &lt;cuda.h&gt;
#include &lt;cuda_runtime.h&gt;

// From https://stackoverflow.com/a/14038590/2313889
#define CUDA_CHECK(ans) { gpuAssert((ans), __FILE__, __LINE__); }
inline void gpuAssert(cudaError_t code, const char *file, int line, bool abort=true)
{
   if (code != cudaSuccess) 
   {
      fprintf(stderr,"GPU Assert Error: %s %s %d\n", cudaGetErrorString(code), file, line);
      if (abort) exit(code);
   }
}

std::vector&lt;at::Tensor&gt; multiplication_complex_cuda(at::Tensor x, at::Tensor h) {

    const int THREADS = 1024;
    const int B = x.size(0);
    const int F = h.size(0);
    const int C = x.size(1);
    const int H = x.size(2);
    const int W = x.size(3);
    const int PLANE_SIZE = H*W;

    const auto Z = (H*W + THREADS - 1)/THREADS;
    const dim3 GRID_SIZE(B, F, Z);

    auto out = torch::zeros(
        {B, F, H, W, 2},
        torch::TensorOptions().device(x.device().type(), x.device().index())
        );

    auto start = std::chrono::high_resolution_clock::now();

    AT_DISPATCH_FLOATING_TYPES(x.scalar_type(), "multiplication_complex_cuda",
    ([&amp;] {
        multiplication_traditional_cuda_kernel&lt;scalar_t&gt;&lt;&lt;&lt;GRID_SIZE, THREADS&gt;&gt;&gt;(
        x.packed_accessor32&lt;scalar_t, 5, torch::RestrictPtrTraits&gt;(),
        h.packed_accessor32&lt;scalar_t, 5, torch::RestrictPtrTraits&gt;(),
        out.packed_accessor32&lt;scalar_t, 5, torch::RestrictPtrTraits&gt;(),
        THREADS, C, W, PLANE_SIZE);
    }));

    cudaDeviceSynchronize();

    auto stop = std::chrono::high_resolution_clock::now();
    auto duration = at::zeros({1});
    duration[0] = std::chrono::duration_cast&lt;std::chrono::nanoseconds&gt;(stop - start).count();

    return {out, duration};
}

</code></pre>
<hr>
<p>Though I got around the problem. I would still appreciate if someone could point what was wrong before. As you can see, I avoided the ATen functions to handle Stream. It would be handy to know how to use them.</p>
<p>Thank you all.</p>