Writing CUDA Extension, where to start?

username: eduardo4jesus
cooked: <p>I am trying to implement some operations in Torch, using GPU, however they are way slower than what they suppose to be, and I am assuming that is due to overhead of having it all in python.</p>
<p>Basically, I am working with DSP, and basically the only transform available is the FFT. Actually, I have to use other types of transforms, such as DCT, however I sort of implement them either via a linear layer or via FFT, see <a href="https://github.com/zh217/torch-dct" rel="nofollow noopener">torch_dct</a>.</p>
<p>I am considering implementing them as a CUDA kernel, not sure how, yet.</p>
<p>I started taking a look on <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html#writing-a-mixed-c-cuda-extension" rel="nofollow noopener">Writing a Mixed C Cuda Extension</a>, which I have a hard time to understand. I do understand some of it, but I also wonder where I could find more documentation on what function calls I could make.</p>
<p>Then I read <a href="https://www.nvidia.com/docs/IO/116711/sc11-cuda-c-basics.pdf" rel="nofollow noopener">CUDA C/C++ Basics</a>. Maybe the next document to digest would be <a href="https://docs.nvidia.com/cuda/pdf/CUDA_C_Programming_Guide.pdf" rel="nofollow noopener">CUDA C++ PROGRAMMING GUIDE</a></p>
<p>I am not really sure if I am following the right path, so I am asking for some directions.</p>

username: tom
cooked: <p>Basically, you can call any maths functions in cuda. FP32/INT32 is reasonably fast, fp64/int64 (for indexing) is significantly slower</p>
<p>I would recommend looking at CUDA programming patterns for typical ops, e.g.</p>
<ul>
<li>pointwise operations (the easiest),</li>
<li>reductions (e.g. the combination of warp shuffles and shared memory),</li>
</ul>
<p>The other part to know is how global / local memory and caching works.<br>
Also look at code, PyTorch has implementations for many ops, so does torchvision for the detection ops.</p>
<p>There are also some resources looking at kernels for PyTorch, e.g.  I discuss the <a href="https://lernapparat.de/sinkhorn-kernel/" rel="nofollow noopener">a reduction pattern on my blog</a>. This isn’t terribly efficient for matrix multiplication-like patterns, but it’s faster than CPU. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Best regards</p>
<p>Thomas</p>

username: eduardo4jesus
cooked: <p>Thank you, <a class="mention" href="/u/tom">@tom</a></p>
<p>I am wondering with I should try to implement these transforms in CUDA, or just have a C++/Cuda implementation of my application, having the transforms implemented based on the FFT.</p>
<ul>
<li>If I want to compare these transforms with the FFT, it will be challenging to come up with the a CUDA code as good as the one for cuFFT.</li>
<li>If I do the transforms based on cuFFT, I would have a faster development, but the comparison would be kind of pointless, since it is based on cuFFT.</li>
</ul>
<p>I guess I will start with the C++ code in ATen lib.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="eduardo4jesus" data-post="3" data-topic="91253">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eduardo4jesus/48/20203_2.png" class="avatar"> eduardo4jesus:</div>
<blockquote>
<p>If I do the transforms based on cuFFT, I would have a faster development, but the comparison would be kind of pointless, since it is based on cuFFT.</p>
</blockquote>
</aside>
<p>You could compare your custom CUDA extension with the CPU version, which would not use <code>cuFFT</code> and should thus be a proper reference. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: eduardo4jesus
cooked: <p>I don’t think that would be a fair comparison.</p>

username: ptrblck
cooked: <p>I’m not thinking about the performance, but functionality.<br>
If you are certain that your code works perfectly fine, you won’t need to compare it to the CPU outputs of course. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>