Custom CUDA Extension with Dynamic Parallelism

username: hibagus
cooked: <p>Hi,</p>
<p>I am trying to create a custom module using CUDA with Dynamic Parallelism. I have followed the basic module extension from <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">Pytorch Documentation</a>.</p>
<p>My newmodule.cpp is as follows</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;

cudaError_t newmodule_cuda(...);

torch::Tensor newmodule(
    const torch::Tensor&amp; self,
    const torch::Tensor&amp; weight,
    const torch::Tensor&amp; bias,
    int64_t pad)
{
	// do some operations
        newmodule_cuda(...);
        // do some operations
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
  m.def("forward", &amp;newmodule, "newmodule CUDA");
}
</code></pre>
<p>My newmodule_cuda.cu is as follows.</p>
<pre><code class="lang-auto">#include &lt;cuda.h&gt;
#include &lt;cuda_runtime.h&gt;
#include &lt;driver_functions.h&gt;
#include &lt;torch/extension.h&gt;
#include &lt;ATen/ATen.h&gt;

template &lt;typename scalar_t&gt;
__global__ 
void child(.....)
{
	// do operation with tensor
}

template &lt;typename scalar_t&gt;
__global__
void parent(...)
{
	// do parent operation
	
	// launching child
		
	// Convolve
	child&lt;&lt;&lt;numBlock, numThread&gt;&gt;&gt;(...);
	child&lt;&lt;&lt;numBlock, numThread&gt;&gt;&gt;(...);
	// do parent operation	
}

cudaError_t newmodule_cuda(...)
{
	torch::Device deviceCPU(torch::kCPU);
    torch::Device deviceGPU(torch::kCPU);
	if (torch::cuda::is_available())
    {
        std::cout &lt;&lt; "CUDA is available! Run on GPU." &lt;&lt; std::endl;
		// do preparation
	    AT_DISPATCH_FLOATING_TYPES(self.type(), "newmodule_cuda", ([&amp;] {
			parent&lt;scalar_t&gt;&lt;&lt;&lt;numBlock,numThread&gt;&gt;&gt;(...);	
		}));
        cudaDeviceSynchronize();
    }
	else
	{
		// CPU code
		
	}
	return cudaSuccess;
}
</code></pre>
<p>My setup.py is as follows.</p>
<pre><code class="lang-auto">from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
 
setup(
    name='newmodule',
    ext_modules=[
        CUDAExtension('convtbcglu_cuda', [
            'newmodule.cpp',
            'newmodule_cuda.cu',
        ],
		extra_compile_args={'cxx': ['-Wall'], 'nvcc': ['-arch=sm_70']})                  
    ],
    cmdclass={
        'build_ext': BuildExtension
    })
</code></pre>
<p>With this setup, I get an error when executing python setup.py install</p>
<pre><code class="lang-auto">error: kernel launch from __device__ or __global__ functions requires separate compilation mode

</code></pre>
<p>Then, I add ‘-rdc=true’, ‘-lcudadevrt’ to my setup.py as follows.</p>
<pre><code class="lang-auto">from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
 
setup(
    name='newmodule',
    ext_modules=[
        CUDAExtension('convtbcglu_cuda', [
            'newmodule.cpp',
            'convtbcglu_cuda.cu',
        ],
		extra_compile_args={'cxx': ['-Wall'], 'nvcc': ['-arch=sm_70', '-rdc=true', '-lcudadevrt']})                   
    ],
    cmdclass={
        'build_ext': BuildExtension
    })
</code></pre>
<p>It is compiled successfuly, but when I am trying to run the newmodule, it returns error.</p>
<pre><code class="lang-auto">undefined symbol: __cudaRegisterLinkedBinary_50
</code></pre>
<p>I have read this <a href="https://discuss.pytorch.org/t/link-errors-for-extension-with-rdc-true/20432">Issue</a> but still cannot figure out the solution.</p>
<p>Any help would be appreciated.</p>
<p>Thank you.</p>

username: Mikolaj_Hnatiuk
cooked: <p>try changing<br>
<strong>global</strong><br>
void child(…)<br>
to</p>
<p><strong>device</strong><br>
void child(…)<br>
Let me know if that helps</p>

username: alexmath
cooked: <p>Reproduced the error with.</p>
<p>Python: 3.7.5<br>
PyTorch: 1.3.1<br>
nvcc: 10.0.130<br>
c++ is gcc 7.4.0<br>
os: Ubuntu</p>
<p>I have no  idea what is causing this.<br>
Dynamic parallelism is vital for my current task, will be following this post closely.</p>
<p><a class="mention" href="/u/hibagus">@hibagus</a>: Did you try to compile things by hand?</p>

username: alexmath
cooked: <p>Reproduced error also with pytorch 1.2.0.</p>
<p>I discovered</p>
<p>nvcc -v     -&gt;   10.0.130</p>
<p>import torch<br>
torch.version.cuda # gives 10.1.243</p>
<p>Downgrading to pytorch 1.2.0 made the two versions match, but it did not resolve the issue.</p>

username: alexmath
cooked: <p>This caused following compilation error:</p>
<p>error: a <strong>device</strong> function call cannot be configured</p>
<p>Is there any reason you expected this to work?</p>

username: Vincent-bin
cooked: <p>I got the same issue here, have you solved it?</p>

username: Ajinkya_Ambatwar
cooked: <p>I am also facing the exact same issue. Reading from this stackoverflow <a href="https://stackoverflow.com/questions/22115197/dynamic-parallelism-undefined-reference-to-cudaregisterlinkedbinary-linking" rel="noopener nofollow ugc">question</a>, it suggests we need an intermediate step to do device linking. How can this be achieved here?<br>
<a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/ptrblck">@ptrblck</a> any suggestion on how to build dynamic parallelism ?</p>

username: Vincent-bin
cooked: <p>Currently, the Pytorch build script doesn’t support it yet. You need to add an explicit step to do a device link with the –dlink option when you are using ‘-rdc=true’. For example:</p>
<pre><code class="lang-auto">nvcc –arch=sm_35 –dc a.cu b.cu
nvcc –arch=sm_35 –dlink a.o b.o –o dlink.o
g++ a.o b.o dlink.o x.cpp –lcudart
</code></pre>
<p>I have changed my build script and it works well.</p>

username: Ajinkya_Ambatwar
cooked: <p><a class="mention" href="/u/vincent-bin">@Vincent-bin</a> can you please tell how did you manage?<br>
Inside the build.py or setup.py script?<br>
Where exactly did you add these instructions?</p>

username: fsoler
cooked: <p>Having the same issue on latest pytorch. ¿Did you find a solution?</p>