detail::getCUDAHooks().initCUDA() triggers SEGV

lampadephoria: <p>Hi All,</p>
<p>I have C++ module subclass torch::nn::Module. I built the module into libmnist_torch.so.</p>
<blockquote>
<p>MNISTImpl::MNISTImpl() {<br>
conv1_  = register_module(“conv1_”,<br>
torch::nn::Conv2d(torch::nn::Conv2dOptions(1, 6, 5).stride(1)));<br>
pool1_  = register_module(“pool1_”, torch::nn::AvgPool2d(torch::nn::AvgPool2dOptions(2).stride(2)));<br>
conv2_  = register_module(“conv2_”, torch::nn::Conv2d(torch::nn::Conv2dOptions(6, 16, 5).stride(1)));<br>
pool2_  = register_module(“pool2_”, torch::nn::AvgPool2d(torch::nn::AvgPool2dOptions(2).stride(2)));<br>
fc1_    = register_module(“fc1_”,   torch::nn::Linear(400, 120));<br>
fc2_    = register_module(“fc2_”,   torch::nn::Linear(120, 84));<br>
fc3_    = register_module(“fc3_”,   torch::nn::Linear(84, 10));<br>
}<br>
…</p>
</blockquote>
<p>Then from another project’s build script, I link libmnist_torch.so,</p>
<blockquote>
<p>// Grab GPU if available<br>
torch::Device device(torch::kCPU);<br>
if (torch::cuda::is_available()) {<br>
<strong>device = torch::Device(torch::kCUDA, 0);</strong><br>
}<br>
…<br>
myMNIST = MNIST();<br>
<strong>myMNIST-&gt;to(device);</strong></p>
</blockquote>
<p><code>myMNIST-&gt;to(device)</code> triggers SEGV exception. The stacktrace is</p>
<blockquote>
<p><span class="hashtag-raw">#1</span>  0x00007fc8029ca181 in std::unique_ptr&lt;THCState, void (<em>)(THCState</em>)&gt;::reset (<br>
this=0x7fc7fef36c58 <a>at::globalContext()::globalContext_+24</a>,<br>
__p=0x7fc7f8ce40f0 <a>at::Context::Context():{lambda(THHState*)#2}::_FUN(THHState*)</a>)<br>
at …/gcc-7.4.0/bin/…/include/c++/7.4.0/bits/unique_ptr.h:376<br>
<span class="hashtag-raw">#2</span>  0x00007fc8029c9e82 in std::unique_ptr&lt;THCState, void (<em>)(THCState</em>)&gt;::operator= (<br>
this=0x7fc7fef36c58 <a>at::globalContext()::globalContext_+24</a>, __u=…)<br>
at …/gcc-7.4.0/bin/…/include/c++/7.4.0/bits/unique_ptr.h:283<br>
<strong><span class="hashtag-raw">#3</span>  0x00007fc8029c4064 in at::Context::lazyInitCUDA()::{lambda()<span class="hashtag-raw">#1</span>}::operator()() const (this=0x7fc8091bd150)</strong><br>
**    at …/pytorch/torch/include/ATen/Context.h:74**<br>
<span class="hashtag-raw">#4</span>  0x00007fc8029d7def in std::__invoke_impl&lt;void, at::Context::lazyInitCUDA()::{lambda()<span class="hashtag-raw">#1</span>}&gt;(std::__invoke_other, at::Context::lazyInitCUDA()::{lambda()<span class="hashtag-raw">#1</span>}&amp;&amp;) (__f=…)<br>
at …/gcc-7.4.0/bin/…/include/c++/7.4.0/bits/invoke.h:60<br>
…</p>
</blockquote>
<p>Frame 3 shows</p>
<blockquote>
<p>(gdb) f 3<br>
<span class="hashtag-raw">#3</span>  0x00007fc8029c4064 in at::Context::lazyInitCUDA()::{lambda()<span class="hashtag-raw">#1</span>}::operator()() const (this=0x7fc8091bd150)<br>
at …/pytorch/torch/include/ATen/Context.h:74<br>
74            thc_state = detail::getCUDAHooks().initCUDA();</p>
</blockquote>
<p>It seems that torch::Device(torch::kCUDA, 0) tries to initiate CUDA again and fails.</p>
<p>My questions are:</p>
<ol>
<li>Why does the second CUDA initialization, <code>myMNIST-&gt;to(device)</code>, inside a .so fail why the caller of such .so has initialized CUDA already <code>device = torch::Device(torch::kCUDA, 0);</code>?</li>
<li>How can I pass a flag to torch so that <code>myMNIST-&gt;to(device)</code> would not call call <code>initCUDA()</code> again?</li>
</ol>
<p>Thank you very much.</p>

ptrblck: <p>Which <code>libtorch</code> version are you using?<br>
Also, could you share an executable project so that we could reproduce this issue?</p>

lampadephoria: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
I built libtorch from source with tag v1.6.0.<br>
If the caller program does not trigger CUDA, i.e. <code>device = torch::Device(torch::kCPU);</code>, then <code>myMNIST-&gt;to(device)</code> is a no-op.<br>
And the caller program finishes without an issue.</p>
<p>Let me call the caller program <code>main</code> and the MNIST library <code>libmodule.so</code>. I have google tests built along side of <code>libmodule.so</code> and all the tests including ones engaging CUDA pass.</p>
<p><code>main</code> build script links with <code>-lmodule</code>. I wonder whether <code>main</code> is built with different libstdc++ (and/or other runtime libraries) from what <code>libmodule.so</code> is built upon. So far, I can’t verify such theory yet.</p>
<p>Would you please suggest anything else?</p>

ptrblck: <aside class="quote no-group" data-username="lampadephoria" data-post="3" data-topic="101024">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lampadephoria/48/27603_2.png" class="avatar"> lampadephoria:</div>
<blockquote>
<p>Would you please suggest anything else?</p>
</blockquote>
</aside>
<p>A minimal project to reproduce this issue would be very welcome, as I don’t know what can cause this issue.</p>