CUDA Error when doing multiprocessing on CPU

ArmenAg: <p>I am trying to run the A3C algorithm using the code provided here: <a href="https://github.com/ikostrikov/pytorch-a3c" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ikostrikov/pytorch-a3c: PyTorch implementation of Asynchronous Advantage Actor Critic (A3C) from "Asynchronous Methods for Deep Reinforcement Learning".</a></p>
<p>After fixing some trivial bugs in the code that deal with expand_as, I get an error thrown:</p>
<blockquote>
<p>[2017-07-30 21:27:20,583] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,900] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,905] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,915] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,922] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,933] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,938] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,950] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,950] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,982] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,973] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:20,999] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:21,022] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:21,029] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:21,073] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:21,077] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:21,107] Making new env: PongDeterministic-v4<br>
[2017-07-30 21:27:21,122] Making new env: PongDeterministic-v4<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called recursively<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called recursively<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error</p>
</blockquote>
<p>The code I am referencing does not make any calls to CUDA, so why would I be getting this error?</p>
<p>Thanks.</p>

Peng_Wang: <p>We come across the same problem, any solution to this?</p>

atgambardella: <p>I’ve run into the same issue. Another thing is that pytorch seems to be allocating space on the GPU, despite moving no tensors there</p>

ArmenAg: <p>I still have not found a solution to this.</p>

Crazyai: <p>The same error. Have you found the solution? It is strange that when I run the same python script on another computer with the same configuration(CUDA8.0 + CUDNN6 + Python2.7, but different OS: Ubuntu14.04 vs Ubuntu16.04), it works well.</p>

Crazyai: <aside class="quote no-group quote-modified" data-username="ArmenAg" data-post="1" data-topic="5603">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/armenag/48/3597_2.png" class="avatar"> ArmenAg:</div>
<blockquote>
<p>terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  CUDA error (3): initialization error<br>
terminate called recursively<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
terminate called after throwing an instance of ‘std::runtime_error’</p>
</blockquote>
</aside>
<p>It seems that this error occurs when do backward computing of the final loss in each training process.</p>

Crazyai: <p>Emm… I have solved the problem. Are you using the latest version (0.2.0)? I went back to version 0.1.12, and the problem is fixed. I think maybe it is a bug.</p>

stacked.twix: <p>Version of which package do you mean?<br>
I’m facing the same issue with CUDA8.0.61, Python 2.7, Ubuntu 14.04</p>

Crazyai: <p>CUDA8.0, CUDNN6.0, Python 2.7(3.5 the same problem), Ubuntu16.04. I roll back to PyTorch version 0.1.12, then the problem is fixed.</p>

stacked.twix: <p>Downgrading works for me too. Thanks!</p>

Crazyai: <p>Glad to help you. <img src="//discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=5" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>

xiahouzuoxin: <p>Same Error too<br>
CUDA 8.0, pytorch v0.2.0, python 2.7</p>

smth: <p>can any of you tell me the output of <code>nvidia-smi</code></p>

Crazyai: <p>The output of <code>nvidia-smi</code> of mine:</p>
<pre><code class="lang-auto">Wed Aug 23 19:34:42 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN X (Pascal)    Off  | 0000:08:00.0     Off |                  N/A |
| 36%   60C    P2    96W / 250W |   6795MiB / 12189MiB |     61%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN X (Pascal)    Off  | 0000:09:00.0     Off |                  N/A |
| 30%   52C    P2    56W / 250W |  10363MiB / 12189MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN X (Pascal)    Off  | 0000:88:00.0     Off |                  N/A |
| 35%   59C    P2   105W / 250W |   6845MiB / 12189MiB |     96%      Default |
+-------------------------------+----------------------+----------------------+
|   3  TITAN X (Pascal)    Off  | 0000:89:00.0     Off |                  N/A |
| 40%   66C    P2   127W / 250W |   5027MiB / 12189MiB |     96%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     73255    C   ./build/tools/caffe                           6793MiB |
|    1      8248    C   ./build/tools/caffe                           4381MiB |
|    1      8464    C   ./build/tools/caffe                           2469MiB |
|    1     67991    C   python                                        3511MiB |
|    2     23050    C   ./build/tools/caffe                           6843MiB |
|    3     23050    C   ./build/tools/caffe                           5025MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>But the confusing part is that the python script implements a3c using CPU, none of business with GPU(from my perspective).</p>

florin: <p>Some more info here: <a href="https://github.com/pytorch/pytorch/issues/2517#issuecomment-325039259" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/2517#issuecomment-325039259</a></p>
<p>I think this can be solved by setting <code>mp.set_start_method('spawn')</code> before any cuda call (including setting the rng, for example) when dealing with cuda <code>share_memory</code> objects.</p>

Jiankai: <aside class="quote no-group quote-modified" data-username="florin" data-post="15" data-topic="5603">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/florin/48/103_2.png" class="avatar"> florin:</div>
<blockquote>
<p>mp.set_start_method(‘spawn’)</p>
</blockquote>
</aside>
<p>Same error!</p>
<p>But</p>
<pre><code class="lang-auto">AttributeError: 'module' object has no attribute 'set_start_method'
</code></pre>
<p>It seems that <code>torch.multiprocessing</code> has no set_start_method()</p>

Jiankai: <p>I meet the same error.<br>
Could you please tell me how to go back to version 0.1.12? The method is not provided on <a href="http://pytorch.org/" rel="nofollow noopener">http://pytorch.org/</a><br>
Thank you.</p>

Crazyai: <p>If not familiar with Git, you can just go to this website: <a href="https://github.com/pytorch/pytorch/releases" rel="nofollow noopener">https://github.com/pytorch/pytorch/releases</a> to download v0.1.12 source code and build pytorch from scratch following the instructions provided in README of pytorch’s repo.</p>

florin: <p>I am sure <code>torch.multiprocessing</code> has this method implemented. Maybe you are on an OS other than Linux and the method is not implemented in python’s <code>multiprocessing</code> in the first place?</p>

Jiankai: <p>Probably because Python 2.7 has no <code>set_start_method</code><img src="//discuss.pytorch.org/images/emoji/apple/no_mouth.png?v=5" title=":no_mouth:" class="emoji" alt=":no_mouth:"><img src="//discuss.pytorch.org/images/emoji/apple/no_mouth.png?v=5" title=":no_mouth:" class="emoji" alt=":no_mouth:"></p>