A call to torch.cuda.is_available makes an unrelated multi-processing computation crash?

username: FrancoisFleuret
cooked: <p>This piece of code:</p>
<pre><code>import torch

def create_one(batch_size):
    return torch.ByteTensor(batch_size, 128, 128)

# torch.cuda.is_available()                                                                                                                                                                    

pool = torch.multiprocessing.Pool(torch.multiprocessing.cpu_count())

data = pool.map(create_one, torch.LongTensor(1000).fill_(100))
</code></pre>
<p>works fine, but if the cuda.is_available line is uncommented, it crashes with the following error:</p>
<pre><code>Exception in thread Thread-3:
Traceback (most recent call last):
  File "/home/fleuret/misc/anaconda3/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/fleuret/misc/anaconda3/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/fleuret/misc/anaconda3/lib/python3.5/multiprocessing/pool.py", line 429, in _handle_results
    task = get()
  File "/home/fleuret/misc/anaconda3/lib/python3.5/multiprocessing/connection.py", line 251, in recv
    return ForkingPickler.loads(buf.getbuffer())
  File "/home/fleuret/misc/anaconda3/lib/python3.5/site-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
    fd = df.detach()
  File "/home/fleuret/misc/anaconda3/lib/python3.5/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/home/fleuret/misc/anaconda3/lib/python3.5/multiprocessing/reduction.py", line 181, in recv_handle
    return recvfds(s, 1)[0]
  File "/home/fleuret/misc/anaconda3/lib/python3.5/multiprocessing/reduction.py", line 160, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata
</code></pre>
<p>Did I miss something? Works fine on a machine without cuda.</p>

username: smth
cooked: <p>when using multiprocessing with CUDA, it is important to use the <code>spawn</code> method instead of the default <code>fork</code> method.<br>
<a href="http://pytorch.org/docs/notes/multiprocessing.html#sharing-cuda-tensors" class="onebox" target="_blank">http://pytorch.org/docs/notes/multiprocessing.html#sharing-cuda-tensors</a></p>
<pre><code class="lang-auto">import torch.multiprocessing as multiprocessing
multiprocessing.set_start_method('spawn')
</code></pre>
<p>This is a restriction of CUDA/NVIDIA.</p>

username: FrancoisFleuret
cooked: <aside class="quote no-group quote-modified" data-username="smth" data-post="2" data-topic="4075">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"> smth:</div>
<blockquote>
<p>import torch.multiprocessing as multiprocessing<br>
multiprocessing.set_start_method(‘spawn’)</p>
</blockquote>
</aside>
<p>So this should work as-is?</p>
<pre><code>#!/usr/bin/env python                                                                                                                                                                          

import torch
import torch.multiprocessing as multiprocessing

if torch.cuda.is_available():
    multiprocessing.set_start_method('spawn')

def create_one(batch_size):
    return torch.ByteTensor(batch_size, 128, 128)

pool = torch.multiprocessing.Pool(torch.multiprocessing.cpu_count())

data = pool.map(create_one, torch.LongTensor(1000).fill_(100))
</code></pre>
<p>Because it still does not:</p>
<pre><code>  File "/home/fleuret/misc/anaconda3/lib/python3.5/multiprocessing/context.py", line 231, in set_start_method
    raise RuntimeError('context has already been set')
RuntimeError: context has already been set
</code></pre>

username: smth
cooked: <p>my understanding was incorrect for the reason of the error.</p>
<p>According to <a href="https://github.com/pytorch/pytorch/issues/973">https://github.com/pytorch/pytorch/issues/973</a> what is actually happening is that the multiprocessing Pool has a rotating set of workers, which are not waiting for the transfer of the file descriptors (of Tensor sharedmem) to finish from worker to main process, before which the workers are being recycled.</p>
<p>We’ll investigate this further on <a href="https://github.com/pytorch/pytorch/issues/973">https://github.com/pytorch/pytorch/issues/973</a></p>
<p>I noticed that</p>
<pre><code class="lang-auto">pool = multiprocessing.Pool(multiprocessing.cpu_count(), maxtasksperchild=1
</code></pre>
<p>does fix the error (remove the <code>set_start_method</code> from the code, i’ll look into that separately). However, having only one task per child before it’s reinitialized is probably not very efficient.</p>

username: Michael_Petrochuk
cooked: <p>Looks like <code>set_start_method</code> did not work for me but <code>mp = mp.get_context('spawn')</code> did. Hope that provides some help.</p>

username: williamdjones
cooked: <p>I’ve also tried using forkserver as a start method and found an non-negligible improvement in speed, though this is noted in the multiprocessing docs (I’m referring specifically to python 3): <a href="https://docs.python.org/3/library/multiprocessing.html" rel="nofollow noopener">https://docs.python.org/3/library/multiprocessing.html</a></p>

username: lakehanne
cooked: <p>Any news wrt this, people?</p>

username: visoft
cooked: <p>Hi!</p>
<p>Using <a class="mention" href="/u/michael_petrochuk">@Michael_Petrochuk</a> solution is enough:</p>
<pre><code class="lang-auto">import torch.multiprocessing
...
def test_worker_in_mpi():
    mp = torch.multiprocessing.get_context('forkserver')
    pool = mp.Pool(processes=1) # ,maxtasksperchild=1
    rez = pool.map(worker, params_iterator) 
</code></pre>
<p>And this was a pytest. At least for my toy example (inside the worker I create a 2 layer net and pass a tensor through it) it works without any main guard, <code>maxtasksperchild=1</code> or<br>
<code>set_start_method</code></p>
<p>Thanks!</p>

username: mashuai191
cooked: <p>most probably i meet the same issue here. I’m using Flask to server a web service which workd as multi-processing to predict images. Then i got following error.</p>
<pre><code class="lang-auto">Exception in thread Thread-19:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/content/flask-video-streaming/base_camera.py", line 97, in _thread
    for frame in frames_iterator:
  File "&lt;ipython-input-22-98a25d1b92f5&gt;", line 38, in frames
    predictor = DefaultPredictor(cfg)
  File "/content/detectron2_repo/detectron2/engine/defaults.py", line 163, in __init__
    self.model = build_model(self.cfg)
  File "/content/detectron2_repo/detectron2/modeling/meta_arch/build.py", line 19, in build_model
    return META_ARCH_REGISTRY.get(meta_arch)(cfg)
  File "/content/detectron2_repo/detectron2/modeling/meta_arch/rcnn.py", line 41, in __init__
    pixel_mean = torch.Tensor(cfg.MODEL.PIXEL_MEAN).to(self.device).view(num_channels, 1, 1)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 197, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (3) : initialization error at /pytorch/aten/src/THC/THCGeneral.cpp:54
</code></pre>
<p>after some google searching, i tried following solutions but none of them working. Anyone have some suggestions to let me go forward? thanks in advance!</p>
<pre><code class="lang-auto">if __name__ == "__main__":

#    manager.run(host='0.0.0.0', threaded=True)

    print(torch.multiprocessing.get_start_method())

    torch.multiprocessing.set_start_method('spawn', force=True)

    #torch.multiprocessing = torch.multiprocessing.get_context('spawn')

    #torch.multiprocessing.set_start_method('spawn')

    #torch.multiprocessing.set_start_method('forkserver', force=True)

    app.run(host='0.0.0.0', threaded=False, processes=2)
</code></pre>

username: Mona_Jalal
cooked: <p>So, I have the same exact error.</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/63913372/runtimeerror-cuda-runtime-error-3-initialization-error-at-opt-conda-conda" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2414957/mona-jalal" target="_blank" rel="nofollow noopener">
    <img alt="Mona Jalal" src="https://i.stack.imgur.com/0MuAW.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/63913372/runtimeerror-cuda-runtime-error-3-initialization-error-at-opt-conda-conda" target="_blank" rel="nofollow noopener">RuntimeError: cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch-nightly_1553749772122/work/aten/src/THC/THCGeneral.cpp:51</a>
</h4>

<div class="tags">
  <strong>python, anaconda, pytorch, resnet</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2414957/mona-jalal" target="_blank" rel="nofollow noopener">
    Mona Jalal
  </a>
  on <a href="https://stackoverflow.com/questions/63913372/runtimeerror-cuda-runtime-error-3-initialization-error-at-opt-conda-conda" target="_blank" rel="nofollow noopener">04:54AM - 16 Sep 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I am running this code from GoodNews repo:</p>
<pre><code class="lang-auto">[jalal@goku GoodNews]$ python train.py --cnn_weight data/resnet152-b121ed2d.pth
DataLoader loading json file:  data/data_news.json
vocab size is  37200
DataLoader loading h5 file:  data/data_news_label.h5 /scratch2/goodnewsdata/data_news_image.h5
read 489229 images of size 3x256x256
max sequence length in data is 31
assigned 445433 images to split train
assigned 19376 images to split val
assigned 24420 images to split test
WARNING:tensorflow:From train.py:49: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch-nightly_1553749772122/work/aten/src/THC/THCGeneral.cpp line=51 error=3 : initialization error
Traceback (most recent call last):
  File "train.py", line 280, in &lt;module&gt;
    train(opt)
  File "train.py", line 81, in train
    cnn_model.cuda()
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 263, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 190, in _apply
    module._apply(fn)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 196, in _apply
    param.data = fn(param.data)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 263, in &lt;lambda&gt;
    return self._apply(lambda t: t.cuda(device))
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/cuda/__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (3) : initialization error at /opt/conda/conda-bld/pytorch-nightly_1553749772122/work/aten/src/THC/THCGeneral.cpp:51
Closing remaining open files:data/data_news_label.h5...done/scratch2/goodnewsdata/data_news_image.h5...done
</code></pre>
<p>These files have the keyword ‘multiprocessing’:</p>
<pre><code class="lang-auto">[jalal@goku GoodNews]$ rg 'multiprocessing'
dataloader.py
11:# import multiprocessing
12:# from multiprocessing import Process
13:# from multiprocessing.dummy import Pool as ThreadPool
14:# from pathos.multiprocessing import Pool

get_data/get_imgs_only/get_images.py
12:# from multiprocessing.dummy import Pool as ThreadPool

get_data/with_article_urls/get_data_with_urls.py
14:# from multiprocessing.dummy import Pool as ThreadPool

get_data/with_api/get_data_api.py
9:import multiprocessing
15:from multiprocessing.dummy import Pool as ThreadPool
</code></pre>
<p>However, I am not sure how to proceed. Could you please have a look at the files in the repo?</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/furkanbiten/GoodNews" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/22663927?s=400&amp;v=4" class="thumbnail onebox-avatar" width="388" height="388">

<h3><a href="https://github.com/furkanbiten/GoodNews" target="_blank" rel="nofollow noopener">furkanbiten/GoodNews</a></h3>

<p>Good News Everyone! - CVPR 2019 . Contribute to furkanbiten/GoodNews development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I have:</p>
<pre><code class="lang-auto">$ nvidia-smi
Fri Sep 18 23:25:50 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:05:00.0 Off |                  N/A |
|  0%   24C    P8    19W / 250W |     55MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:06:00.0 Off |                  N/A |
|  0%   27C    P8    12W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2637      G   /usr/bin/X                         39MiB |
|    0   N/A  N/A      2903      G   /usr/bin/gnome-shell               12MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>and</p>
<pre><code class="lang-auto">$ python
Python 3.6.7 | packaged by conda-forge | (default, Nov  6 2019, 16:19:42) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.0.0.dev20190328'

&gt;&gt;&gt; torch.version.cuda
'8.0.61'
&gt;&gt;&gt; torch.cuda.is_available()
False

&gt;&gt;&gt; torch.backends.cudnn.enabled
True

$ cat /usr/local/cuda/version.txt 
CUDA Version 10.0.130

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
</code></pre>
<p>I am not sure how to fix the problem by looking at <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver" rel="nofollow noopener">https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver</a></p>
<p>More information here <a href="https://stackoverflow.com/questions/63965122/fixing-torch-cuda-is-available-when-it-is-false" rel="nofollow noopener">https://stackoverflow.com/questions/63965122/fixing-torch-cuda-is-available-when-it-is-false</a></p>