Dataloader num_workers>1 cuda initialization error 3

Lin1: <p>Hello everyone,</p>
<p>I have implemented a c++ preprocessor with pybind11. The c++ preprocessor is imported by a customized dataset wraped by torch.utils.data.Dataloader. One of the preprocessor’s member function is then called in <strong>getitem</strong> for generating feature/label for further training. It works fine.</p>
<p>Based on this preprocessor, I hope to re-implement  part of the feature/label generation function with cuda kernels, then copy the result on gpu to cpu, then pass to the python side. The error shows up: cuda initialization error 3 (points to cudamemcpy/cudamemset/…functions), when i set num_works &gt; 0 (it works fine when num_workers = 0).</p>
<p>Any idea for solving it?</p>
<p>Environment:<br>
OS: ubuntu 18.04<br>
Pytorch: 1.11.0<br>
Pybind11: 2.10.0<br>
GPU: 3090 *1<br>
Cuda: 11.3</p>
<p>Thanks,<br>
Lin</p>

ptrblck: <p>Since you are trying to re-initialize a CUDA context, you would need to use the <code>"spawn"</code> start method as described <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing">here</a>.</p>

Lin1: <p>hello <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks for the reply.</p>
<p>I am a little bit confused. As the dataloader’s “num_worker” param seems indicating using multi-threads for fetch batches of data, using multi-process(mp.spawn) means using processes instead of threads? I.e., using 64 processes (64 dataloader instance with num_worker=1) instead of using 64 threads (1 dataloader instance with num_worker=64) , such that cuda kernels(based on pybind11) would work inside each dataloader instance?</p>

ptrblck: <p><code>num_workers&gt;=1</code> will spawn a new process for each worker, so that each of them can load and process a batch of samples in the background while the main process is busy with the model training.<br>
Using mutli-threading is often not a good idea in Python as you would hit the Global Interpreter Lock and might not be able to preload data in the background.</p>

Lin1: <p>Thanks for helping figure out what is happening <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>I tried ddp with mp.spawn, it works. Basically, I follow the tutorial <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">here</a> and the error disappears.</p>
<p>It is still unclear for me why torch.multiprocessing works. I did several experiments:</p>
<ol>
<li>NO mp.spawn      , one gpu, num_workers=1, the dataset is initialized 1 times, error occur.</li>
<li>DDP + mp.spawn , two gpu, num_workers=1, the dataset is initialized 4 times, no error.</li>
</ol>
<p>The key part seems whether the dataset class is initialized for each process (two main processes + two sub-processes for two gpus?) Note: cuda memory is initialized within the c++/cuda preprocessor.</p>
<p>I dig a little into the dataloader’s source code, find that the dataset instance is passed directly to the dataloader as a param and do not find any func calling the dataset’s <strong>init</strong>().</p>
<p>My questions are:</p>
<ol>
<li>Do the dataset class being initialized outside the dataloader class in mp.spawn mode?</li>
<li>“re-initialize a CUDA context” mentioned in first comment indicates the cuda memory initialization in c++ side?</li>
<li>how mp.spawn init dataset for each worker while, without mp.spwan, dataset class is initialized only once?</li>
</ol>
<p>Many thanks for your time.</p>