Should I use 'spawn' method to start multi-processing?

username: ycszen
cooked: <p>I use <code>torch.distributed </code> to train my model.<br>
When I use <code>torch.multiprocessing.set_start_method('spawn')</code>, the gpu usage memory will be increased with the increasing <code>num_workers</code>.<br>
However, when I don’t use  <code>torch.multiprocessing.set_start_method('spawn')</code>, the gpu usage memory is consistent with different <code>num_workers</code>.</p>
<p>Therefore, should I use <code>spawn</code> to start multi-processing ?<br>
What’s the influence of the <code>set_start_method('spawn')</code> ?<br>
Why the increasing <code>num_workers</code> increases the gpu usage memory when <code>spawn</code> mode?</p>

username: mrshenli
cooked: <p>When using GPU, I believe <code>spawn</code> should be used, as according to this <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing" rel="nofollow noopener">multiprocessing best practices</a> page, CUDA context (~500MB) does not fork. This could also be the reason why you see increasing GPU memory footprint when using more spawned processes, as each process will have its dedicated CUDA context.</p>
<p>Curious, can you allocate a different GPU to each different process? Or do they have to use the same GPU in your application?</p>

username: ycszen
cooked: <p>For distributed parallel training, we do allocate one process for one GPU to train model.</p>
<p>However, in the dataloader, because it also adopts the multi-processing, the increasing <code>num_workers</code> will increase the GPU memory footprint.</p>
<p>In my opinion, the increased GPU memory footprint due to the increasing <code>num_workers</code> can not help to training (faster of better performance).</p>

username: mrshenli
cooked: <p>If those dataloader processes do not use GPU, I guess they can use fork instead?</p>
<p>cc <a class="mention" href="/u/vincentqb">@vincentqb</a> for DataLoader questions.</p>

username: ayalaa2
cooked: <p>I want to bump this post, I’m having this exact problem right now. Each additional worker my processes are spawning for data loading is resulting in an increase of about 500 MiB <em>per</em> worker.</p>
<p>Does anyone know how to fix this?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="ayalaa2" data-post="5" data-topic="85521">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ayalaa2/48/21195_2.png" class="avatar"> ayalaa2:</div>
<blockquote>
<p>Each additional worker my processes are spawning for data loading is resulting in an increase of about 500 MiB <em>per</em> worker.</p>
</blockquote>
</aside>
<p>The 500MB is about the size of a CUDA context. Does those processes use GPUs?</p>

username: ayalaa2
cooked: <p>They shouldn’t.</p>
<p>Specifically, I’m trying to run my code with 2 GPUs, thus I spawn two processes. Each initialize their own DataLoader object with <code>num_workers=2</code>. I find that there’s 6 processes in total using GPU memory, I definitely expect the two main processes to utilize GPU memory but I don’t understand why the dataloader worker processes are also utilizing GPU memory.</p>
<p>If I run my code without DDP, I do not see this issue.</p>

username: mrshenli
cooked: <p>Let’s continue discussions in <a href="https://discuss.pytorch.org/t/distributeddataparallel-causes-dataloader-workers-to-utilize-gpu-memory/88731" class="inline-onebox">DistributedDataParallel causes Dataloader workers to utilize GPU memory</a></p>