Multi-GPU training on Windows 10?

nickvu: <p>Whelp, there I go buying a second GPU for my Pytorch DL computer, only to find out that multi-gpu training doesn’t seem to work <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"> Has anyone been able to get DataParallel to work on Win10?<br>
One workaround I’ve tried is to use Ubuntu under WSL2, but that doesn’t seem to work in multi-gpu scenarios either…</p>

ptrblck: <p>Based on <a href="https://discuss.pytorch.org/t/how-to-make-rpc-work-with-wsl/96429/5">this post</a> it seems that DDP is coming first to Windows (which should also be faster than <code>nn.DataParallel</code> if you are using a single process per GPU), while other data parallel utilities seem to be on the roadmap.</p>

mrshenli: <p>Hey <a class="mention" href="/u/nickvu">@nickvu</a>, I would expect <code>DataParallel</code> to work with Windows. What error did you see when using it?</p>
<p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> <a class="mention" href="/u/ngimel">@ngimel</a></p>
<p>Echoing <a class="mention" href="/u/ptrblck">@ptrblck</a>’s comment, yep, Windows support for DDP (with only FileStore rendezvous and Gloo backend) will be included in v.17 as a beta feature. If you encountered any issue with it, please join the discussion here:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/42095" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/42095" target="_blank" rel="noopener nofollow ugc">[RFC] Add Windows support to torch.distributed package</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-07-27" data-time="04:51:00" data-timezone="UTC">04:51AM - 27 Jul 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/gunandrose4u" target="_blank" rel="noopener nofollow ugc">
          <img alt="gunandrose4u" src="https://avatars2.githubusercontent.com/u/52735340?v=4" class="onebox-avatar-inline" width="20" height="20">
          gunandrose4u
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
Enable torch.distributed package supported on windows platform, this feature is only the first step, limited features supported compare to linux...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: windows</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">oncall: distributed</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">windows-triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


nickvu: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>!<br>
The main issue was that there is no NCCL for Windows I think. So yes, the code with a DP-wrapped model would run, and the two GPUs would even show up as active, but the training time would be exactly the same as when using 1 GPU, leading me to think that it’s not really splitting the load…<br>
Any advice on how to tackle it? I would honestly prefer to work with DataParallel since I’m on a single PC - DDP seems much more involved.<br>
Thanks!</p>

mrshenli: <p>Hey <a class="mention" href="/u/nickvu">@nickvu</a>, thanks for sharing the details.  DP is less efficient compared to DDP, as it needs to replicate the model in every iteration. And IIUC, currently there is no plan to improve DP performance. cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a></p>
<p>So if perf is the main concern, DDP might be a better choice.</p>

solarflarefx: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> and <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
If NCCL is supported only in Linux and DDP uses NCCL, how would the support in Windows actually work?  For example, I was training a network using <a href="https://github.com/facebookresearch/detectron2" rel="noopener nofollow ugc">detectron2</a> and it looks like the parallelization built in uses DDP and only works in Linux.</p>
<p>How would the upcoming support of DDP in Windows work (unless I’m mistaken it would likely not use NCCL)?  Would the speeds be slower as compared to doing the same training on Linux?</p>
<p>Finally, how would using WSL2 compare to the native Windows support of DDP?</p>

mrshenli: <aside class="quote no-group" data-username="solarflarefx" data-post="6" data-topic="100207">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/898d66/48.png" class="avatar"> solarflarefx:</div>
<blockquote>
<p>If NCCL is supported only in Linux and DDP uses NCCL, how would the support in Windows actually work?</p>
</blockquote>
</aside>
<p>Currently, DDP can only run with GLOO backend.</p>
<blockquote>
<p>For example, I was training a network using <a href="https://github.com/facebookresearch/detectron2" rel="noopener nofollow ugc">detectron2</a> and it looks like the parallelization built in uses DDP and only works in Linux.</p>
</blockquote>
<p>MSFT helped us enabled DDP on Windows in PyTorch v1.7. Currently, the support only covers file store (for rendezvous) and GLOO backend. So when calling <code>init_process_group</code> on windows, the backend must be <code>gloo</code>, and <code>init_method</code> must be file. To run on a distributed environment, you can provide a file on a network file system. Please see <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">this tutorial</a> and search for <code>win32 </code>.</p>
<blockquote>
<p>Would the speeds be slower as compared to doing the same training on Linux?</p>
</blockquote>
<p>Yep, Gloo backend will be slower than NCCL.</p>
<blockquote>
<p>Finally, how would using WSL2 compare to the native Windows support of DDP?</p>
</blockquote>
<p>I am not aware of this. Let me ask MSFT experts to join the discussion. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

gunandrose4u: <p><a class="mention" href="/u/nickvu">@nickvu</a>, glad to see you want use DDP on Windows. We will do a benchmark for DDP on Windows. At meantime we will compare the performance of DDP between on native Windows and WSL v2. Will update you with result here.</p>

gunandrose4u: <p><a class="mention" href="/u/nickvu">@nickvu</a>, here is the result we compared. Please notice that the result is compared to Linux VM, not WSL<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/1/11a338897fd0ced07352f26d464b021356a5155e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/11a338897fd0ced07352f26d464b021356a5155e" title="results"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/11a338897fd0ced07352f26d464b021356a5155e_2_690x355.jpeg" alt="results" data-base62-sha1="2w1OAU06dmYeQ2I8MXSPTuei4Hk" width="690" height="355" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/11a338897fd0ced07352f26d464b021356a5155e_2_690x355.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/11a338897fd0ced07352f26d464b021356a5155e_2_1035x532.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/11a338897fd0ced07352f26d464b021356a5155e_2_1380x710.jpeg 2x" data-dominant-color="F0EFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">results</span><span class="informations">2478×1276 302 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

solarflarefx: <p>Thanks for posting this and showing the metrics.</p>
<p>I am wondering if anything has changed since the past.  Can WSL2 do DDP now?</p>

solarflarefx: <p><a class="mention" href="/u/gunandrose4u">@gunandrose4u</a></p>
<p>So I’m trying to run train_net.py in Windows.  I have a machine with 4 GPUs.  This works fine in Linux but not in Windows.  In Windows I get:</p>
<p>raise RuntimeError(“No rendezvous handler for {}://”.format(result.scheme))<br>
RuntimeError: No rendezvous handler for tcp://</p>
<p>Does this have anything to do with nccl vs gloo?  Can I force a gloo backend to get DDP working in detectron2?</p>
<p>Are there changes I would have to make to get this working in Windows?</p>
<p>I believe it would involve changes to detectron2\engine\launch.py</p>
<p>Would it be something like:</p>
<p>dist.init_process_group(backend=“gloo”, init_method=file:///c:/libtmp/test.txt, world_size=world_size, rank=global_rank)</p>

pyTorchLearner: <p>Thank you posting the benchmark results. I am trying to reproduce the same results ourselves in my Windows environment but I am not able to see the same speed-up you get. Are you using an open benchmark? Or even if you could give us a few pointers on the ImageNet algo, training set and parameters it would be very appreciated. Thank you!</p>

TheLongSentance: <p>Hi folks, can anyone provide any guidance/point me at any links so I can understand more of what happens when (GPU-enabled) PyTorch is run on Windows 10 with 2 GPUs? I am doing an NLP project using the HuggingFace libraries and when I run it I get the warning <strong>“<em>UserWarning: PyTorch is not compiled with NCCL support</em>”</strong> which I can understand from discussion in this thread is (still) to be expected on Windows until NCCL is available.</p>
<p>So I run the script to run training on my model and looking at the performance tab of Task Manager, both GPU 0 and GPU 1 jump from effectively 0 memory usage to approximately similar 18GB of 24B on each GPU. <strong>Should I expect that both GPUs get busy given the NCCL issue above? Both GPUs seem busy, so what is happening?</strong></p>
<p>Any clues very much appreciated!!!</p>
<p>And by the way, it is not my machine, I am remoting into it so not an option to install Linux to avoid Windows 10. That said, the owner of the machine has said he is happy to temporarily physically remove one of the GPUs so at least I would have more certainty of what is going on with only 1 GPU on Windows.</p>

TheLongSentance: <p>Have physically unistalled one GPU so problem over I hope.</p>

sshuair: <p>Could you share your windows multi-gpu training code? I can not training with multi-gpu with DDP</p>

sshuair: <aside class="quote no-group" data-username="gunandrose4u" data-post="9" data-topic="100207">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/5f9b8f/48.png" class="avatar"> gunandrose4u:</div>
<blockquote>
<p>esult we compared. Please</p>
</blockquote>
</aside>
<p>Has your code worked?</p>

Ali_Waqas: <pre><code class="lang-auto">os.environ['CUDA_VISIBLE_DEVICES'] = "0"
</code></pre>
<p>You can always use this in your train.py file to control which device is accessible to Pytorch.</p>