Is there will have total 48g memory if I use nvlink to connect two 3090?

prophet_zhan: <p>Now, we bought a 4 way rtx3090 24g GPU server, and I want to confirm there will be total 48g memory while I use nvlink to connect two 3090.</p>
<p>If rtx3090 supports this feature, how should I change my pytorch code?</p>
<p>Thanks.</p>

osalpekar: <blockquote>
<p>I want to confirm there will be total 48g memory while I use nvlink to connect two 3090.</p>
</blockquote>
<p>That sounds right, but since this is a GPU hardware spec-related question, I would ask the NVIDIA team directly.</p>
<blockquote>
<p>If rtx3090 supports this feature, how should I change my pytorch code?</p>
</blockquote>
<p>With 2 GPUs and nvlink connecting them, I would use DistributedDataParallel (DDP) for training. At a high level, you can spawn 2 CPU processes, 1 for each GPU, and create a NCCL Process Group to have fast data transfer between the 2 GPUs. Then you can simply wrap your model with DDP and train.</p>
<p>Here are some references in terms of how to implement:</p>
<ul>
<li>
<p>docs: <a href="https://pytorch.org/docs/master/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch master documentation</a></p>
</li>
<li>
<p>Implementation: <a href="https://pytorch.org/docs/master/notes/ddp.html#example" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch master documentation</a></p>
</li>
<li>
<p>Example: <a href="https://github.com/pytorch/examples/tree/master/distributed/ddp" rel="noopener nofollow ugc">https://github.com/pytorch/examples/tree/master/distributed/ddp</a></p>
</li>
</ul>

nicofish: <p>Nvidia has officially said that they will stop supporting SLI. the 3090 does still have the the connector.</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://nvidia.custhelp.com/euf/assets/themes/standard/images/favicon.ico" class="site-icon" width="48" height="48">
      <a href="https://nvidia.custhelp.com/app/answers/detail/a_id/5082/~/nvidia-sli-support-transitioning-to-native-game-integrations" target="_blank" rel="noopener nofollow ugc">nvidia.custhelp.com</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://nvidia.custhelp.com/app/answers/detail/a_id/5082/~/nvidia-sli-support-transitioning-to-native-game-integrations" target="_blank" rel="noopener nofollow ugc">NVIDIA SLI Support Transitioning to Native Game Integrations | NVIDIA</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I am not sure if the loss of driver profiles also means the loss of driver support going forwards and I do not know if this will affect using the cards for computation. Hopefully someone else can chime in as I am curious to know.</p>

ptrblck: <aside class="quote no-group" data-username="prophet_zhan" data-post="1" data-topic="100378">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prophet_zhan/48/25592_2.png" class="avatar"> prophet_zhan:</div>
<blockquote>
<p>and I want to confirm there will be total 48g memory while I use nvlink to connect two 3090.</p>
</blockquote>
</aside>
<p>No, the devices should not show up as a single GPU with 48GB.<br>
You can connect them via <code>nvlink</code> and use a data or model parallel approach.</p>

prophet_zhan: <p>Actually, I cannot see there is a single 48GB GPU in my server but can get the effection equals to 48GB on training. Such as I can set a higher batch_size without any other change, right?</p>

ptrblck: <aside class="quote no-group" data-username="prophet_zhan" data-post="5" data-topic="100378">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prophet_zhan/48/25592_2.png" class="avatar"> prophet_zhan:</div>
<blockquote>
<p>Such as I can set a higher batch_size without any other change, right?</p>
</blockquote>
</aside>
<p>I’m not sure what “without any other change” means, but you would either have to use a data parallel approach (which needs code changes) or a model parallel (also knows as model sharding), which also needs code changes.<br>
But yes, you can use multiple devices in PyTorch.</p>

gyuko: <p>Hi, sorry for my questions but I have some doubts about data parallel. Can I use data parallel on two nvidia RTX 3060 gpus, which are not compatible with nvlink? Or to use data parallel do I necessarily have to have GPU with nvlink or SLI?</p>

spacecraft1013: <p>You do not need nvlink to use data parallel, nvlink just makes the interconnect bandwidth between them faster, it is not required.</p>

gyuko: <p>So I can use data parallel with any GPUs, right?<br>
Thanks you so much!!</p>

ptrblck: <p>Yes, you should be able to use <code>nn.DataParallel</code> with any GPUs and with your 3060s also.<br>
As <a class="mention" href="/u/spacecraft1013">@spacecraft1013</a> explained you won’t necessarily need <code>nvlink</code>, but it would speed up the p2p communication.</p>
<p>Also note, that we generally recommend to use <code>DistributedDataParallel</code> with a single process per device for the best performance.</p>

gyuko: <p>Thanks you so much <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

thefreeman: <p>Do you know how much faster it would be training on 2x 3090 with NVLink compared to 2x 3090 without NVLink? 10%? 30%? 50%?</p>