Thoughts on use of CPU ram as a swap for GPU

username: ReyhaneAskari
cooked: <p>I want to use cpu RAM as a swap to GPU ram to allow oversubscription. Is there a way to implement this in cuda or C++?</p>
<p>As a first step, I <a href="https://github.com/ReyhaneAskari/pytorch/commit/fddb7941c8c783d29c5b4163f9d3661475bd49af" rel="nofollow noopener">tried to replace</a> every <strong>cudaMalloc</strong> with <strong>cudaMallocManaged</strong> and tested it with a simple gemm on a pascal GPU. I increase the size of the matrix and check when I get a memory error. Right now both cases crash at the same time, so we don’t really have any speedup. So do you have any thoughts on this?</p>

username: kouyoumin
cooked: <p>Hi, tried <a href="https://github.com/kouyoumin/pytorch/commit/1dc152f0ae4bac9bd6ec483f57df388e172a4de2" rel="nofollow noopener">something similar</a> and it passed <a href="https://gist.github.com/kouyoumin/26757000be7e985664438a1c213f376a" rel="nofollow noopener">my simple test</a>.</p>

username: mtbrandy
cooked: <p>Large Model Support implements exactly what you describe:</p>
<p>Wiki: <a href="https://github.com/mtbrandy/pytorch/wiki/Large-Model-Support" rel="nofollow noopener">https://github.com/mtbrandy/pytorch/wiki/Large-Model-Support </a><br>
Code: <a href="https://github.com/mtbrandy/pytorch/tree/v1.1.0-LMS" rel="nofollow noopener">https://github.com/mtbrandy/pytorch/tree/v1.1.0-LMS </a></p>
<p>This is the contribution proposal for the LMS feature that is currently available in the Watson Machine Learning solution. For information about that solution, see <a href="https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.1/navigation/wmlce_getstarted_pytorch.html" rel="nofollow noopener">https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.1/navigation/wmlce_getstarted_pytorch.html</a></p>

username: Dale_Song
cooked: <aside class="quote no-group quote-modified" data-username="mtbrandy" data-post="3" data-topic="13193">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mtbrandy/48/13044_2.png" class="avatar"> mtbrandy:</div>
<blockquote>
<p><a href="https://www.ibm.com/support/knowledgecenter/SS5SF7_1.6.1/navigation/wmlce_getstarted_pytorch.html" class="inline-onebox" rel="noopener nofollow ugc">IBM Documentation</a></p>
</blockquote>
</aside>
<p>Is there more tech details about LMS feature?</p>

username: mtbrandy
cooked: <p>You might be interested in the following blog post which walks through a usecase for PyTorch LMS and some performance considerations:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://www.ibm.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://developer.ibm.com/linuxonpower/2019/12/10/deeplabv3-image-segmentation-model-with-pytorch-lms/" target="_blank" title="09:25AM - 10 December 2019" rel="nofollow noopener">Power developer portal – 10 Dec 19</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:132/74;"><img src="https://developer.ibm.com/linuxonpower/wp-content/uploads/sites/105/2019/12/Fig1.png" class="thumbnail" width="132" height="74"></div>

<h3><a href="https://developer.ibm.com/linuxonpower/2019/12/10/deeplabv3-image-segmentation-model-with-pytorch-lms/" target="_blank" rel="nofollow noopener">DeepLabv3+ image segmentation model with PyTorch LMS - Power developer portal</a></h3>

<p>Large Model Support (LMS) technology enables training of large deep neural networks that would exhaust GPU memory while training. PyTorch is a relatively new and popular Python-based open source deep learning framework built by Facebook for faster...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: mohotmoz
cooked: <p><a class="mention" href="/u/kouyoumin">@kouyoumin</a> thank you so much for your post - I am trying to replicate with the most recent version of PyTorch and it looks like the spots where you edited the code are different now - any pointers as to where these changes could be made? (really excited to try this on our new NVSwitch’d A100s to see if the unified memory can truly be “one giant GPU” as NVIDIA claims with a contiguous address space…)</p>

username: kouyoumin
cooked: <p><a class="mention" href="/u/mohotmoz">@mohotmoz</a> My idea was simple: replacing cudaMalloc with cudaMallocManaged. However, there’s still a check preventing cuda op from accessing tensors on another cuda device. So I modified the logic to allow that condition.<br>
I have updated the code for recent changes of pytorch.</p>