Sudden surge in CPU RAM usage after upgrading pytorch v1.6 to v1.9

username: Amithpn
cooked: <p>Hi,</p>
<p>I am noticing a ~3Gb increase in CPU RAM occupancy after the first .cuda() call. I recently updated the pytorch v1.6 to v1.9.0+cu111. After the upgrade i see there is increase in RAM utilization of ~3 GB when i load the model.</p>
<p>I’ve noticed this behavior in my power edge:</p>
<p>OS: Ubuntu 20.04.4 LTS<br>
Processor:  Intel® Xeon® Gold 6338N CPU @ 2.20GHz, 32 Cores<br>
GPU: Nividia A2<br>
Pytorch: 1.9.0+cu111</p>
<p>Below is the tabular description of the issue im facing:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>torch version</th>
<th>CPU Memory consumption</th>
<th>GPU consumption</th>
</tr>
</thead>
<tbody>
<tr>
<td>1.6</td>
<td>1.2 GB per process</td>
<td>0.9 GB per process</td>
</tr>
<tr>
<td>1.9</td>
<td>3.1 GB per process</td>
<td>1.7 GB per process</td>
</tr>
</tbody>
</table>
</div><p>With this we see other processes are getting a hit due to less RAM available. Can this be addressed? Do we have any solution around the same?</p>

username: ptrblck
cooked: <p>Both used PyTorch releases are old by now so update to the latest stable or nightly release and check if it’s still the case as e.g. CUDA’s lazy loading was enabled which should reduce the device memory usage for &gt;=11.7 and additionally the host memory usage for &gt;=11.8.</p>

username: Amithpn
cooked: <p>I will try to upgrade to the latest torch. But the docker image size will also increase significantly. Any pointers to address that?</p>

username: ptrblck
cooked: <p>It’s unclear which docker image you are using and where the size increase is coming from, so I don’t know if you could address it.</p>

username: Amithpn
cooked: <p>I use ubuntu 20.04 as a base image to package the model as service for inferencing. So when i updated from torch 1.6 to 1.9, the image size grew from 7 Gb to 14 Gb.</p>

username: Amithpn
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , i upgraded to torch 1.13.1 using the below command:</p>
<p>python3.7 -m pip install torch==1.13.1+cu117 torchvision==0.14.1 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
<p>I see below exception when i try to import torchvision.</p>
<p>root@a761eb87f45e:/var/log/supervisor# python3<br>
Python 3.7.7 (default, May  7 2020, 21:25:33)<br>
[GCC 7.3.0] :: Anaconda, Inc. on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torchvision<br>
/opt/conda/lib/python3.7/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: libc10_hip.so: cannot open shared object file: No such file or directory<br>
warn(f"Failed to load image Python extension: {e}")</p>
</blockquote>
</blockquote>
</blockquote>
<p>I found the compatible versions from this link:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/get-started/previous-versions/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/get-started/previous-versions/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://pytorch.org/assets/images/social-share.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://pytorch.org/get-started/previous-versions/" target="_blank" rel="noopener nofollow ugc">Previous PyTorch Versions</a></h3>

  <p>Installing previous versions of PyTorch</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Any idea what is causing the issue?</p>

username: ptrblck
cooked: <p>You are not running into an exception, but a warning in <code>torchvision</code>. Also, <code>1.13.1</code> is not the latest release.</p>

username: Amithpn
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, i updated to the latest torch v2.0.1, cuda 11.7. I see some improvements in terms of GPU utilization and RAM memory usage. What is the improvements in cuda 11.8? Do you advise to update to cuda v11.8?</p>

username: ptrblck
cooked: <p>Yes, since lazy module loading was added in 11.7 and improved in 11.8:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="181074">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>CUDA’s lazy loading was enabled which should reduce the device memory usage for &gt;=11.7 and additionally the host memory usage for &gt;=11.8.</p>
</blockquote>
</aside>