CPU RAM Usage with CUDA is Large (2+GB)

username: gmaggiol
cooked: <p>I’m a developer of a commercial application that requires initializing CUDA between dozens of independent processes. These processes are containerized for versioning - I can’t share much more, but using a single runtime to load several models isn’t possible.</p>
<p>However, initializing CUDA uses upwards of 2GB of RAM (not GPU memory). This is testable like so:</p>
<pre><code class="lang-python">import torch 
a = torch.tensor([0], device="cuda")
</code></pre>
<p>and monitoring RAM from any system tool. From my understanding, this is essentially all driver and library code, and thus the C++ runtime wouldn’t be significantly better.</p>
<p>I’ve tried ONNX (onnxruntime-gpu) and TensorRT in Python. They use about 1.5GB and 1.1GB of RAM respectively, which is still too much for my application. As people are deploying models on mobile devices I’m assuming there must be inference engines that are less memory intensive, but I haven’t found any in my searching that are compatible with NVIDIA GPUs.</p>
<p>Anyone run into similar problems or have any advice? How are people typically deploying models?</p>

username: tom
cooked: <p>This is really a CUDA issue, see thediscussion in</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/12873" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/12873" target="_blank" rel="noopener nofollow ugc">Massive initial memory overhead GPU</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-10-19" data-time="14:34:09" data-timezone="UTC">02:34PM - 19 Oct 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-04-15" data-time="16:42:06" data-timezone="UTC">04:42PM - 15 Apr 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/davidmascharka" target="_blank" rel="noopener nofollow ugc">
          <img alt="davidmascharka" src="https://avatars.githubusercontent.com/u/5611862?v=4" class="onebox-avatar-inline" width="20" height="20">
          davidmascharka
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
There is a huge RAM overhead for using the GPU even for processing small tensors.
Here's a standalone script:
# test.py
import torch
import...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>One option might be to remove unneeded kernels. This is done for mobile already and discussed here:</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/31881" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/31881" target="_blank" rel="noopener nofollow ugc">[Feature Request] reduce CUDA runtime size by selectively compiling PyTorch GPU kernels</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-01-06" data-time="03:14:37" data-timezone="UTC">03:14AM - 06 Jan 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/xxradon" target="_blank" rel="noopener nofollow ugc">
          <img alt="xxradon" src="https://avatars.githubusercontent.com/u/10461663?v=4" class="onebox-avatar-inline" width="20" height="20">
          xxradon
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🚀 Feature
pytorch massive initial memory overhead GPU,we need a way to control the memory and gpu memory during pytorch to cuda()...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: binaries</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: build</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: cuda</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<p>
It’s not too hard to leverage the mobile work if you know how to do it.</p>