Why does moving a model to the GPU increase the CPU memory usage?

peterkim95: <p>I have a simple model:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.fc1 = nn.Linear(303, 128)
        self.fc2 = nn.Linear(128, 1)
        
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
</code></pre>
<p>When I create the model on my CPU as such,</p>
<pre><code class="lang-auto">model = Net()
</code></pre>
<p>Both CPU and GPU memory usage remain unchanged. However, when I then move the model to the GPU,</p>
<pre><code class="lang-auto">model.cuda() # model.to(device) does the same
</code></pre>
<p>My CPU memory usage shoots up from 410MB to 1.95GB and my GPU memory usage goes from 0MB to 716MB.</p>
<p>My question is</p>
<ol>
<li>Before I move the model to the GPU, why does creating the model not affect even my CPU memory? The parameters surely must go somewhere (unless they’re lazily created?)</li>
<li>Why does moving the model to the GPU suddenly consume <strong>both</strong> CPU and GPU memory? Shouldn’t all the weights go entirely to the GPU?</li>
</ol>

eqy: <p>I’m not sure what directly causes (1) but I think there could be reasons behind (2)<br>
(2) Did you create any <code>cuda</code> tensors before moving your model to the GPU? If not, then my handwavy explanation for this is that the moment you create a <code>cuda</code> tensor (e.g., directly declaring one or by moving your model to <code>cuda</code>), this does a whole bunch of things like spin up a <code>cuda</code> context and load a bunch of extra library code corresponding to GPU kernels and all of these things will take up memory.</p>

peterkim95: <p>I did not create any cuda tensors before moving the model so your explanation is certainly possible. Although I am surprised that cuda related libraries will take over a GB of CPU memory.</p>

perekinsun: <p>This is my full code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from memory_profiler import profile
import gc

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(10, 5)
        self.fc2 = nn.Linear(5, 1)
    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        return x
@profile
def m():
    model = Net()
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    model = model.to(device)
    del model
    gc.collect()

m()
</code></pre>
<p>I met the same problem. And I wonder if the library is so big.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/599d61537130f9542fb743643d162deff076216f.png" data-download-href="https://discuss.pytorch.org/uploads/default/599d61537130f9542fb743643d162deff076216f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/9/599d61537130f9542fb743643d162deff076216f.png" alt="image" data-base62-sha1="cMLGbgjuqLjRcvroldAy1DCF6sn" width="690" height="303" data-dominant-color="313131"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1523×671 23.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>