Pytorch cuda out of memory?

username: vymao
cooked: <p>I found the following error that I am getting on <a href="https://github.com/pytorch/pytorch/issues/16417" rel="noopener nofollow ugc">Github</a>, but it doesn’t seem to have been solved. This is the traceback:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/path/to/run.py", line 380, in &lt;module&gt;
    loss = train()
  File "/path/to/run.py", line 64, in train
    out = model(data).view(-1)
  File "/n/scratch3/users/v/vym1/nn/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/path/to/WLGNN.py", line 114, in forward
    xs += [torch.tanh(conv(xs[-1], edge_index, edge_weight))]
  File "/n/scratch3/users/v/vym1/nn/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/n/scratch3/users/v/vym1/nn/lib/python3.7/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 169, in forward
    size=None)
  File "/n/scratch3/users/v/vym1/nn/lib/python3.7/site-packages/torch_geometric/nn/conv/message_passing.py", line 236, in propagate
    out = self.message(**msg_kwargs)
  File "/n/scratch3/users/v/vym1/nn/lib/python3.7/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 177, in message
    return edge_weight.view(-1, 1) * x_j
RuntimeError: CUDA out of memory. Tried to allocate 3.37 GiB (GPU 0; 11.17 GiB total capacity; 5.35 GiB already allocated; 1.46 GiB free; 9.29 GiB reserved in total by PyTorch)
</code></pre>
<p>I have 100GB of memory allocated, and it isn’t clear to me why PyTorch can’t allocate it when it has only allocated a small fraction of the memory in total.</p>

username: Scott_Hoang
cooked: <p>Do you have 100GB on a single GPU? (where can I buy one?)</p>

username: vymao
cooked: <p>I’m working on a cluster and allocated 100 GB of memory. Is this not available for the GPU?</p>

username: siyuchen95
cooked: <p>If the GPU is full, there’s no other way but to get a bigger one.<br>
However, if you expect this to work normally, check if you have the tensors in their correct dimension. Sometimes broadcasting can give you a super big matrix if you are not careful.</p>

username: Scott_Hoang
cooked: <p>You can have 100Gb allocated, but they are spread out across x number of GPUs. You got to make your batch size to fit per GPU.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="vymao" data-post="3" data-topic="102713" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vymao/48/30929_2.png" class="avatar"> vymao:</div>
<blockquote>
<p>I’m working on a cluster and allocated 100 GB of memory. Is this not available for the GPU?</p>
</blockquote>
</aside>
<p>Your current GPU has a capacity of <code>11.17GiB</code> as given in the error message.<br>
Are you allocating 100GB on your system RAM or are you combining the memory of all GPUs?</p>
<p>As explained by others you would have to either reduce the batch size (or model) or trade compute for memory via <code>torch.utils.checkpoint</code>.</p>