CUDA out of memory. Tried to allocate 16.00 GiB (GPU 0; 10.76 GiB total capacity; 536.52 MiB already allocated; 8.83 GiB free; 736.00 MiB reserved in total by PyTorch)

username: koren
cooked: <p>Hi, I successfully ran the code in this project: <a href="https://github.com/erichson/NFM" class="inline-onebox" rel="noopener nofollow ugc">GitHub - erichson/NFM</a> .<br>
For some reason when I tried to add this function, which doesn’t really do anything, I got CUDA out of memory.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9cdf5ccf45d7335d7db271a2770f020d8584524.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b9cdf5ccf45d7335d7db271a2770f020d8584524" title="some_func"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9cdf5ccf45d7335d7db271a2770f020d8584524_2_690x136.jpeg" alt="some_func" data-base62-sha1="qvHBb0trsTC9DZyOE2GQfnPSw16" width="690" height="136" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/9/b9cdf5ccf45d7335d7db271a2770f020d8584524_2_690x136.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9cdf5ccf45d7335d7db271a2770f020d8584524.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/9/b9cdf5ccf45d7335d7db271a2770f020d8584524.jpeg 2x" data-dominant-color="323231"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">some_func</span><span class="informations">804×159 24.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I was calling this function in every mini-batch iteration in the training, and after the first iteration, I got the error. (in the loss.backward() calling)<br>
In addition, if I’m removing lines 43-44 (in the picture), the code is working well.<br>
What could be the cause of this?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="koren" data-post="1" data-topic="150228">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/858c86/48.png" class="avatar"> koren:</div>
<blockquote>
<p>For some reason when I tried to add this function, which doesn’t really do anything,</p>
</blockquote>
</aside>
<p>I’m not sure how to understand this description, as your method is clearly calling into <code>torch.linalg.svd</code> and multiple <code>torch.mm</code> ops.</p>
<aside class="quote no-group" data-username="koren" data-post="1" data-topic="150228">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/858c86/48.png" class="avatar"> koren:</div>
<blockquote>
<p>In addition, if I’m removing lines 43-44 (in the picture), the code is working well.</p>
</blockquote>
</aside>
<p>I would assume that lines 43-44 try to allocate the 16GB or memory. Did you check the memory requirement for these calls (e.g. you could use the input shapes to calculate the expected output shape for the matmul etc.)?</p>

username: koren
cooked: <p>Thanks for the fast replay.</p>
<p>the input shape is [2,65536] (2 is the batch size)<br>
the shape of the ‘u’ variable is [2,2]<br>
the shape of the ‘s’ variable is [2,2]<br>
the shape of the ‘v’ variable is [2,65536]<br>
It looks like the multiplication of ‘u’ ‘s’ and ‘v’ is trying to allocate the 16GB, but it’s a bit wired because the shapes of the variables are not so big.<br>
and in addition for any batch size (not only 2), it’s trying to allocate 16GB.</p>

username: ptrblck
cooked: <p>I think the shape of <code>s</code> would be <code>[2]</code>, but nevertheless I cannot reproduce the issue using the input shape and get a memory allocation of ~2MB:</p>
<pre><code class="lang-python">device = 'cuda'
print(torch.cuda.memory_allocated() / 1024**3)
# 0.0

z = torch.randn(2, 65536, device=device)
print(torch.cuda.memory_allocated() / 1024**3)
# 0.00048828125

u, s, v = torch.linalg.svd(z, full_matrices=False)
print(u.shape)
# torch.Size([2, 2])
print(s.shape)
# torch.Size([2])
print(v.shape)
# torch.Size([2, 65536])
print(torch.cuda.memory_allocated() / 1024**3)
# 0.0009775161743164062

tmp = torch.mm(torch.diag(s), v)
print(tmp.shape)
# torch.Size([2, 65536])
print(torch.cuda.memory_allocated() / 1024**3)
# 0.0014657974243164062

out = torch.mm(u, tmp)
print(out.shape)
# torch.Size([2, 65536])
print(torch.cuda.memory_allocated() / 1024**3)
# 0.0019540786743164062
</code></pre>

username: koren
cooked: <p>I print the Cuda memory allocated as you did, and I got similar results.<br>
My problem was raised when the loss.backward() function was called.<br>
Probably for some reason the back prop trying to allocate 16GB, no matter what is the batch size.</p>

username: ptrblck
cooked: <p>Maybe in another part of the code, but not in the posted one.<br>
You can set the <code>requires_grad</code> attribute of the input to <code>True</code>, reduce the output, and call <code>.backward()</code> on it which yields:</p>
<pre><code class="lang-python">out.mean().backward()
print(torch.cuda.memory_allocated() / 1024**3)
# 0.0024423599243164062
</code></pre>