Running out of memory, when there seemingly should be enough

username: Nicolai_Frost_Jacobs
cooked: <p>I am running a latent space model on a network, the network itself dosent take up more than 2gb memory when stored in the local ram of the computer.</p>
<p>But when I try to run my pytorch model i get the following error:</p>
<p>RuntimeError: cuda runtime error (2) : out of memory at c:\anaconda2\conda-bld\pytorch_1519501749874\work\torch\lib\thc\generic/THCTensorMathPointwise.cu:343</p>
<p>I am running on a gtx 1080ti which should have more than enough memory to handle this.</p>
<pre><code class="lang-auto">def distMatrix(m):
    n = m.size(0)
    d = m.size(1)
    x = m.unsqueeze(1).expand(n, n, d)
    y = m.unsqueeze(0).expand(n, n, d)
    return torch.sqrt(torch.pow(x - y, 2).sum(2) + 1e-4)

def loss(tY):
    d = -distMatrix(tZ)+B
    sigmoidD = torch.sigmoid(d)
    reduce = tY*torch.log(sigmoidD)+(1-tY)*torch.log(1-sigmoidD)
    #remove diagonal
    reduce[torch.eye(n).byte().cuda()] = 0
    return -reduce.sum()

tZ = autograd.Variable(torch.cuda.FloatTensor(Z), requires_grad=True)
B = autograd.Variable(torch.cuda.FloatTensor([0]), requires_grad=True)

tY = autograd.Variable(torch.cuda.FloatTensor(Ytest), requires_grad=False)

losses = []
biases = []
testScore = []

learning_rate = 1e-3
epochs = 10000

sigmoid = np.vectorize(lambda x: math.exp(-np.logaddexp(0, -x)))

percentDone = 0
percent = 2
for i in range(1,epochs):
    count = (float(i)/epochs)*100
    if count % percent == 0:
        print(count)

    l = loss(tY)
    l.backward(retain_graph=True)
    losses.append(float(l))
    biases.append(B.data)
    tZ.data = tZ.data - learning_rate * tZ.grad.data
    B.data = B.data - learning_rate * B.grad.data
    
    tZ.grad.data.zero_()
    B.grad.data.zero_()
</code></pre>
<p>Z is 25059 by 2, Ytest is 25059 by 25059</p>
<p>Any hints on how to avoid the memory error would be greatly appreicated.</p>

username: ptrblck
cooked: <p>I think the loss calculation might blow up your memory usage.<br>
Let’s have a look at <code>distMatrix</code>.<br>
You are calling this function with <code>tZ</code>, which has dimensions <code>[25059, 2]</code> and therefore has 50118 elements.<br>
Then you are creating <code>x</code> and <code>y</code>. They have the same shape of <code>[25059, 25059, 2]</code>, so 1,255,906,962 elements each.</p>
<p>If we use 4 bytes (float32) for each element, we would have <code>(50118+2*1255906962)*4 / 1e9 ~ 10.05GB</code> memory consumption for this method.<br>
With the model itself you end up with more or less 12GB.<br>
For the training you also need to store some activations, which also take some memory.</p>
<p>How much memory does your GPU has?</p>

username: Nicolai_Frost_Jacobs
cooked: <p>The GPU has 11gb ram.<br>
I just try to optimize the cost function (loss), there is no neural network part in my solution, so I do not think I am storing any activations or weights, unless I am misunderstanding how it works. (which might be likely…)</p>
<p>How do I go about solving this problem? Mini-batching? I am new in this field, any hints, ideas or nudges would be a huge help! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: ptrblck
cooked: <p>The <code>loss.backward()</code> call calculates the gradients you are using in the following line with some intermediate values, which also needs some memory.</p>
<p>Batching might help. Is <code>25059</code> your batch size? Do you need it to be that large?<br>
Could you explain the use case a bit more? Maybe the solution is straightforward. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

username: Nicolai_Frost_Jacobs
cooked: <p>Its a latent space model for graphs: Takes in a graph and a Z vector with K dimensions and then translate it to latent space. This is done by optimizing the loss function (the one shown in the question).</p>
<p>I absolutely do not need batches of that size, making them smaller would be great, I am just at a loss of how to go about doing that.</p>

username: Nicolai_Frost_Jacobs
cooked: <p>furthermore when I run my code (on a small network that fits in ram) I get (relatively) low gpu usage:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/1e829765341ba368f787813850c3a26c1816263c.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e829765341ba368f787813850c3a26c1816263c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/1/1e829765341ba368f787813850c3a26c1816263c.png" alt="image" data-base62-sha1="4lU77zbixCwwCtmRe0dw3purKbO" width="553" height="500" data-dominant-color="F7F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">939×848 32.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What could be the cause of this? Shouldnt it be able to run at close to a 100%?</p>

username: bloos
cooked: <p>I think your screenshot answers your question. Your CPU utilization is very high. Most probably you are CPU bottlenecked.</p>