PyTorch Gradients

ritchieng: <p>Normally when we’re doing backprop we would do the following:</p>
<pre><code class="lang-auto">loss.backward()  # This calculates the gradients
optimizer.step()   # This updates the net
</code></pre>
<p>However, what if I wish to accumulate the gradients? Meaning I want to run various <code>loss.backward()</code> multiple times first and accumulate the gradients before applying them in <code>optimizer.step()</code>.</p>
<ol>
<li>What do you recommend for storing the gradients? <code>grads = loss.backward()</code>?</li>
<li>How do I feed the accumulated gradients to <code>optimizer.step</code>?</li>
</ol>

albanD: <p>Hi,</p>
<p>When you perform <code>loss.backward()</code> the gradients are accumulated inplace in each Variable that requires gradient.<br>
That is why you need to perform <code>optimizer.zero_grad()</code> before each backward.</p>
<p>If you want to accumulate gradients from multiple backwards, you can just backward multiple times without resetting the gradients:</p>
<pre><code class="lang-auto">optimizer.zero_grad()

for i in range(minibatch):
    loss = model(batch_data[i])
    loss.backward()

optimizer.step()
</code></pre>

apaszke: <p>Not that loss functions average over the batch size, so if you do multiple backprops you might need to average over the number of the for loop iterations.</p>

ritchieng: <p>How do you average the related variables holding the gradients? <code>loss / len(minibatch)</code>?</p>

apaszke: <p>Yes. Divide it by the number of iterations in the for loop.</p>

Veril: <p>Is there a performance penalty to running backward() multiple times vs. just using a bigger batch (in situations where its an option)?</p>

apaszke: <p>Yes, it’s always going to be slower, but it’s a tradeoff between performance and memory usage. Try to do as few iterations as you can (you can split each batch into smaller sub-batches, so that they nearly fill up the memory).</p>

Zijun_Wei: <p>But it doesn’t affect the performance (accuracy) of the model using the accumulated gradient method right?</p>

jdhao: <p><a class="mention" href="/u/apaszke">@apaszke</a>, <a class="mention" href="/u/alband">@albanD</a>, I also tried to achieve this. As you have said, doing backward() for each sample is slow compared to accumulating the loss, doing one average and then doing backward.  Here is my code</p>
<pre><code class="lang-auto">num_epoch = 10
real_batchsize = 100 # I want to update weight every `real_batchsize`
for epoch in range(num_epoch):
    total_loss = Variable(torch.zeros(1).cuda(), requires_grad=True)
    for batch_idx, (data, target) in enumerate(train_loader):

        data, target = Variable(data.cuda()), Variable(target.cuda())
        output = net(data)

        total_loss = total_loss + loss

        if batch_idx % real_batchsize == 0:

            ave_loss = total_loss/real_batchsize
            ave_loss.backward()
            optimizer.step()
            
            total_loss.data.zero_()
            optimizer.zero_grad()

</code></pre>
<p>The above code will produce an error:</p>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</p>
</blockquote>
<p>I have looked up this issue, but I am not very clear. Just feels that we need a new <code>total_loss</code> after each weight update, so I replace the line</p>
<pre><code class="lang-auto"> total_loss.data.zero_()
</code></pre>
<p>by</p>
<pre><code class="lang-auto">total_loss = Variable(torch.zeros(1).cuda(), requires_grad=True)
</code></pre>
<p>Now it seems to work. But I am 100% sure if I have done it right. Can you give any advice on how to do it properly?</p>

albanD: <p>I think a simpler way to do this would be:</p>
<pre><code class="lang-auto">num_epoch = 10
real_batchsize = 100 # I want to update weight every `real_batchsize`
for epoch in range(num_epoch):
    total_loss = 0
    for batch_idx, (data, target) in enumerate(train_loader):

        data, target = Variable(data.cuda()), Variable(target.cuda())
        output = net(data)
        loss = crit(output, target)
        total_loss = total_loss + loss

        if batch_idx % real_batchsize == 0:

            ave_loss = total_loss/real_batchsize
            optimizer.zero_grad()
            ave_loss.backward()
            optimizer.step()
            total_loss = 0
</code></pre>

jdhao: <p>Can we directly add an int value with torch Variable? Will the average loss and backward gradient calculation be on GPU?</p>

albanD: <p>yes you can add a python number to a <code>Variable</code> (or a <code>Tensor</code>) and the output is going to be a <code>Variable</code> (or a <code>Tensor</code>) of the same type as the input (so if it was on gpu, the output will be on gpu).</p>

fbcotter: <p>Just a note (which you may be aware of):</p>
<p>Doing this is not the same as <a class="mention" href="/u/alband">@albanD</a> 's first answer.</p>
<p>When you call ave_loss.backward() you propagate errors with respect to your (correct) loss, but these errors are functions of what the activations are when the .backward method was called. Since you’ve thrown away all but the last 10 samples, you are making the assumption that the first 90 samples were the same as the last 10.</p>

matthieuheitz: <p>Regarding this tradeoff, do you save time/memory by using <code>retain_graph=True</code> in that situation ?<br>
For example, my current code looks like this:</p>
<pre><code class="lang-auto">x = tensor(x0,requires_grad=True)
loss = 0
for i in range(inputs.numel()):  # For my apps, it's between 5 and 50.
	rec = f(x,i)
	loss += loss_func(inputs[i], rec)
loss.backward()
g = x.grad
</code></pre>
<p>My current problem is that the computational graph takes too much memory because the function <code>f</code> does a lot of computation. So a solution would be to do as <a class="mention" href="/u/alband">@albanD</a> suggested:</p>
<pre><code class="lang-auto">x = tensor(x0, requires_grad=True)
loss = 0
for i in range(inputs.numel()):
	rec = f(x,i)
	loss += loss_func(inputs[i], rec)
	loss.backward()
g = x.grad
</code></pre>
<p>But I feel like the computationnal graph for each iteration of that loop is the same, it’s just the numbers on which we apply it that change. So maybe we could reuse the previous iteration’s graph (by specifying <code>retain_graph=True</code>), could that save some time ? If not, what would happen (in terms of time/memory loss/gain) ?</p>
<pre><code class="lang-auto">x = tensor(x0, requires_grad=True, retain_graph=True)
loss = 0
for i in range(inputs.numel()):
	rec = f(x,i)
	loss += loss_func(inputs[i], rec)
	loss.backward()
g = x.grad
</code></pre>

albanD: <p>Hi,</p>
<p>This is the expected use case: the graph structure is mostly the same, only the values change. The whole framework is built to make this use case efficient.<br>
And you cannot reuse the graph as the graph is associated with the values of each Tensor and so if the values change, you need to recreate it (which is cheap).</p>

MariosOreo: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>I have a question about the <code>params</code> in <code>optimizer.state_dict()</code> and weights in <code>model.state_dict()</code><br>
From the source code, I found that the grad is computed during <code>backward()</code> and the weights are updated during the <code>optimizer.step()</code>.<br>
I try to output the <code>model.state_dict</code> and <code>optimizer.state_dict</code> before and after <code>backward()</code> and <code>optimizer.step()</code> respectively.<br>
If I save the state_dict by</p>
<pre><code class="lang-auto">state_dict = model.state_dict()[key]
</code></pre>
<p>And they are the same, before and after <code>backward()</code>, it means that assignment operation of tensors is shared memory operation?<br>
Another question is what does <code>params</code> in <code>optimizer.state_dict</code> mean? There is not any change before and after <code>backward</code> and <code>optimizer.step</code>, does it mean an address to the weight?</p>
<p>Thanks in advance</p>

albanD: <p><code>.backward()</code> will just populate the <code>.grad</code> fields of the parameters. These gradients are not saved in the state dicts and so nothing will change there.<br>
after <code>opt.step()</code> the values of the parameters will be changed inplace. So if you want to see the difference before and after, you need to clone the original Tensor.</p>
<p>optimizer.state_dict() is dependant on the optimizer itself. It will contain whatever is needed for this optimizer to continue working as if it was not stopped (saving things like momentum terms or statistics).</p>

matthieuheitz: <p>Oh I see, I guess I was confused by the name <code>retain_graph</code>. I’ve searched a bit and see that it was called <code>retain_variables</code> before. So I guess if I use <code>retain_graph=True</code> while putting <code>loss.backward()</code> inside the for loop, it defeats the purpose of saving memory because it will keep in memory the temporary tensors needed for the previous gradient, right ?</p>

albanD: <p>Yes it will retain them until you actually destroy the computational graph. This will increase the peak memory usage during the backward.</p>

karl7: <p>sorry for reply to so old post, i encountered a problem recently:<br>
in Pytorch distribution code, how can i keep the gradient graph (autograd.grad) while using dist.all_reduce() or dist.all_gather()? to avoid the situation that I need to manually calculate the gradient, then backward.</p>