PyTorch Gradients

ritchieng: Normally when we’re doing backprop we would do the following: <SCODE>loss.backward()  # This calculates the gradients
optimizer.step()   # This updates the net
<ECODE>

albanD: Hi, If you want to accumulate gradients from multiple backwards, you can just backward multiple times without resetting the gradients: <SCODE>optimizer.zero_grad()

for i in range(minibatch):
    loss = model(batch_data[i])
    loss.backward()

optimizer.step()
<ECODE>

apaszke: Not that loss functions average over the batch size, so if you do multiple backprops you might need to average over the number of the for loop iterations.

ritchieng: 

apaszke: Yes. Divide it by the number of iterations in the for loop.

Veril: Is there a performance penalty to running backward() multiple times vs. just using a bigger batch (in situations where its an option)?

apaszke: Yes, it’s always going to be slower, but it’s a tradeoff between performance and memory usage. Try to do as few iterations as you can (you can split each batch into smaller sub-batches, so that they nearly fill up the memory).

Zijun_Wei: But it doesn’t affect the performance (accuracy) of the model using the accumulated gradient method right?

jdhao: <SCODE>num_epoch = 10
real_batchsize = 100 # I want to update weight every `real_batchsize`
for epoch in range(num_epoch):
    total_loss = Variable(torch.zeros(1).cuda(), requires_grad=True)
    for batch_idx, (data, target) in enumerate(train_loader):

        data, target = Variable(data.cuda()), Variable(target.cuda())
        output = net(data)

        total_loss = total_loss + loss

        if batch_idx % real_batchsize == 0:

            ave_loss = total_loss/real_batchsize
            ave_loss.backward()
            optimizer.step()
            
            total_loss.data.zero_()
            optimizer.zero_grad()

<ECODE> The above code will produce an error: <SCODE> total_loss.data.zero_()
<ECODE> by <SCODE>total_loss = Variable(torch.zeros(1).cuda(), requires_grad=True)
<ECODE> Now it seems to work. But I am 100% sure if I have done it right. Can you give any advice on how to do it properly?

albanD: I think a simpler way to do this would be: <SCODE>num_epoch = 10
real_batchsize = 100 # I want to update weight every `real_batchsize`
for epoch in range(num_epoch):
    total_loss = 0
    for batch_idx, (data, target) in enumerate(train_loader):

        data, target = Variable(data.cuda()), Variable(target.cuda())
        output = net(data)
        loss = crit(output, target)
        total_loss = total_loss + loss

        if batch_idx % real_batchsize == 0:

            ave_loss = total_loss/real_batchsize
            optimizer.zero_grad()
            ave_loss.backward()
            optimizer.step()
            total_loss = 0
<ECODE>

jdhao: Can we directly add an int value with torch Variable? Will the average loss and backward gradient calculation be on GPU?

albanD: 

fbcotter: Just a note (which you may be aware of): When you call ave_loss.backward() you propagate errors with respect to your (correct) loss, but these errors are functions of what the activations are when the .backward method was called. Since you’ve thrown away all but the last 10 samples, you are making the assumption that the first 90 samples were the same as the last 10.

matthieuheitz: <SCODE>x = tensor(x0,requires_grad=True)
loss = 0
for i in range(inputs.numel()):  # For my apps, it's between 5 and 50.
	rec = f(x,i)
	loss += loss_func(inputs[i], rec)
loss.backward()
g = x.grad
<ECODE> <SCODE>x = tensor(x0, requires_grad=True)
loss = 0
for i in range(inputs.numel()):
	rec = f(x,i)
	loss += loss_func(inputs[i], rec)
	loss.backward()
g = x.grad
<ECODE> <SCODE>x = tensor(x0, requires_grad=True, retain_graph=True)
loss = 0
for i in range(inputs.numel()):
	rec = f(x,i)
	loss += loss_func(inputs[i], rec)
	loss.backward()
g = x.grad
<ECODE>

albanD: Hi,

MariosOreo: <SCODE>state_dict = model.state_dict()[key]
<ECODE> Thanks in advance

albanD: optimizer.state_dict() is dependant on the optimizer itself. It will contain whatever is needed for this optimizer to continue working as if it was not stopped (saving things like momentum terms or statistics).

matthieuheitz: 

albanD: Yes it will retain them until you actually destroy the computational graph. This will increase the peak memory usage during the backward.

karl7: