Understanding graphs and state

tomsercu: There are two questions remaining - the second question is more important. 1) No guarantee that second backward will fail? <SCODE>x = Variable(torch.ones(2,3), requires_grad=True)
y = x.mean(dim=1).squeeze() + 3 # size (2,)
z = y.pow(2).mean() # size 1
y.backward(torch.ones(2))
z.backward() # should fail! But only fails on second execution
y.backward(torch.ones(2)) # still fine, though we're calling it for the second time
z.backward() # this fails (finally!)
<ECODE> 2) Using a net twice on the same input Variable makes a new graph with new state? <SCODE>out = net(inp)
out2 = net(inp) # same input
out.backward(torch.ones(1,1,2,2))
out2.backward(torch.ones(1,1,2,2)) # doesnt fail -> has a different state than the first fw pass?!
<ECODE> Am I right to think that fw-passing the same variable twice constructs a second graph, keeping the state of the first graph around? The problem I see with this design is that often (during testing, or when you detach() to cut off gradients, or anytime you add an extra operation just for monitoring) there’s just a fw-pass on part of the graph - so is that state then kept around forever and just starts consuming more memory on every new fw-pass of the same variable? I understand that the volatile flag is probably introduced for this problem and I see it’s used during testing in most example code. 
fake = netG(noise).detach() to avoid bpropping through netG  https://github.com/pytorch/examples/blob/master/dcgan/main.py#L216 43
 test on non-volatile variables: https://github.com/pytorch/examples/blob/master/super_resolution/main.py#L74 19
 If you finetune only top layers of a feedforward net, bottom layers see only fw-passes But in general, if I understand this design correctly, this means anytime you have a part of a network which isn’t backpropped through, you need to supply volatile flag? Then when you use that intermediate volatile variable in another part of the network which is backpropped through, you need to re-wrap and turn volatile off?

apaszke: Now, here’s some description on when do we keep the state around:

tomsercu: Thanks for the elaborate answer, the graph going out of scope with the output variable is the essential part I was missing here. Let me know if you think it’s useful to make the  notebook with your answer into a full tutorial, I think these autograd graph/state mechanics are a bit underdocumented atm. Or maybe some explanation could be added to the autograd note in the docs.

apaszke: I agree the materials we have right now aren’t very detailed but we didn’ thave a lot of time to expand them. If you’d have a moment to write that down and submit a notebook or a PR to the notes I’ll merge them. Thanks!

pietromarchesi: 

Nabarun_Goswami: Hi, <SCODE>fake = netG(inputs)
Loss1 = criterion(netD1(fake), real_label)
Loss2 =  criterion(netD2(fake), real_label)
Loss3 =  criterion(netD3(fake), real_label)
Loss_other = criterion_other(fake, target)
Loss = Loss1 + Loss2 + Loss3 + Loss_other
Loss.backward()
<ECODE> does this create the graphs for each of the netDs? Will be wrong if I did <SCODE>Loss1 = criterion(netD1(fake).detach(), real_label)
Loss2 =  criterion(netD2(fake).detach(), real_label)
Loss3 =  criterion(netD3(fake).detach(), real_label)
Loss_other = criterion_other(fake, target)
Loss = Loss1 + Loss2 + Loss3 + Loss_other
Loss.backward()
<ECODE> to save some memory, since I don’t need to backprop through netD? Will there be any difference in backpropping?

dzimm: Hi, if I understand you correctly, you want to train netD1 with loss1, netD3 with loss2 ,… and netG with loss_other ? <SCODE>Loss1 = criterion(netD1(fake.detach()), real_label)
Loss2 =  criterion(netD2(fake.detach()), real_label)
Loss3 =  criterion(netD3(fake.detach()), real_label)
Loss_other = criterion_other(fake, target)
Loss = Loss1 + Loss2 + Loss3 + Loss_other
Loss.backward()
<ECODE> , where you detach the fake, thus loss1, etc can be propagated back through netD1 etc (but still not though netG, if you want to propagate through netG and not netD1 you can try to set .requires_grad=False for all paramters in netD1, but not sure if it will work, since it only works on leaves).

Nabarun_Goswami: Hi, Actually no, don’t want to train netD* at all, all the losses are for training the netG, and all netD* are fixed. The last point of your reply kind of hit the point, I want to propagate through netG without propagating through netD*. .requires_grad = False looks promising, but I am not sure either, would be great if someone can clarify on that.

tomsercu: 

riccardosamperna: Adam, you said that: