Two networks: one of the variables needed for gradient computation has been modified by an inplace operation:

Fnu_Ankur: <p>I am training two networks at a time.<br>
network 1: 3 Linear layers; fc1, fc2, fc3<br>
network2: 3 Linear Layers + 2 additional matrices.</p>
<p>Requires_grad = True for all the parameters of network 1<br>
network 2 copies all the parameters from network 1 using load_state_dict(…, strict=False), all the parameters(weights and bias) of network 2 don’t require grad. Only 2 additional matrices requires Grad.</p>
<p>While training:</p>
<ol>
<li>As soon as I update network 1, or take a backpropagation step, version of network1.fc3.weight.version=2</li>
<li>I make a copy of this parameters at network 2. version remains 2<br>
When I take a backpropagation step in network 2, its version is making computations the error.<br>
"is at version 2; expected version 1 "</li>
</ol>

ptrblck: <p>Are you seeing the same error also if <code>network1</code> is trained alone without the usage of <code>network2</code>?<br>
If not, could you post a minimal code snippet showing your training approach?</p>

Fnu_Ankur: <p>network1 alone works fine. Since, no copying/loading of parameters needs to be done.</p>
<p>Minimal Code:</p>
<pre><code class="lang-auto">
#Training Loop
network1.optimizer.zero_grad()
network2.optimizer.zero_grad()

#network2 loads all the weights and bias from network1
#require_grad = False for all network2 parameters except two matrices
network2.load_state_dict(network1.state_dict(), strict=False)        

#training _data
states, actions, rewards, states_, terminated, truncated 

q_pred = network1.forward(states)
s_pred = network2.forward(states)

q_target = target

loss_dqn = network1.loss(q_target, q_pred)
loss_dqn.backward()
network1.optimizer.step()


loss_sym = network2.loss(q_pred, s_pred)
loss_sym.backward()
network2.optimizer.step()
</code></pre>

Fnu_Ankur: <p>I get this runtime error:<br>
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad().</p>
<p>and upon updating retains_graph=True, throws an:<br>
one of the variables needed for gradient computation has been modified by an inplace operation error.</p>
<p>As soon as I run,<br>
<code>network2.load_state_dict(network1.state_dict(), strict=False)  </code><br>
All the weight and bias parameter’s version change from 1 to 2 while the version of network1 parameters remains at 1.<br>
This version 2 conflicts with the<br>
<code> loss_sym.backward()</code></p>

ptrblck: <p>I tried to create an executable code snippet using your pseudo-code but this works for me:</p>
<pre><code class="lang-python">network1 = models.resnet18()
network2 = models.resnet18()

optimizer1 = torch.optim.Adam(network1.parameters())
optimizer2 = torch.optim.Adam(network2.parameters())

for _ in range(10):
    #Training Loop
    optimizer1.zero_grad()
    optimizer2.zero_grad()
    
    #network2 loads all the weights and bias from network1
    #require_grad = False for all network2 parameters except two matrices
    network2.load_state_dict(network1.state_dict(), strict=True) 
    
    x = torch.randn(1, 3, 224, 224)
    
    q_pred = network1(x)
    s_pred = network2(x)
    
    loss_dqn = q_pred.mean()
    loss_dqn.backward()
    optimizer1.step()
    
    loss_sym = s_pred.mean()
    loss_sym.backward()
    optimizer2.step()
</code></pre>
<p>Could you check what the missing differences are?</p>

Fnu_Ankur: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for the prompt help.<br>
If you make dependence of one output with the second, I can reproduce the error.</p>
<pre><code class="lang-auto">network1 = models.resnet18()
network2 = models.resnet18()

optimizer1 = torch.optim.Adam(network1.parameters())
optimizer2 = torch.optim.Adam(network2.parameters())

loss1 = nn.MSELoss()
loss2 = nn.MSELoss()

for _ in range(10):
    #Training Loop
    optimizer1.zero_grad()
    optimizer2.zero_grad()
    
    #network2 loads all the weights and bias from network1
    #require_grad = False for all network2 parameters except two matrices
    network2.load_state_dict(network1.state_dict(), strict=True) 
    
    x = torch.randn(1, 3, 224, 224)
    
    q_pred = network1(x)
    s_pred = network2(x)
    q_pred_1 = torch.zeros_like(q_pred)


    loss_dqn = loss1(q_pred, q_pred_1)
    loss_dqn.backward()
    optimizer1.step()
    
    loss_sym = loss2(q_pred, s_pred)
    loss_sym.backward()
    optimizer2.step()
</code></pre>
<p>Both are different loss with different networks, yet load_state_dict messes with the version.</p>

ptrblck: <p>Your code will raise this error directly:</p>
<pre><code class="lang-python">RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed).
</code></pre>
<p>since you are trying to call <code>backward</code> twice using <code>q_pred</code>.<br>
Using <code>loss_dqn.backward(retain_graph=True)</code> fixes this error but will raise the expected:</p>
<pre><code class="lang-python">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation...
</code></pre>
<p>error since you are using stale forward activations in <code>network1</code>. This is caused by the <code>optimizer1.step()</code> call which will update the parameters and will thus make the stored intermediate forward activations created in <code>q_pred = network1(x)</code> stale.<br>
To fix this you can move the <code>opimizer1.step()</code> call down which will work.<br>
Note that this behavior is not changes if the <code>load_state_dict</code> is removed or kept as it’s just caused by an invalid parameter update.</p>

Fnu_Ankur: <p>Thank you, a lot. I appreciate your help!<br>
It clears a lot of things.</p>
<p>One additional query:<br>
Why does this work? Shouldn’t call from optimizer2.step() makes q_pred stale?</p>
<pre><code class="lang-auto">loss_sym = loss2(q_pred, s_pred)
loss_sym.backward(retain_graph = True)
optimizer2.step()

loss_dqn = loss1(q_pred, q_pred_1)
loss_dqn.backward()
optimizer1.step()
    
</code></pre>

ptrblck: <p>No, since <code>q_pred</code> was created by <code>network1</code> while <code>optimizer2.step()</code> updates the parameters of <code>network2</code>.</p>