Multiple forward before backward,the loss depend on one forward but the train result depend on all forward

username: Zhao-zw
cooked: <p>the model training code as follows:<br>
optimizer.zero_grad()<br>
feat = model_w(x2_w)<br>
loss, p_inds, n_inds, dist_ap, dist_an, dist_mat = global_loss(tri_loss, feat, y_w, normalize_feature= False)<br>
loss.backward()<br>
optimizer.step()<br>
if i add other forward, code as follows:<br>
optimizer.zero_grad()<br>
feat1 = model_w(G_result.detach())<br>
feat = model_w(x2_w)<br>
loss, p_inds, n_inds, dist_ap, dist_an, dist_mat = global_loss(tri_loss, feat, y_w, normalize_feature= False)<br>
loss.backward()<br>
optimizer.step()</p>
<p>the training result is different.<br>
Can anybody confirm that what I’m doing is correct? If it isn’t could someone indicate what the correct way would be?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>The additional forward pass using <code>G_result</code> will update the running stats in all batch norm layers, if you are using them in your model.<br>
This will most likely result in a different final accuracy.</p>

username: Zhao-zw
cooked: <aside class="quote no-group quote-modified" data-username="ptrblck" data-post="2" data-topic="67089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>则额外的正向传递使用将更新所</p>
</blockquote>
</aside>
<p>but rhe backward loss only denpend on the feat, if i need the feat1 and  want the G_result has no influence on the final accuracy,what should i do is correct?</p>

username: ptrblck
cooked: <p>The gradients will be calculated using the loss calculated by <code>feat</code>, that’s correct.<br>
However, as explained, the forward pass using <code>G_result</code> might update e.g. the running statistics of batch norm layers, if they are used in the model.<br>
If you don’t want to update these stats, you could set the model to <code>model.eval()</code> before passing <code>G_result</code> to it, and back to <code>model.train()</code> afterwards.</p>

username: Zhao-zw
cooked: <p>I do this as follow</p>
<pre><code>   optimizer.zero_grad() 
    model_w.eval()
    feat2 = model_w(G_result.detach())  
    model_w.train()
    feat = model_w(x2_w)
    loss, p_inds, n_inds, dist_ap, dist_an, dist_mat = global_loss(tri_loss, feat, y_w, normalize_feature= False) 
    loss.backward()
    optimizer.step()
</code></pre>
<p>the result is still wrong.Is there any other problem?</p>

username: ptrblck
cooked: <p>Could you explain, what makes the result wrong?</p>
<p>How large is the difference in final accuracy and what variance do you expect?<br>
I.e. if you run your code without the <code>G_result</code> forward pass (with different seeds, if you set them), how large is the variance in final accuracy then compared to your current run?</p>

username: Zhao-zw
cooked: <p>after training 10 epoch，if add the G_result forward pass,there will be a 10% difference in accuracy<br>
if without G_result(with different seeds),there will be a 1% difference in accuracy</p>

username: ptrblck
cooked: <p>How many runs did you run and how large is the variance?</p>

username: Zhao-zw
cooked: <p>Thank you.I know what you mean.the G_result is the GAN result，it has the different  data distribution with real data and I will try fixed the bn layers when training.</p>