Multiple model.forward followed by one loss.backward

username: blackberry
cooked: <p>I have seen some triplet loss implementations in PyTorch, which call <strong>model.forward</strong> on anchor, positive and negative images; then compute triplet loss and finally call <strong>loss.backward</strong> and <strong>optimizer.step</strong>, something like this:</p>
<pre><code class="lang-auto">anchor_embed = model.forward(anchor_images)
pos_embed = model.forward(pos_images)
neg_embed = model.forward(neg_images)

loss = triplet_loss.forward(anchor_embed, pos_embed, neg_embed)

optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<p>I think these implementations are not correct, since it will only consider the outputs of neg_embed = model.forward(neg_images) during bakpropagation, and disregard the anchor and positive images. As far as I know, in PyTorch, gradients are accumulated, but when we call model.forward it will overwrite the previous computations (which makes sense).</p>
<p>Then, the right implementation should concatenate the anchor,positive,negative images and perform a single model.forward, then compute loss and finally loss.backward and optimizer.step.</p>
<p>Is this right? Or am I missing something?</p>
<p>Thanks.</p>

username: ptrblck
cooked: <p>When you compute the forward pass using different inputs, each output will have its own computation graph attached to it. It’s not overwritten by the next call, so there should be no problem in your current approach.</p>
<p>You are right that the gradients are accumulated in consecutive backward calls, but that doesn’t seem to be the case in your example.</p>

username: blackberry
cooked: <p>Thanks.<br>
So, this means, three computation graphs will be computed in my example, one for each forward pass.<br>
Then, when we do loss.backward, the gradients will be computed for each computation graph using the same loss.<br>
Finally, the optimizer will update the weights using all three graphs.</p>
<p>Then, the question is, would this be equivalent to doing a single forward pass on the concatenated inputs and then doing loss.backward (which is the intended outcome)?<br>
Even if they are mathematically equivalent (? need to check), the three computation graphs make it inefficient.</p>

username: gfotedar
cooked: <p>Were you able to ever verify this?<br>
For my experiment the 2 are coming out to be not equivalent. Multiple forward passes tho slower and memory inefficient is leading to better score and I’m not at all sure why.</p>

username: blackberry
cooked: <p>In my experiments, the results were similar. There might be slight differences though. The difference in your case might be due to batch size: the effective batch size might be larger in multiple forward passes.</p>

username: Stefano_Savian
cooked: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/blackberry">@blackberry</a>, <a class="mention" href="/u/alband">@albanD</a> ,</p>
<p>I am in a similar situation. In the pseudo code below I try to do some multiple forward pass before computing the loss.</p>
<p>when grad == no. there is no gradient flowing in the second forward pass (see picture)<br>
when grad == yes the gradient is let flow through the second dynamic graph.</p>
<p>I have observed differences in these two cases.<br>
I went through many posts and cannot understand what is going on under the hood.</p>
<p>To my understanding when doing forward pass with gradient. 2 values per node(of the dynamic graph) are created. We have 2 backward graphs, but linked to the same model, thus<br>
1- How are the weights updated these cases?<br>
2- Same question as <a class="mention" href="/u/blackberry">@blackberry</a>:  are these approaches  equivalent to doing a single forward pass on the concatenated inputs and then doing loss.backward ?</p>
<p>Many thanks for your help!<br>
Stefano</p>
<pre><code class="lang-auto">inputs_a
inputs_b
inputs_c
if args.GRAD == 'no':
    with torch.no_grad():
        output_a = model(input_a)
        output_b = model(input_b)
        output_c = model(input_c)
elif args.GRAD == 'yes':
    output_a = model(input_a)
    output_b = model(input_b)
    output_c = model(input_c)

Compute LOSSa,LOSSb,LOSSc
LOSS = LOSSa+LOSSb+LOSSc
optimizer.zero_grad()
loss.backward()
optimizer.step()
</code></pre>
<blockquote>
<p>Blockquote</p>
</blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b436019c88865c9a30a77b602472d7d8ae9100d.png" data-download-href="https://discuss.pytorch.org/uploads/default/9b436019c88865c9a30a77b602472d7d8ae9100d" title="Picture 1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b436019c88865c9a30a77b602472d7d8ae9100d_2_690x375.png" alt="Picture 1" data-base62-sha1="m9wmg7V125Yw1fr8GSnJn4JFaeV" width="690" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b436019c88865c9a30a77b602472d7d8ae9100d_2_690x375.png, https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b436019c88865c9a30a77b602472d7d8ae9100d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b436019c88865c9a30a77b602472d7d8ae9100d.png 2x" data-dominant-color="C4C8D3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Picture 1</span><span class="informations">874×475 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: blackberry
cooked: <p>Hi <a class="mention" href="/u/stefano_savian">@Stefano_Savian</a>,</p>
<p>My understanding is that they are NOT always equivalent, but close. For example, if there are batchnorm layers in the network, the batchnorm parameters will be updated differently due to different batch sizes.<br>
There might be other (behind the scenes) differences too.<br>
So, if you have memory, it is probably better to do a single forward pass.</p>

username: Stefano_Savian
cooked: <p>Dear <a class="mention" href="/u/blackberry">@blackberry</a>  Many thanks for your answer!</p>
<p>I see, we are “artificially changing the batch size” when doing multiple forward propagations.<br>
I understand it would be better to stick to known behaviors, but I find it interesting to do multiple forward propagations because it allows for more flexibility of the network.</p>
<p>So referring to case args.GRAD == True of my previous example. And referring to the figure below representing one single learnable weight W0. How will W0 be updated after the backpropagation?</p>
<p>Steps.<br>
1 forward propagate inputs (will create G0 to populate the gradient)<br>
2 forward propagate other inputs (will create G0 to populate the gradient), creates new dynamic graph linked to the same network and rooted at LOSS<br>
3 Calculate backward (green dashed arrows). Now we have two gradient values assigned to the same weight</p>
<p>4 How is W0 Updated? What’s the mechanism?</p>
<p>Could you give me some more hints? Maybe a toy example to figure out what is going on?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/blackberry">@blackberry</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96aebcb401934b97c36412465b6cd751b7952cfd.png" data-download-href="https://discuss.pytorch.org/uploads/default/96aebcb401934b97c36412465b6cd751b7952cfd" title="FWD_grad"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96aebcb401934b97c36412465b6cd751b7952cfd_2_523x500.png" alt="FWD_grad" data-base62-sha1="luZZzzfkPK4ovl6Y0Qw54rna2zX" width="523" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96aebcb401934b97c36412465b6cd751b7952cfd_2_523x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/6/96aebcb401934b97c36412465b6cd751b7952cfd.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/6/96aebcb401934b97c36412465b6cd751b7952cfd.png 2x" data-dominant-color="645F65"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">FWD_grad</span><span class="informations">572×546 25.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi,</p>
<p>The two <code>W0</code> there are actually a single Tensor that is used twice.<br>
So just like any other case where you re-use a Tensor, the mathematical result is to accumulate the contributions for each use of it.<br>
If you use something like torchviz (<a href="https://github.com/szagoruyko/pytorchviz/">https://github.com/szagoruyko/pytorchviz/</a>) to print the computational graph, you will see that both evaluations point to a single circle for each weights.</p>