Triplet Loss Backprop Equations

Ahmad_Nedal: <p>I am implementing a triplet network in Pytorch where I use 3 forward passes and a single backward pass, similar to what is described <a href="https://discuss.pytorch.org/t/multiple-model-forward-followed-by-one-loss-backward/20868">here</a>. My model is trained by optimizing triplet loss. After running the code, everything works as expected.</p>
<p>However, I am still confused about the computation of backprop in this implementation. Specifically, as PyTorch accumulates the derivatives, the gradients of the triplet loss w.r.t. to the last linear layer (embedding) (shown <a href="https://stackoverflow.com/questions/33330779/whats-the-triplet-loss-back-propagation-gradient-formula" rel="noopener nofollow ugc">here</a>) always add up to zero. Of course, this cannot be true as the network eventually learns meaningful embeddings.</p>
<p>Any explanation for this fallacy?</p>

albanD: <p>Hi,</p>
<p>This is surprising indeed.<br>
Do you see this gradient being non-zero after some time during training?<br>
Also if you ignore gradients from one branch, is it non-zero?</p>
<pre><code class="lang-python"># To ignore gradient from one branch, just detach it:
base = net(inp_base)
pos = net(inp_pos)
neg = net(inp_neg)

# To ignore gradients from the neg branch
neg = neg.detach()

loss = crit(base, pos, neg)
loss.backward()
</code></pre>

Ahmad_Nedal: <p>Hi <a class="mention" href="/u/alband">@albanD</a>, thanks a lot for replying!</p>
<p>I attached two photos showing the gradients of the embedding layer once without detaching any embedding and once after detaching the pos embeddings. I also fixed the seed and the model (both are at same epoch and optimizer state). The model is trained sufficiently long at the time of showing the gradients. The gradients are not equal but non-zero, which is expected.</p>
<p>What I do not understand is that adding up the gradients (the link is above) gives zero in theory. This means the network will not update its parameters. However, this is luckily not the case. Now I am simply confused how triplet loss is actually optimized given my implementation. (Ps. I might have overlooked a detail in the theory part as well.)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/920cd862919be3594e4332dc5e71b76a21af28fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/920cd862919be3594e4332dc5e71b76a21af28fb" title="posOff"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/920cd862919be3594e4332dc5e71b76a21af28fb.png" alt="posOff" data-base62-sha1="kQ1elWBBRygAUUzSgoSVkoMQLXl" width="690" height="203" data-dominant-color="060606"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">posOff</span><span class="informations">1328×392 14.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81a517ef64dfa25b54ac6a275f7cdd4c1c415bf6.png" data-download-href="https://discuss.pytorch.org/uploads/default/81a517ef64dfa25b54ac6a275f7cdd4c1c415bf6" title="allOn"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81a517ef64dfa25b54ac6a275f7cdd4c1c415bf6.png" alt="allOn" data-base62-sha1="iuTiTVNHwENJeMqnMzSHUVLCiiO" width="690" height="222" data-dominant-color="070707"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">allOn</span><span class="informations">1208×390 14.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <aside class="quote no-group" data-username="Ahmad_Nedal" data-post="3" data-topic="103269">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ahmad_nedal/48/31187_2.png" class="avatar"> Ahmad_Nedal:</div>
<blockquote>
<p>What I do not understand is that adding up the gradients (the link is above) gives zero in theory.</p>
</blockquote>
</aside>
<p>I am not sure to follow that. It gives 0 only in some cases no? In others it depends on the difference between the branches right?</p>

Ahmad_Nedal: <p>I am sorry if that was confusing. The code <strong>never</strong> gives 0 gradient at any case. But I am wondering how exactly PyTorch gets the gradients.</p>
<p>Isn’t it that when I do loss.backward(), the gradients will be accumulated over the base, pos and neg instances? If so, the weights at the embedding layer are actually the same tensors, thus I expect they will be <strong>updated by the sum of gradients</strong> (times const). Surprisingly, in theory, this sum turns out to be always zero, as shown in the figure. But since I do not get zeros, there must be something that I do not understand.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/1916ae8356640d1ec1de47553ee9acd1a3f5b2f9.png" data-download-href="https://discuss.pytorch.org/uploads/default/1916ae8356640d1ec1de47553ee9acd1a3f5b2f9" title="Ltrip_grad"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/1916ae8356640d1ec1de47553ee9acd1a3f5b2f9_2_690x309.png" alt="Ltrip_grad" data-base62-sha1="3zWwZf5RSsMkS9gZnWRkrURAdZf" width="690" height="309" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/1916ae8356640d1ec1de47553ee9acd1a3f5b2f9_2_690x309.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/1916ae8356640d1ec1de47553ee9acd1a3f5b2f9_2_1035x463.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/1916ae8356640d1ec1de47553ee9acd1a3f5b2f9_2_1380x618.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Ltrip_grad</span><span class="informations">1434×644 40.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Right but these formulas give the gradients wrt to the output of the net, not the weights. And you never actually sum the gradient contribution of the outputs (because these are different Tensors), only the weights and actually shared and so gradients are accumulated.</p>
<p>Does that explain what you see?</p>

Ahmad_Nedal: <p>Yes it does explain a lot, thank you very much <a class="mention" href="/u/alband">@albanD</a>!</p>
<p>I was implicitly assuming that df/dW_last are the same for all the three instances because the weights are identical. It is not the case; the derivatives are input-dependent. So instead of having a <strong>weighted sum</strong> of the 3 grads, I was getting Const * the sum of grads (which is 0).</p>