What is wrong with my derivation - backpropagation

username: smani
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/787114433bc55bfce393f50f72b308cc934dc750.png" data-download-href="https://discuss.pytorch.org/uploads/default/787114433bc55bfce393f50f72b308cc934dc750" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/787114433bc55bfce393f50f72b308cc934dc750_2_690x304.png" alt="Capture" data-base62-sha1="hbtzb9rqMQmRcJEgonQhjMNPKog" width="690" height="304" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/787114433bc55bfce393f50f72b308cc934dc750_2_690x304.png, https://discuss.pytorch.org/uploads/default/original/2X/7/787114433bc55bfce393f50f72b308cc934dc750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/787114433bc55bfce393f50f72b308cc934dc750.png 2x" data-dominant-color="F5F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">836×369 35.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: KFrank
cooked: <p>Hello smani -</p>
<aside class="quote no-group quote-modified" data-username="smani" data-post="1" data-topic="47038" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smani/48/36520_2.png" class="avatar"> smani:</div>
<blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/787114433bc55bfce393f50f72b308cc934dc750.png" data-download-href="https://discuss.pytorch.org/uploads/default/787114433bc55bfce393f50f72b308cc934dc750" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/787114433bc55bfce393f50f72b308cc934dc750_2_690x304.png" alt="Capture" data-base62-sha1="hbtzb9rqMQmRcJEgonQhjMNPKog" width="690" height="304" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/787114433bc55bfce393f50f72b308cc934dc750_2_690x304.png, https://discuss.pytorch.org/uploads/default/original/2X/7/787114433bc55bfce393f50f72b308cc934dc750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/787114433bc55bfce393f50f72b308cc934dc750.png 2x" data-dominant-color="F5F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">836×369 35.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</blockquote>
</aside>
<p>First, could you please post text rather than screenshots?<br>
(It makes things searchable, lets screen readers work, etc.)<br>
If you need to post an equation that you can’t format adequately<br>
using the forum’s markdown, you could post a screenshot of<br>
that, but do then refer to and describe the equation in your text.</p>
<p>I haven’t verified your analytic computation of the gradient;<br>
I’ll take your word for it.</p>
<p>But I think the issue is the following:</p>
<p>You say “In my understanding when the loss is zero there should<br>
not be any updates.”  In general, this isn’t correct.  It’s true that we<br>
often work with loss functions that are non-negative (never less<br>
than zero), and often “loss = 0” implies “true solution found.”  But<br>
the value of the loss function has no particular meaning in the<br>
pytorch framework.  It is just something you minimize to train<br>
your network.  Nothing prevents your loss function from having<br>
a minimum of less than zero, and nothing prevents pytorch from<br>
finding that minimum.</p>
<p>In gradient descent, the (negative of the) gradient tells you the<br>
direction to move in parameter space to make your loss smaller.<br>
(That is, algebraically smaller – less positive, which is the same<br>
as more negative.)</p>
<p>Your example illustrates this general point.  Your loss function<br>
has the constant lambda in it.  But the locations of any minima<br>
of your loss function don’t depend on lambda, the gradient is<br>
independent of lambda, and the optimization algorithm<br>
(presumably gradient descent) doesn’t care about lambda.</p>
<p>If your loss function happens to be zero for some particular<br>
value of lambda, training won’t stop, nor should it – the fact<br>
that your loss was zero was merely an artifact of that particular<br>
value of lambda.</p>
<p>Good luck.</p>
<p>K. Frank</p>