Change weights only if loss decreases

username: vainaijr
cooked: <p>is anything wrong if I change weights only when loss decreases, otherwise let them stay as they are.</p>
<pre><code class="lang-auto">loss_stored = 1000000000000
for i in range(10):
  optimizer.zero_grad()
  input = torch.randn(3, 3, requires_grad=True)
  loss = model(input).abs().sum()
  loss.backward()
  print(list(model.parameters())[0].sum())
  if loss &lt; loss_stored:
    loss_stored = loss
    optimizer.step()
  print(list(model.parameters())[0].sum())
  print(loss)
</code></pre>

username: JuanFMontesinos
cooked: <p>It makes not sense at all. Some samples may be harder than other, thus, their loss will be harder.<br>
Dummy example:<br>
Classify this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ecde7f2d9c77b5067294e3a6dd512a72995a1e91.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ecde7f2d9c77b5067294e3a6dd512a72995a1e91" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ecde7f2d9c77b5067294e3a6dd512a72995a1e91_2_489x500.jpeg" alt="image" data-base62-sha1="xNrnzuaQbzmMeoDIg5PH76rsY1j" width="489" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ecde7f2d9c77b5067294e3a6dd512a72995a1e91_2_489x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ecde7f2d9c77b5067294e3a6dd512a72995a1e91_2_733x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ecde7f2d9c77b5067294e3a6dd512a72995a1e91.jpeg 2x" data-dominant-color="918D85"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">940×960 212 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Now classify this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b691b6599c857639aec8d9be51cc82e8967c0a0d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b691b6599c857639aec8d9be51cc82e8967c0a0d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b691b6599c857639aec8d9be51cc82e8967c0a0d_2_375x500.jpeg" alt="image" data-base62-sha1="q355nDHAtcEP6poBMblESt3cbxP" width="375" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b691b6599c857639aec8d9be51cc82e8967c0a0d_2_375x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b691b6599c857639aec8d9be51cc82e8967c0a0d_2_562x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b691b6599c857639aec8d9be51cc82e8967c0a0d.jpeg 2x" data-dominant-color="9A8D55"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">600×800 109 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Lastly classify this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f56611fad5e7e8bff218b973515c5c373f534e0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6f56611fad5e7e8bff218b973515c5c373f534e0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f56611fad5e7e8bff218b973515c5c373f534e0_2_540x500.jpeg" alt="image" data-base62-sha1="fSW46vlRYvwzYKqFOlt8Gq6Wtri" width="540" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f56611fad5e7e8bff218b973515c5c373f534e0_2_540x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f56611fad5e7e8bff218b973515c5c373f534e0.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f56611fad5e7e8bff218b973515c5c373f534e0.jpeg 2x" data-dominant-color="5F544D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">640×592 76.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Last case is harder, loss will be higher but you aren’t teaching the network to discern that as you aren’t updating weights.</p>