Loss decreases but gets stuck after a few epochs and oscillates

username: nam1410
cooked: <p>Continuing the discussion from <a href="https://discuss.pytorch.org/t/loss-oscillating-and-not-minimizing/177796">Loss oscillating and not minimizing</a> (reference code snippet in previous post):</p>
<p>Hi,<br>
My previous challenge entailed the non-minimization and oscillation of loss during the training. Upon a thorough investigation, I identified a broken computation graph that impeded updating the model parameters. At present, the loss demonstrates a downward trend, but it becomes stagnant after a few epochs.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88fc1d90fcfe169c9feba1a54f375bc7aecbf076.png" data-download-href="https://discuss.pytorch.org/uploads/default/88fc1d90fcfe169c9feba1a54f375bc7aecbf076" title="Screenshot 2023-04-20 at 12.00.33 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88fc1d90fcfe169c9feba1a54f375bc7aecbf076_2_653x500.png" alt="Screenshot 2023-04-20 at 12.00.33 PM" data-base62-sha1="jxP5Dyl5LI8Wx8elTup6GSz4mq2" width="653" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88fc1d90fcfe169c9feba1a54f375bc7aecbf076_2_653x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88fc1d90fcfe169c9feba1a54f375bc7aecbf076_2_979x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/8/88fc1d90fcfe169c9feba1a54f375bc7aecbf076.png 2x" data-dominant-color="F6F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-04-20 at 12.00.33 PM</span><span class="informations">1088×832 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As can be observed from the graph, there is no significant improvement beyond approximately 125 epochs, remaining more or less constant afterward.</p>
<p>NOTE: I have tried the above experiments for Learning rates ranging from 1e-2 to 1e-6; Weight decay from 1e-3 to 1e-6; for optimizers both Adam and SGD; epochs from 50 to 200 (with and without early stopping). The loss graph for all the experiments conducted so far is similar to the above snapshot.</p>
<p>Any help is appreciated.</p>

username: ptrblck
cooked: <p>I’m still skeptical about your solution of making sure the <code>target</code> tensor is attached to a computation graph as mentioned <a href="https://discuss.pytorch.org/t/loss-oscillating-and-not-minimizing/177796/7">here</a>, since it’s common to assume a static target while the model outputs should of course be attached to the computation graph created in the forward pass of the model.</p>

username: nam1410
cooked: <p>I understand your point, and I may have been unclear in my previous post. Unfortunately, <code>target_val</code> was somehow attached to the computational graph in my calculations, which was one of the issues. Additionally, I had performed some mathematical functions (non-differentiable) internally on <code>pred_value</code> in the <code>forward</code> pass, which raised further complications. I have addressed the glitch in the computation process caused by such problematic tasks.</p>
<p>However, currently, I am facing a hurdle where the loss is stuck at a certain point after a few epochs.</p>