Training Loss Randomly Varying on Startup

TheVictor_777: <p>Hello, I am currently training a CNN model with the AdamW optimizer. I notice that when I initiate the education process, the loss is often unacceptably high. I solve the issue by simply restarting the training script up to three times. With every execution, the loss varies significantly until I hit a fine, low starting point. I have concluded that the optimizer must randomly initiate values within itself every time, which can result in a high loss or otherwise. Is this correct?</p>

ptrblck: <aside class="quote no-group" data-username="TheVictor_777" data-post="1" data-topic="198928">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/5daacb/48.png" class="avatar"> TheVictor_777:</div>
<blockquote>
<p>I have concluded that the optimizer must randomly initiate values within itself every time, which can result in a high loss or otherwise.</p>
</blockquote>
</aside>
<p>I would rather guess the model’s random parameter initialization is not optimal and you might want to change it. What makes you think the optimizer is at fault? If the very first loss output is already high, the optimizer would not have been used by then.</p>

TheVictor_777: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Your theory cannot be the case, as my model is saved to a file and loaded on each startup. The only thing I determine can be randomly initiated is the optimizer. Even so, if I save the optimizer and load it in the next training session, the loss still varies randomly.</p>

ptrblck: <p>This would mean that you are not checking the initial loss, but the loss after a few update steps?<br>
If not, how can the optimizer influence the initial loss?<br>
Also, could you explain which part of <code>AdamW</code> is random and which parameters are initialized in this optimizer?</p>

TheVictor_777: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Let me explain. My training program operates as shown:</p>
<ol>
<li>Import modules, define model, dataset, other functions.</li>
<li>Setup model, load state dictionary from file is possible.</li>
<li>Generate AdamW optimizer with MSELoss function.</li>
<li>Train on batches while tracking average loss per epoch.</li>
<li>After each epoch, print average loss.</li>
</ol>
<p>The average loss, especially for the first epochs, is what varies the most with every training session. For example, I could get this on the first execution:</p>
<pre><code class="lang-auto">Epoch 1/50000 finished with an average training loss of: 0.00483 and an average test loss of: 0.00799
</code></pre>
<p>Too high. Restart training script:</p>
<pre><code class="lang-auto">Epoch 1/50000 finished with an average training loss of: 0.00371 and an average test loss of: 0.00782
</code></pre>
<p>That’s more like it. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"> The <strong>exact same</strong> model file, dataset, loss function, and optimizer class are used in both instances. I changed nothing. How can you explain this?<br>
As for your question regarding which parts of the optimizer are randomly generated… I haven’t an inkling. <img src="https://discuss.pytorch.org/images/emoji/apple/person_shrugging.png?v=12" title=":person_shrugging:" class="emoji" alt=":person_shrugging:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Any other random operation, e.g. the shuffling in the <code>DataLoader</code> or transformations, could cause it.<br>
Let me know if you narrowed down what exactly is randomly initialized in the optimizer potentially causing the difference in loss.</p>