akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Learning rate decay and Weight decay..difference?

username: davidlee
cooked: <p>Hi guys,</p>
<p>I am wondering if anyone can clearly say difference between<br>
the learning rate decay (Learning rate scheduling) and<br>
Weight decay (L2 regularization)?</p>
<p>I know the learning rate decay is for updating weights slowly or quickly and<br>
the weight decay is to give loss function a penalty for avoid overfitting.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a049d3a1fc303d60c6cde70d95bbb9ceb62b28b.png" data-download-href="https://discuss.pytorch.org/uploads/default/6a049d3a1fc303d60c6cde70d95bbb9ceb62b28b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a049d3a1fc303d60c6cde70d95bbb9ceb62b28b.png" alt="image" data-base62-sha1="f7SuR4qCMnIdiKp6U6NM5ulkbpx" width="690" height="60" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">761×67 2.22 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
like the photo I’ve attached,<br>
But is it right that those two parameters would affect on updating the weights at the same time?</p>
<p>Then, is it preferable to use lr schedule and weight decay at the same time?</p>
<p>Thanks in advance!!</p>

username: lycaenidae
cooked: <p>L2 regularization also called weight decay is one of the regularization methods like (L1 or Dropout) as you have also mentioned ‘avoiding overfitting’</p>
<p>Learning rate decay aims to update learning rate hyperparameter considering time. Because choosing the learning rate is challenging as a value too small may result in a long training process that could get stuck, whereas a value too large may result in learning a sub-optimal set of weights too fast or an unstable training process. We want to take ‘bigger’ steps when we started training but we want to be careful while closing to optimum. That is we don’t want to overshoot to optimum.</p>
<p>Regarding your question, yes you can both of them like L2 to prevent overfitting, learning rate decay not to overshoot.</p>
<p>I hope it is clear a little bit.</p>
Validaion loss and convergence

username: netaglazer
cooked: <p>i have a model, that for the same parameters the loss validation sometimes convergent and sometimes not.<br>
does it mean anything?</p>

username: Topsoil
cooked: <p>Please elaborate on what do you exactly mean by not converging: do you mean that your trainging loss goes down while your validation loss goes up, or that both of your losses do not always go down to the same value in the same number of batches?</p>
<p>Also, are you training “usual” network, or a gan?</p>

username: netaglazer
cooked: <p>2 cases:</p>
<ol>
<li>
<p>the train goes down, the accuracy train goes to 1, and the validation goes down too, and the validation accuracy goes to 1 too</p>
</li>
<li>
<p>he train goes down, the accuracy train goes to 1, and the validation loss goes up</p>
</li>
</ol>
<p>those two cases happening for the same parameters</p>

username: Topsoil
cooked: <p>So, the case 1 looks like a normal training process, and case 2 looks like overfitting: model learns your training dataset, but not learning general relationships in the input data (Its like instead of prepairing for the test, you memorized the answers)</p>
<p>Try to simplify your model, giving it less neurons, preventing it from memorising test data, and see if problem persists. Also, some graphs of loss and accuracies would be nice</p>