ReduceLROnPlateau not doing anything?

username: cod3licious
cooked: <p>I’m trying to use the ReduceLROnPlateau scheduler but it doesn’t do anything, i.e. not decrease the learning rate after my loss stops decreasing (and actually starts to increase over multiple epochs quite a bit).</p>
<p>Here is the code:</p>
<pre><code class="lang-python">        criterion = nn.MSELoss()
        optimizer = optim.Adam(self.model.parameters(), lr=lr, weight_decay=weight_decay)
        lr_scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, verbose=True)

        for epoch in range(epochs):
            running_loss = 0.0
            for i, data in enumerate(trainloader):
                x_batch, s_batch = data
                x_batch, s_batch = x_batch.to(self.device), s_batch.to(self.device)
                optimizer.zero_grad()
                outputs = self.model(x_batch)
                loss = criterion(outputs, s_batch)
                running_loss += loss.item()
                loss.backward()
                optimizer.step()
            lr_scheduler.step(running_loss)
</code></pre>
<p>What am I missing? <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=6" title=":confused:" class="emoji" alt=":confused:"></p>

username: ptrblck
cooked: <p>How many epochs do you have?<br>
The default value for <code>patience</code> is set to <code>10</code>.<br>
Maybe you have to lower it a bit?</p>

username: cod3licious
cooked: <p>no, I’m using 25 epochs and the loss stops decreasing around epoch 10 or 11 so that should be fine…</p>

username: cod3licious
cooked: <p>ok, super weird. I tried it and set patience to 0 and it worked - at epoch 11 it decreased the LR! I feel like this might be a bug?</p>

username: ptrblck
cooked: <p>Here is a small dummy example and it seems to work:</p>
<pre><code class="lang-python">model = nn.Linear(10, 2)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    optimizer, patience=10, verbose=True)

for i in range(25):
    print('Epoch ', i)
    scheduler.step(i)
</code></pre>
<p>Setting <code>patience=0</code> reduces the lr until <code>eps</code> is met.</p>

username: cod3licious
cooked: <p>here is my output from training, this was with patience=1:</p>
<pre><code class="lang-auto">[0.0005]
[epoch 1] loss: 301427.7770613
[0.0005]
[epoch 2] loss: 160481.1660669
[0.0005]
[epoch 3] loss: 117763.6467930
[0.0005]
[epoch 4] loss: 81528.4339311
[0.0005]
[epoch 5] loss: 61726.3664427
[0.0005]
[epoch 6] loss: 46429.0652291
[0.0005]
[epoch 7] loss: 35138.6479489
[0.0005]
[epoch 8] loss: 23490.1695382
[0.0005]
[epoch 9] loss: 17156.3515503
[0.0005]
[epoch 10] loss: 13263.6756480
[0.0005]
[epoch 11] loss: 10980.2816269
[0.0005]
[epoch 12] loss: 11231.1720589
[0.0005]
[epoch 13] loss: 7322.0585567
[0.0005]
[epoch 14] loss: 9150.4534486
Epoch    14: reducing learning rate of group 0 to 5.0000e-05.
[5e-05]
[epoch 15] loss: 12504.9455976
[5e-05]
[epoch 16] loss: 935.6753716
[5e-05]
[epoch 17] loss: 793.3272605
[5e-05]
[epoch 18] loss: 719.4522403
</code></pre>
<p>from my understanding (or what I want) is for the LR to be decreased after epoch 12 already, where the loss starts to increase. Why isn’t that the case? Does that have to do with the relative threshold? I found this part pretty confusing in the documentation…</p>
<p>if I have patience=10, then this is the output, i.e., the LR does not get reduced:</p>
<pre><code class="lang-auto">[0.0005]
[epoch 1] loss: 301427.7770613
[0.0005]
[epoch 2] loss: 160481.1660669
[0.0005]
[epoch 3] loss: 117763.6467930
[0.0005]
[epoch 4] loss: 81528.4339311
[0.0005]
[epoch 5] loss: 61726.3664427
[0.0005]
[epoch 6] loss: 46429.0652291
[0.0005]
[epoch 7] loss: 35138.6479489
[0.0005]
[epoch 8] loss: 23490.1695382
[0.0005]
[epoch 9] loss: 17156.3515503
[0.0005]
[epoch 10] loss: 13263.6756480
[0.0005]
[epoch 11] loss: 10980.2816269
[0.0005]
[epoch 12] loss: 11231.1720589
[0.0005]
[epoch 13] loss: 7322.0585567
[0.0005]
[epoch 14] loss: 9150.4534486
[0.0005]
[epoch 15] loss: 12504.9455976
[0.0005]
[epoch 16] loss: 10636.9359541
[0.0005]
[epoch 17] loss: 8429.2563446
[0.0005]
[epoch 18] loss: 5167.7016573
[0.0005]
[epoch 19] loss: 7893.0735840
[0.0005]
[epoch 20] loss: 7127.2575889
[0.0005]
[epoch 21] loss: 8848.1689746
[0.0005]
[epoch 22] loss: 7766.2236686
[0.0005]
[epoch 23] loss: 8599.2677275
[0.0005]
[epoch 24] loss: 9155.7630087
[0.0005]
[epoch 25] loss: 7540.2192680
</code></pre>
<p>so patience influences the threshold?!</p>

username: ptrblck
cooked: <p>As far as I see the losses, the minimum seems to be in epoch18 with <code>5167</code>.<br>
Even though the loss was a bit shaky before that, the min loss was still lowered before the patience was satisfied. After epoch 18 you would have to stay over <code>5167</code> for the next 10 epochs.<br>
If you want to change this behavior, you could also play around with <code>cooldown</code>:</p>
<blockquote>
<p><strong>cooldown</strong> (<a href="https://docs.python.org/3/library/functions.html#int"> <em>int</em> </a>) – Number of epochs to wait before resuming normal operation after lr has been reduced. Default: 0.</p>
</blockquote>

username: cod3licious
cooked: <p>But since cooldown is by default 0, there shouldn’t be any waiting. For patience = 10 the loss decreases until epoch 11 (10980) and then increases in epoch 12 (11231) so I would assume that after this increase the LR is reduced (since the scheduler has no idea that a better minimum will come along later). But the loss value at which point the LR is decreased depends on the patience, even though, from my understanding, patience should only determine after how many epochs the reducing starts, not how much worse the value has to get for it to start reducing (which, as I understand, should be controlled by threshold). So this seems like a bug, no?</p>
<p>To expand on your minimal example:</p>
<pre><code class="lang-python">model = nn.Linear(10, 2)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    optimizer, patience=10, verbose=True)

test_loss = [301427.7770613, 160481.1660669, 117763.6467930, 81528.4339311, 61726.3664427, 46429.0652291, 35138.6479489, 23490.1695382, 17156.3515503, 13263.6756480, 10980.2816269, 11231.1720589, 7322.0585567, 9150.4534486, 12504.9455976, 10636.9359541, 8429.2563446, 5167.7016573, 7893.0735840, 7127.2575889, 8848.1689746, 7766.2236686, 8599.2677275, 9155.7630087, 7540.2192680]

for i in test_loss:
    print('loss: ', i)
    scheduler.step(i)
</code></pre>
<p>–&gt; with patience 10, the LR wont be reduced at all, with patience 0 it is reduced after epoch 11, with patience 5 after epoch 23…</p>

username: cod3licious
cooked: <p>ok, I couldn’t help myself and created an issue: <a href="https://github.com/pytorch/pytorch/issues/11305" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/11305</a> this just seems too weird to work as intended like this…</p>

username: ptrblck
cooked: <p>The patience is applied to the last minimal loss value and the subsequent values.<br>
Let’s analyze the behavior for <code>patience=0</code>:</p>
<ul>
<li>Until epoch10 the loss is decreasing (starting with epoch0).</li>
<li>The loss in epoch11 increases; since <code>patience=0</code>, we are decreasing the lr. The current min value is <code>10980</code> from epoch9.</li>
<li>The loss decreases to <code>7322</code> in epoch 12, which is our new minimum</li>
<li>After it the loss increases in every epoch and the lr is also decreased in every epoch until <code>eps</code> is met.</li>
</ul>
<p>Let’s now have a look at <code>patience=5</code>:</p>
<ul>
<li>Again the loss decreases until epoch10=10980, which is our current minimum</li>
<li>The loss increases in epoch11; our “current” patience is now 4; the current minimum stays the same.</li>
<li>Epoch 12 yields a new minimum of <code>7322</code>. Patience is back to 5.</li>
<li>The loss increases in the next 4 epochs. Patience is lowering to 4, 3, 2, 1.</li>
<li>Epoch17 yields a new minimum of <code>5167</code>. Patience is back to 5.</li>
<li>The loss increases for the rest of the epoch. After 5 increases, the “current” patience is at 0 and the lr will be decreased.</li>
</ul>

username: cod3licious
cooked: <p>aaaahhh… ok, I thought <code>patience</code> only applies to the initial epochs, like, “let the model fool around for the first X epochs, then start monitoring and decreasing the LR”. Sorry, this clears things up! A default value of 10 seems a bit high though then.</p>
<p>Thanks so much!</p>

username: ptrblck
cooked: <p>No worries. I just realized your point of view a few moments ago. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"><br>
Yeah, it might be a bit high, but it also depends on your dataset etc. so I think it’s a valid base value.</p>
<p>If you want to skip some initial epochs, you might want to add a condition like:</p>
<pre><code class="lang-python">if epoch &gt; 10:
    scheduler.step(loss)
</code></pre>
<p>to your training loop.</p>

username: learner47
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, after the patience runs out and the lr is decreased to a minimum value allowed by eps, is there a chance that lr could go back to its original value if the loss begins to decrease? Or is it the case that, once eps is hit, it would be stuck at that lr forever?</p>

username: ptrblck
cooked: <p>I think the <code>ReduleLROnPlateau</code> would only reduce the learning rate, so it would be stuck at this value.<br>
However, you could adapt this scheduler to increase the learning rate again using a similar approach.</p>

username: alirezakazemipour
cooked: <p>Your commitment to support is unbelievable!<br>
Thank you. <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=9" title=":heart:" class="emoji" alt=":heart:"></p>

username: paepcke
cooked: <p>Agreeing with alirezakazemipour: I am learning from your posts every day. Thank you for being so active and patient.</p>

username: Kimonili
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Just a follow-up question on top of this discussion. Does the learning rate get reduced even if the difference between 2 epochs is very very small (for a patience = 0)?</p>
<p>Or there is a condition based on which the difference between two epochs should be higher than some value?</p>

username: ptrblck
cooked: <p>The used <code>threshold</code> defines the needed change to trigger the learning rate reduction:</p>
<blockquote>
<p><strong>threshold</strong> (<a href="https://docs.python.org/3/library/functions.html#float"><em>float</em></a>) – Threshold for measuring the new optimum, to only focus on significant changes. Default: 1e-4.</p>
</blockquote>

username: Kimonili
cooked: <p>Yeah I also checked the source code of class to make sure how it is calculated. I think it has to do with the threshold mode as well.</p>
<pre><code class="lang-auto">def is_better(self, a, best):
        if self.mode == 'min' and self.threshold_mode == 'rel':
            rel_epsilon = 1. - self.threshold
            return a &lt; best * rel_epsilon

        elif self.mode == 'min' and self.threshold_mode == 'abs':
            return a &lt; best - self.threshold

        elif self.mode == 'max' and self.threshold_mode == 'rel':
            rel_epsilon = self.threshold + 1.
            return a &gt; best * rel_epsilon

        else:  # mode == 'max' and epsilon_mode == 'abs':
            return a &gt; best + self.threshold
</code></pre>
<p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> !!</p>