Why is the learning rate increasing

username: arash_javanmard
cooked: <p>I am kind of confused about the behavior of some LR-Schedulers - e.g. LinearLR and ConstantLR.<br>
The doc [<a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.LinearLR.html#torch.optim.lr_scheduler.LinearLR" rel="noopener nofollow ugc">here</a> and <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ConstantLR.html#torch.optim.lr_scheduler.ConstantLR" rel="noopener nofollow ugc">here</a>] says, that they <strong>decay</strong> the learning rate. But as in the example and my own code, they actually increase it. What am I getting wrong?</p>
<pre><code class="lang-auto">import numpy as np
import torch
import matplotlib.pyplot as plt

np.set_printoptions(precision=5, suppress=True)
model = torch.nn.Linear(2, 1)
lr = 0.001
epochs = 100

def run_experiment(optimizer, epochs=epochs):
    lrs = []
    for i in range(epochs):
        optimizer.step()
        lrs.append(optimizer.param_groups[0]["lr"])
        scheduler.step()

    lrs = np.array(lrs)

    plt.plot(range(epochs),lrs)
    plt.scatter(range(epochs), lrs)
    print(lrs)
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
scheduler = torch.optim.lr_scheduler.LinearLR(optimizer, start_factor=0.5, total_iters=epochs)
run_experiment(optimizer)
# Outputs: [0.0005  0.00051 0.00051 0.00052 ..., 0.001]
</code></pre>
<p>Regards<br>
Arash</p>

username: azhanmohammed
cooked: <p>You have used the argument <code>start_factor</code>, as per the docs <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.LinearLR.html#torch.optim.lr_scheduler.LinearLR" rel="noopener nofollow ugc">here</a> <code>start_factor</code> is the number we multiply our learning rate with towards the start of epoch, so each time we multiply, it increases. To decay the learning rate continuously, you will have you use <code>end_factor</code> and set <code>start_factor</code> as 1, which will lead to a smaller result after multiplication.</p>
<pre><code class="lang-python">np.set_printoptions(precision=5, suppress=True)
model = torch.nn.Linear(2, 1)
lr = 0.001
epochs = 100

def run_experiment(optimizer, epochs=epochs):
    lrs = []
    for i in range(epochs):
        optimizer.step()
        lrs.append(optimizer.param_groups[0]["lr"])
        scheduler.step()
    lrs = np.array(lrs)
    print(lrs)
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
scheduler = torch.optim.lr_scheduler.LinearLR(optimizer, start_factor=1, end_factor=0.1, total_iters=epochs)
run_experiment(optimizer)
</code></pre>
<p>Which gives the result:</p>
<pre><code class="lang-python">[0.001   0.00099 0.00098 0.00097 0.00096 0.00096 0.00095 0.00094 0.00093
 0.00092 0.00091 0.0009  0.00089 0.00088 0.00087 0.00086 0.00086 0.00085
 0.00084 0.00083 0.00082 0.00081 0.0008  0.00079 0.00078 0.00077 0.00077
 0.00076 0.00075 0.00074 0.00073 0.00072 0.00071 0.0007  0.00069 0.00068
 0.00068 0.00067 0.00066 0.00065 0.00064 0.00063 0.00062 0.00061 0.0006
 0.00059 0.00059 0.00058 0.00057 0.00056 0.00055 0.00054 0.00053 0.00052
 0.00051 0.0005  0.0005  0.00049 0.00048 0.00047 0.00046 0.00045 0.00044
 0.00043 0.00042 0.00041 0.00041 0.0004  0.00039 0.00038 0.00037 0.00036
 0.00035 0.00034 0.00033 0.00032 0.00032 0.00031 0.0003  0.00029 0.00028
 0.00027 0.00026 0.00025 0.00024 0.00023 0.00023 0.00022 0.00021 0.0002
 0.00019 0.00018 0.00017 0.00016 0.00015 0.00014 0.00014 0.00013 0.00012
 0.00011]
</code></pre>