Decreasing the learning rate

Berny: <p>Hi. I created a network and I have to train it. I used this code to set the optimizer:</p>
<p>“”“optimizer = optim.Adam(model.parameters(), lr = 1e-2)”“”</p>
<p>and this one for the training:</p>
<p>“”“epochs = 20<br>
for epoch in range(epochs):<br>
loss = 0<br>
if <strong>name</strong> == ‘<strong>main</strong>’:<br>
for (batch_features, _) in train_loader:<br>
batch_features = batch_features.view(-1, 10694).to(device)<br>
optimizer.zero_grad()<br>
outputs = model(batch_features)<br>
train_loss = criterion(outputs[0], batch_features)<br>
train_loss.backward()<br>
optimizer.step()<br>
loss += train_loss.item()<br>
loss = loss / len(train_loader)<br>
print(“epoch : {}/{}, loss = {:.6f}”.format(epoch + 1, epochs, loss))”“”</p>
<p>Is there a way to decrease the learning rate (for example lr = lr/1.1) every 3 epochs?</p>

ArchieGertsman: <p>Yes, you can use a learning rate scheduler: <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.MultiplicativeLR.html#torch.optim.lr_scheduler.MultiplicativeLR" rel="noopener nofollow ugc">MultiplicativeLR</a> is the one you are looking for.</p>

Berny: <p>Ok thanks, I have only a doubt about it. I used these instructions before the for loop:</p>
<p><span class="hashtag">#set</span> the scheduler<br>
scheduler = StepLRScheduler(optimizer, decay_t = 2, decay_rate=0.9)<br>
lr_per_epoch = get_lr_per_epoch(scheduler, epochs)</p>
<p>And the following inside the for loop:</p>
<p><span class="hashtag">#Scheduler</span> step<br>
scheduler.step(epoch)</p>
<p>Can be ok or do I have to insert all the instructions inside the for loop?</p>

ArchieGertsman: <p>You only need to initialize the scheduler once, so you are correct to leave those first calls out of the loop. Within the outer loop, you can call <code>scheduler.step()</code> every three epochs using a conditional like <code>if epoch % 3 == 0</code>.</p>

Berny: <p>Do you mean something like this?</p>
<p>epochs = 40<br>
scheduler = StepLRScheduler(optimizer, decay_t = 3, decay_rate= 0.9)<br>
lr_per_epoch = get_lr_per_epoch(scheduler, epochs)<br>
for epoch in range(epochs):<br>
loss = 0<br>
if <strong>name</strong> == ‘<strong>main</strong>’:<br>
for (batch_features, _) in train_loader:<br>
batch_features = batch_features.view(-1, 10694).to(device)<br>
optimizer.zero_grad()<br>
outputs = model(batch_features)<br>
train_loss = criterion(outputs[0], batch_features)<br>
train_loss.backward()<br>
optimizer.step()<br>
loss += train_loss.item()<br>
loss = loss / len(train_loader)<br>
if epoch % 3 == 0:<br>
scheduler.step(epoch)<br>
print(“epoch : {}/{}, loss = {:.6f}”.format(epoch + 1, epochs, loss))</p>

ArchieGertsman: <p>Actually, you don’t need the <code>if epoch % 3 == 0</code> for <code>StepLRScheduler</code>, as it automatically does it for you (you specified to decrease the learning every three epochs when initializing <code>scheduler</code>).</p>
<p>Also, I’m not sure which documentation you are reading, but I believe the class you’re looking for is <a href="https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.StepLR.html#steplr" rel="noopener nofollow ugc">StepLR</a> from <code>torch.optim.lr_scheduler</code>, rather than <code>StepLRScheduler</code>. Its arguments are called <code>step_size</code> and <code>gamma</code> rather than <code>decay_t</code> and <code>decay_rate</code>, and you need not pass <code>epoch</code> into <code>scheduler.step()</code>.</p>

Berny: <p>Thank you for the help.</p>