Learning scheduler

pedram1: <p>Hi everyone</p>
<p>I tried to use scheduler however I faced with the below error:<br>
name ‘StepLR’ is not defined<br>
has anyone know how I can fix it?<br>
Below my I wrote my training codes:<br>
<span class="hashtag">#training</span> loop<br>
loss = 0<br>
epoch_num = 0<br>
error = []<br>
scheduler = StepLR(optimizer, step_size=20, gamma=0.5)<br>
for epoch in range(num_epochs):<br>
for spec in train_loader:<br>
img = spec.cuda()</p>
<pre><code>output = model (img)
loss = criterion (output,img)

optimizer.zero_grad()
loss.backward()
optimizer.step()
error.append(loss.item())
scheduler.step()


if epoch%10 == 9:
  epoch_num += epoch_num
  plt.plot(error)


  print ('\r Train Epoch : {}/{} \tLoss : {:.4f}'.format (epoch+1,num_epochs,loss/32))
</code></pre>
<p>model_save_name = ‘Autoencoder for Li project  11 feb(2) (mid power) febTanh activation function Feb 2020 DMSO dataset’<br>
path = F"/content/drive/My Drive/{model_save_name}"<br>
torch.save(model.state_dict(), path)</p>
<p>thanks you</p>

ptrblck: <p>Which PyTorch version are you using?<br>
If you are not using the latest stable release, could you please update?</p>

pedram1: <p>Hi Patrick</p>
<p>I am using latest version of pytorch.</p>

ptrblck: <p>How did you import it?</p>
<pre><code class="lang-python">from torch.optim.lr_scheduler import StepLR
</code></pre>
<p>works for me.</p>

pedram1: <aside class="quote no-group" data-username="pedram1" data-post="1" data-topic="69421">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pedram1/48/21032_2.png" class="avatar"> pedram1:</div>
<blockquote>
<p>scheduler.step()</p>
</blockquote>
</aside>
<p>WOW</p>
<p>May be I out scheduler.step() in the wrong spot.<br>
Should it be in epoch loop? or in train_loader loop</p>

alx: <p>scheduler.step() should be right after optimizer.step() and both should be in the training loop.</p>
<pre><code class="lang-auto">for epoch in range(N_EPOCHS):
   for batch_i, (X, y) in train_dataloader:
     optimizer.step() 
     scheduler.step()
</code></pre>

pedram1: <p>I did this as well, however when I print my lr in every epoch I can see that it dose not change. <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=9" title=":expressionless:" class="emoji" alt=":expressionless:"></p>
<p>learning_rate = 0.0001</p>
<p>num_epochs = 200</p>
<p>model = Autoencoder().cuda()</p>
<p>criterion = nn.MSELoss()</p>
<p>optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,</p>
<p>weight_decay=1e-5)</p>
<p><span class="hashtag">#training</span> loop</p>
<p>loss = 0</p>
<p>epoch_num = 0</p>
<p>error = []</p>
<p>scheduler = StepLR(optimizer, step_size=30, gamma=0.5)</p>
<p>for epoch in range(num_epochs):</p>
<p>for spec in train_loader:</p>
<pre><code>img = spec.cuda()

output = model(img)

loss = criterion (output,img)

optimizer.zero_grad()

loss.backward()

optimizer.step()

scheduler.step()

error.append(loss.item())
</code></pre>
<p>if epoch%10 == 9:</p>
<pre><code>  epoch_num+=epoch_num
</code></pre>
<p>print (’\r Train Epoch : {}/{} \tLoss : {:.4f}’.format (epoch+1,num_epochs,loss/50))</p>
<p>print(learning_rate)</p>
<p>model_save_name = ‘Autoencoder for Artificial dataset (7) 3 class of data  12 feb Avgpool Tanh activation function’</p>
<p>path = F"/content/drive/My Drive/{model_save_name}"</p>
<p>torch.save(model.state_dict(), path)</p>
<p>plt.plot(error)</p>
<p>plt.xlabel(‘Number of iteration’)</p>
<p>plt.ylabel(‘Loss (MSE)’)</p>
<p>plt.title(‘Loss function vs # of iterations for Artificial dataset’)</p>
<p>even worst than that is, training is not working, loss stays constant</p>

pedram1: <p>tnx<br>
however when I print lr in every epoch I can see that, it dose not change.</p>

ptrblck: <p>You are currently printing the <code>learning_rate</code> variable, which is set to <code>0..0001</code>.<br>
Use <code>print(scheduler.get_last_lr())</code> instead to get the learning rate, which will be applied.</p>