Print current learning rate of the Adam Optimizer?

flupadup: <p>At the beginning of a training session, the Adam Optimizer takes quiet some time, to find a good learning rate. I would like to accelerate my training by starting a training with the learning rate, Adam adapted to, within the last training session.<br>
Therefore, I would like to print out the current learning rate, Pytorchs Adam Optimizer adapts to, during a training session.<br>
thanks for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

lelouedec: <blockquote>
<p>for param_group in optimizer.param_groups:<br>
print(param_group[‘lr’])</p>
</blockquote>
<p>should do the job</p>

jpeg729: <p>There is a paper that proposes tuning the learning rate using the gradient of the update rule with respect to the learning rate itself. Basically, it dynamically learns the learning rate during training.</p>
<p>If you are interested, here is the paper <a href="https://arxiv.org/abs/1703.04782">https://arxiv.org/abs/1703.04782</a> and my implementation <a href="https://github.com/jpeg729/pytorch_bits/blob/master/optim/adam_hd.py#L67">https://github.com/jpeg729/pytorch_bits/blob/master/optim/adam_hd.py#L67</a></p>

flupadup: <p>thanks for the quick response, but unfortunately, this command just prints out the initial learning rate and not the current adapted one. I would really surprise me, if it wasn’t possible, get the adapted learning rate somehow…</p>

flupadup: <p>thank you, I will look into it!</p>

lelouedec: <p>this method print the currently used learning rate by the optimizer.</p>

flupadup: <p>at the moment I am just using two input images for my training. When I start a training session with the network, pretrained by me, the error increases by some magnitudes (from a few hundred to 10.000 up to 40.000) and commutes than back to the level, where it was at the end of the last session. Through all this the learning rate printed out on the console is always the same, initial one, what makes ne sense to me.<br>
I don’t know what else could be the reason for this big temporal fluctuation of the error.</p>

jpeg729: <p>Did you save the optimizer state with the model?<br>
Different optimizers tend to find different solutions so changing optimizers or resetting their state can perturbe training. That is why it can be important to not only save the model parameters but also the optimizer state.</p>
<p>Of course, in this case, that might have nothing to do with the fluctuation of the error that you are seeing.</p>

Jonathan_R_Williford: <p>Adam has a separate learning rate for each parameter. The <code>param_group['lr']</code> is a kind of base learning rate that does not change. There is no variable in the PyTorch Adam implementation that stores the dynamic learning rates.</p>
<p>One could save the optimizer state, as mentioned here:</p><aside class="quote" data-post="1" data-topic="2610">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/958977/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/saving-and-loading-a-model-in-pytorch/2610">Saving and loading a model in Pytorch?</a> 
  </div>
  <blockquote>
    If I have a model class and a trainer class. I create an instance of the model and train it. 
model = mymodel()
train = trainer.train(model...) 

How can I save the model to a file, after it has been trained and how can I then reload it and continue training? I searched for this but didn’t get an answer.
  </blockquote>
</aside>

<p>The PyTorch implementation of Adam can be found here:<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/optim/adam.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/_modules/torch/optim/adam.html</a></p>
<p>The line <code>for p in group['params']:</code> iterates over all the parameters and calculates the learning rate for each parameter.</p>
<p>I found this helpful: <a href="http://ruder.io/optimizing-gradient-descent/index.html#adam" rel="noopener nofollow ugc">http://ruder.io/optimizing-gradient-descent/index.html#adam</a></p>

henrique: <aside class="quote no-group" data-username="Jonathan_R_Williford" data-post="9" data-topic="15204">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jonathan_r_williford/48/5606_2.png" class="avatar"> Jonathan_R_Williford:</div>
<blockquote>
<p>dynamic learning rates</p>
</blockquote>
</aside>
<p>Sorry guys, but these comments seem very misleading!<br>
Optimizers have a fixed learning rate for all parameters. <code>param_group['lr']</code> would allow you to set a different LR for each layer of the network, but it’s generally not used very often, and most people have 1 single LR for the whole nn.<br>
What Adam does is to save a running average of the gradients for each parameter (not a LR!). The learning rate is still the same throughout the whole training, unless you use a lr_scheduler like CosineAnnealingLR etc.</p>

Mithun_Paul: <p>also you might want to look into : ReduceLROnPlateau in <a href="https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html" rel="nofollow noopener">lrScheduler</a></p>

Prasad_Raghavendra: <p>Then, it is not the comments that are misleading, but, the library itself. It should have something like LR_view or something like that (at the very least). I am doing some optimiser research and I am having a pretty difficult time trying to get some LR out.</p>