Different learning rate for different type of module

username: dlmacedo
cooked: <p>How can I have all PReLU to have a learning rate that is 0.1 times the used by the other layer?</p>

username: SimonW
cooked: <p>On top of my head, there are two options:</p>
<ol>
<li>write your own lr scheduler (see examples here:<a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py">https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py</a>)</li>
<li>use different optimizers for different parts of your network.</li>
</ol>

username: ptrblck
cooked: <p>You  can check out <a href="http://pytorch.org/docs/master/optim.html#per-parameter-options">optim-per-parameter-options</a>, where there is a small example how to set different learning rates for your layers.</p>
<blockquote>
<p>Optimizer s also support specifying per-parameter options. To do this, instead of passing an iterable of Variable s, pass in an iterable of dict s. Each of them will define a separate parameter group, and should contain a params key, containing a list of parameters belonging to it. Other keys should match the keyword arguments accepted by the optimizers, and will be used as optimization options for this group.</p>
<p>optim.SGD([<br>
{‘params’: model.base.parameters()},<br>
{‘params’: model.classifier.parameters(), ‘lr’: 1e-3}<br>
], lr=1e-2, momentum=0.9)</p>
</blockquote>

username: dlmacedo
cooked: <p>The problema is my customized module is used in many places in the model… They are spread… Not thogheter… But they have the same type… Like use PReLU in diferent places of the model…</p>

username: SimonW
cooked: <p>I just remembered this option and came here to comment. Thanks for pointing that out before me <span class="mention">@ptrbick</span>!</p>

username: SimonW
cooked: <p>Oh so you want its parameter to have 0.1 of the original gradient no matter where it is used? How about adding a automatic backward hook on that module’s parameter? You can do that in the constructor even.</p>

username: dlmacedo
cooked: <aside class="quote no-group" data-username="SimonW" data-post="6" data-topic="8999">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simonw/48/1702_2.png" class="avatar"> SimonW:</div>
<blockquote>
<p>backward hook</p>
</blockquote>
</aside>
<p>Could you please give a piece of example code or link?</p>

username: dlmacedo
cooked: <blockquote>
<pre><code>class DDReLU(nn.Module):
def __init__(self):
    super(DDReLU, self).__init__()
    self.threshold = nn.Parameter(torch.rand(1), requires_grad=True)
    self.register_backward_hook(lambda grad: grad * 0.1)
    self.ReLU = nn.ReLU(True)

def forward(self, x):
    return self.ReLU(x) + self.threshold
</code></pre>
</blockquote>

username: dlmacedo
cooked: <p><a href="http://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html?highlight=hook#forward-and-backward-function-hooks" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/former_torchies/nn_tutorial.html?highlight=hook#forward-and-backward-function-hooks</a></p>
<p><a href="http://pytorch.org/docs/master/autograd.html" rel="nofollow noopener">http://pytorch.org/docs/master/autograd.html</a></p>

username: drcege
cooked: <p>I encountered an issue of implementing dynamic learning rate. I want to give each tensor a different lr before each backward (i.e., before <code>optim.step</code>).</p>
<p>After skimming the source code of <code>torch.optim</code>, <code>torch.optim.adam</code>, and <code>torch.optim.sgd</code>, I realized that this is impossible as lr is passed as a fixed value when defining the optimizer.</p>
<p>I think what I need is a learning rate <code>hook</code> to modify lr of each tensor before optim.step, which is somehow like module’s backward hook from which we can modify the gradients.</p>

username: ptrblck
cooked: <p>You can use different learning rates for each parameter via the <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options">Per-parameter options</a>. However, if you want to manipulate it in every step, you might either need to recreate the optimizer (in which care stateful optimizers would re-initialize their running stats) or you could indeed use backward hooks to manipulate the <code>.grad</code> attribute directly.</p>

username: drcege
cooked: <p>Thanks, recreating the optimizer might be a solution but could be cumbersome and inefficient.</p>
<p>I considered manipulating the <code>.grad</code> attribute directly, but for optimizers with moving average like Adam, I found it’s hard to achieve the same effect.</p>