How to set different learning rate for weight and bias in one layer?

elysion: <p>In Caffe, we can set different learning rate for weight and bias in one layer.<br>
For example:</p>
<pre><code class="lang-auto">layer {
    name: "conv2"
    type: "Convolution"
    bottom: "bn_conv2"
    top: "conv2"
    param {
       lr_mult: 1.000000*
    }
   param {
        lr_mult: 0.100000
    }
    convolution_param {
        num_output: 64
        kernel_size: 3
        stride: 1
        pad: 1
        weight_filler {
            type: "msra"
        }
        bias_filler {
            type: "constant"
            value: 0
        }
    }
}
</code></pre>
<p>the learning rate of weight and bias is leaning rate*lr_mult.</p>
<p>In pytorch, is it possible to set different learning rate for weight and bias in  one layer?<br>
How to write the program？</p>

jpeg729: <p>This might help <a href="http://pytorch.org/docs/0.3.0/optim.html#per-parameter-options" rel="nofollow noopener">http://pytorch.org/docs/0.3.0/optim.html#per-parameter-options</a></p>

elysion: <p>Thank you! I read the doc file. The Example seems to set different learning rate for different layers. The doc said we can use dict or param_group to set learning rate for different layers.<br>
I’m new in pytorch. May be there is a way to set weight/bias wise learning rate, but I can’t find it.<br>
would  you please tell me more about this？Thank you.</p>

jpeg729: <p>The example shows how to set different parameters for <code>layer.parameters()</code> you just need to dig a little deeper into the details.</p>
<p>E.g. for a Linear layer, the weight and bias parameters are named <code>mylayer.weight</code> and <code>mylayer.bias</code>.</p>
<pre><code class="lang-auto">optim.SGD([
                {'params': mylayer.weight},
                {'params': mylayer.bias, 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)</code></pre>

elysion: <p>Thank you so much for your patient  guidance ! I tried this code.  It reports error like this:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/mitc/pycharm-2017.3.3/helpers/pydev/pydevd.py", line 1668, in &lt;module&gt;
    main()
  File "/home/mitc/pycharm-2017.3.3/helpers/pydev/pydevd.py", line 1662, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/mitc/pycharm-2017.3.3/helpers/pydev/pydevd.py", line 1072, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/mitc/lcy/Pytorch-SR/nd_vdsr_unfold.py", line 137, in &lt;module&gt;
    ], lr=0.1, weight_decay=0.0001)
  File "/home/mitc/anaconda2/envs/lcy-pytorch/lib/python2.7/site-packages/torch/optim/adam.py", line 28, in __init__
    super(Adam, self).__init__(params, defaults)
  File "/home/mitc/anaconda2/envs/lcy-pytorch/lib/python2.7/site-packages/torch/optim/optimizer.py", line 61, in __init__
    raise ValueError("can't optimize a non-leaf Variable")
ValueError: can't optimize a non-leaf Variable
</code></pre>
<p>my code is：</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):#1,3,11,13,1
        super(Net, self).__init__()
        self.layer11 = nn.Sequential(
            nn.BatchNorm3d(num_features=1,momentum=0.999,affine=False),
            nn.ReLU(inplace=True),
            nn.Conv3d(in_channels=1,out_channels=16,kernel_size=(3,3,3),padding=(1,1,1),bias=True))
        self.layer21 = nn.Sequential(
            nn.BatchNorm3d(num_features=16, momentum=0.999, affine=False),
            nn.ReLU(inplace=True),
            nn.Conv3d(in_channels=16, out_channels=16, kernel_size=(3, 3, 3), padding=(1, 1, 1), bias=True))
....

 def forward(self, x, residual):
        #residual = x1
        out = self.layer11(x)
        out = self.layer21(out)
        out = self.layer22(out)
       ....
       out = torch.add(out, residual)
        return out 

if __name__=="__main__":
    net = Net()
    optimizer = optim.Adam([
                {'params': net.layer11[2].weight},
                {'params': net.layer11[2].bias, 'lr': 0.01}
            ], lr=0.1, weight_decay=0.0001)
.....

</code></pre>

jpeg729: <p>This toy example works.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.layer = nn.Linear(1, 1)
        self.layer.weight.data.fill_(1)
        self.layer.bias.data.fill_(1)

    def forward(self, x):
        return self.layer(x)

if __name__=="__main__":
    net = Net()
    optimizer = optim.Adam([
                {'params': net.layer.weight},
                {'params': net.layer.bias, 'lr': 0.01}
            ], lr=0.1, weight_decay=0.0001)
    out = net(Variable(torch.Tensor([[1]])))
    out.backward()
    optimizer.step()
    print("weight", net.layer.weight.data.numpy(), "grad", net.layer.weight.grad.data.numpy())
    print("bias", net.layer.bias.data.numpy(), "grad", net.layer.bias.grad.data.numpy())
</code></pre>
<p>Output is</p>
<pre><code class="lang-auto">weight [[ 0.90000004]] grad [[ 1.]]
bias [ 0.99000001] grad [ 1.]
</code></pre>
<p>As you can see, <code>weight</code> has been updated by ~<code>0.1 * weight.grad</code> and <code>bias</code> has been updated using ~<code>0.01 * bias.grad</code>.</p>
<p>The error you get suggests that you have asked the optimiser to optimise a Variable that isn’t a parameter of your model. But your partial code sample seems fine.</p>

elysion: <p>Thank you so much!<br>
By running your code , I find there are bugs in pytorch version 0.1.12.<br>
I change pytorch version. it worked.</p>

zxd123456: <p>Hello. Your solution is correct.<br>
But I met the problem because the model has to much embeddings like  “module.mixed_stem2.branch0.0.bn.weight”<br>
So I always get wrong when I use your method because the syntax of “batch0.0”<br>
What should I do to iter the weight and bias of this model so I can set different learning rate for them?</p>

elysion: <p>Try  “batch0[0]” instead of “batch0.0”.<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/wkentaro/pytorch-fcn/blob/master/examples/voc/train_fcn32s.py#L105" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/wkentaro/pytorch-fcn/blob/master/examples/voc/train_fcn32s.py#L105" target="_blank" rel="nofollow noopener">wkentaro/pytorch-fcn/blob/master/examples/voc/train_fcn32s.py#L105</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="95" style="counter-reset: li-counter 94 ;">
<li>cfg = configurations[args.config]</li>
<li>out = get_log_dir('fcn32s', args.config, cfg)</li>
<li>resume = args.resume</li>
<li>
</li>
<li>os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu)</li>
<li>cuda = torch.cuda.is_available()</li>
<li>
</li>
<li>torch.manual_seed(1337)</li>
<li>if cuda:</li>
<li>    torch.cuda.manual_seed(1337)</li>
<li class="selected">
</li>
<li># 1. dataset</li>
<li>
</li>
<li>root = osp.expanduser('~/data/datasets')</li>
<li>kwargs = {'num_workers': 4, 'pin_memory': True} if cuda else {}</li>
<li>train_loader = torch.utils.data.DataLoader(</li>
<li>    torchfcn.datasets.SBDClassSeg(root, split='train', transform=True),</li>
<li>    batch_size=1, shuffle=True, **kwargs)</li>
<li>val_loader = torch.utils.data.DataLoader(</li>
<li>    torchfcn.datasets.VOC2011ClassSeg(</li>
<li>        root, split='seg11valid', transform=True),</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
gives a solution for weigfht and bias wise learning rate setting.<br>
Just use function get_parameters()</p>
<pre><code class="lang-auto">def get_parameters(model, bias=False):
    import torch.nn as nn
    modules_skipped = (
        nn.ReLU,
        nn.MaxPool2d,
        nn.Dropout2d,
        nn.Sequential,
        torchfcn.models.FCN32s,
        torchfcn.models.FCN16s,
        torchfcn.models.FCN8s,
    )
    for m in model.modules():
        if isinstance(m, nn.Conv2d):
            if bias:
                yield m.bias
            else:
                yield m.weight
        elif isinstance(m, nn.ConvTranspose2d):
            # weight is frozen because it is just a bilinear upsampling
            if bias:
                assert m.bias is None
        elif isinstance(m, modules_skipped):
            continue
        else:
            raise ValueError('Unexpected module: %s' % str(m))
</code></pre>

zxd123456: <p>waooooo~~~~<br>
Thank you very much. Your code is really simple and can solve my problem well.<br>
I tried all night to construct a big for iteration to implement this function. But your method is so amazing. Thank you!</p>

abe7: <p>Can you just multiply the gradients for specific layers after <code>loss.backward()</code> and before <code>optimizer.step()</code> by a constant? Would that have the same effect?</p>