Different Learning rate for certain parameters

Harshit_Goyal: <p>These are the parameters of my Deep Learning model, to the right are their shapes. I want the learning rate of the parameters <code>rho</code> in each layer to be <code>0.01</code> initially and <code>0.001</code> intially. How can I do that? I saw other forumns but most just tell about setting different learning rate for specific layers initially.</p>
<p>Here are the <code>optimizer</code> and <code>scheduler</code> I’m using</p>
<pre><code class="lang-auto">optimizer = optim.Adam(model.parameters(), lr = 0.001 , weight_decay = 0.0001)  
scheduler = ReduceLROnPlateau(optimizer, mode = 'min', factor = 0.1 , patience = 5)
</code></pre>
<pre><code class="lang-auto">ResNet_arch.conv0.weight torch.Size([16, 3, 3, 3])
ResNet_arch.norm0.rho torch.Size([1, 16, 1, 1])
ResNet_arch.norm0.weight torch.Size([1, 16, 1, 1])
ResNet_arch.norm0.bias torch.Size([1, 16, 1, 1])
ResNet_arch.block1.s_layers.conv1.weight torch.Size([16, 16, 3, 3])
ResNet_arch.block1.s_layers.norm1.rho torch.Size([1, 16, 1, 1])
ResNet_arch.block1.s_layers.norm1.weight torch.Size([1, 16, 1, 1])
ResNet_arch.block1.s_layers.norm1.bias torch.Size([1, 16, 1, 1])
ResNet_arch.block1.s_layers.conv2.weight torch.Size([16, 16, 3, 3])
ResNet_arch.block1.s_layers.norm2.rho torch.Size([1, 16, 1, 1])
ResNet_arch.block1.s_layers.norm2.weight torch.Size([1, 16, 1, 1])
ResNet_arch.block1.s_layers.norm2.bias torch.Size([1, 16, 1, 1])
ResNet_arch.block2.s_layers.conv1.weight torch.Size([16, 16, 3, 3])
ResNet_arch.block2.s_layers.norm1.rho torch.Size([1, 16, 1, 1])
ResNet_arch.block2.s_layers.norm1.weight torch.Size([1, 16, 1, 1])
ResNet_arch.block2.s_layers.norm1.bias torch.Size([1, 16, 1, 1])
ResNet_arch.block2.s_layers.conv2.weight torch.Size([16, 16, 3, 3])
ResNet_arch.block2.s_layers.norm2.rho torch.Size([1, 16, 1, 1])
ResNet_arch.block2.s_layers.norm2.weight torch.Size([1, 16, 1, 1])
ResNet_arch.block2.s_layers.norm2.bias torch.Size([1, 16, 1, 1])
ResNet_arch.block3.d_layers.conv1.weight torch.Size([32, 16, 3, 3])
ResNet_arch.block3.d_layers.norm1.rho torch.Size([1, 32, 1, 1])
ResNet_arch.block3.d_layers.norm1.weight torch.Size([1, 32, 1, 1])
ResNet_arch.block3.d_layers.norm1.bias torch.Size([1, 32, 1, 1])
ResNet_arch.block3.d_layers.conv2.weight torch.Size([32, 32, 3, 3])
ResNet_arch.block3.d_layers.norm2.rho torch.Size([1, 32, 1, 1])
ResNet_arch.block3.d_layers.norm2.weight torch.Size([1, 32, 1, 1])
ResNet_arch.block3.d_layers.norm2.bias torch.Size([1, 32, 1, 1])
ResNet_arch.block3.d_downsample.convd.weight torch.Size([32, 16, 1, 1])
ResNet_arch.block3.d_downsample.normd.rho torch.Size([1, 32, 1, 1])
ResNet_arch.block3.d_downsample.normd.weight torch.Size([1, 32, 1, 1])
ResNet_arch.block3.d_downsample.normd.bias torch.Size([1, 32, 1, 1])
ResNet_arch.block4.s_layers.conv1.weight torch.Size([32, 32, 3, 3])
ResNet_arch.block4.s_layers.norm1.rho torch.Size([1, 32, 1, 1])
ResNet_arch.block4.s_layers.norm1.weight torch.Size([1, 32, 1, 1])
ResNet_arch.block4.s_layers.norm1.bias torch.Size([1, 32, 1, 1])
ResNet_arch.block4.s_layers.conv2.weight torch.Size([32, 32, 3, 3])
ResNet_arch.block4.s_layers.norm2.rho torch.Size([1, 32, 1, 1])
ResNet_arch.block4.s_layers.norm2.weight torch.Size([1, 32, 1, 1])
ResNet_arch.block4.s_layers.norm2.bias torch.Size([1, 32, 1, 1])
ResNet_arch.block5.d_layers.conv1.weight torch.Size([64, 32, 3, 3])
ResNet_arch.block5.d_layers.norm1.rho torch.Size([1, 64, 1, 1])
ResNet_arch.block5.d_layers.norm1.weight torch.Size([1, 64, 1, 1])
ResNet_arch.block5.d_layers.norm1.bias torch.Size([1, 64, 1, 1])
ResNet_arch.block5.d_layers.conv2.weight torch.Size([64, 64, 3, 3])
ResNet_arch.block5.d_layers.norm2.rho torch.Size([1, 64, 1, 1])
ResNet_arch.block5.d_layers.norm2.weight torch.Size([1, 64, 1, 1])
ResNet_arch.block5.d_layers.norm2.bias torch.Size([1, 64, 1, 1])
ResNet_arch.block5.d_downsample.convd.weight torch.Size([64, 32, 1, 1])
ResNet_arch.block5.d_downsample.normd.rho torch.Size([1, 64, 1, 1])
ResNet_arch.block5.d_downsample.normd.weight torch.Size([1, 64, 1, 1])
ResNet_arch.block5.d_downsample.normd.bias torch.Size([1, 64, 1, 1])
ResNet_arch.block6.s_layers.conv1.weight torch.Size([64, 64, 3, 3])
ResNet_arch.block6.s_layers.norm1.rho torch.Size([1, 64, 1, 1])
ResNet_arch.block6.s_layers.norm1.weight torch.Size([1, 64, 1, 1])
ResNet_arch.block6.s_layers.norm1.bias torch.Size([1, 64, 1, 1])
ResNet_arch.block6.s_layers.conv2.weight torch.Size([64, 64, 3, 3])
ResNet_arch.block6.s_layers.norm2.rho torch.Size([1, 64, 1, 1])
ResNet_arch.block6.s_layers.norm2.weight torch.Size([1, 64, 1, 1])
ResNet_arch.block6.s_layers.norm2.bias torch.Size([1, 64, 1, 1])
ResNet_arch.fc__.weight torch.Size([25, 64])
ResNet_arch.fc__.bias torch.Size([25])
</code></pre>

ptrblck: <p>You could use the same <a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options">per-parameter option</a> but could pass the actual parameters manually to the <code>optimizer</code> instead of all parameters from a layer.</p>

Harshit_Goyal: <p>Can you please show the syntax with few parameters. I saw this documentation but couldn’t figure it out.<br>
Thanks</p>