Per Parameter Options in libtorch C++

username: melspec
cooked: <p>I have essentially the same problem as <a href="https://discuss.pytorch.org/t/how-to-set-different-learning-rate-for-weight-and-bias-in-one-layer/13450">this post</a> about per-parameter options in the optimizer.  I am using the c++ api (libtorch) instead of python.  Is there an example anywhere of how to set the learning rate per layer using libtorch?  Thanks!</p>

username: ptrblck
cooked: <p>Based on <a href="https://github.com/pytorch/pytorch/blob/45c5bac87049a9a0593d4d5f060cc8d4b0f83db8/test/cpp/api/optim.cpp#L153-L155">this</a>, <a href="https://github.com/pytorch/pytorch/issues/40098#issuecomment-645042879">this</a>, and <a href="https://github.com/pytorch/pytorch/issues/35640#issuecomment-605953263">this</a> code snippets you should be able to add new parameter groups via:</p>
<pre><code class="lang-auto"> auto&amp; params_groups = optimizer.param_groups();
 params_groups.push_back(OptimizerParamGroup(params));
</code></pre>
<p>and set the learning rate via:</p>
<pre><code class="lang-auto">static_cast&lt;torch::optim::AdamOptions&amp;&gt;(adam_optimizer.param_groups()[0].options()).lr()
</code></pre>
<p>for <code>Adam</code> (you might want to change it to the appropriate optimizer type).</p>

username: melspec
cooked: <p>Thank you.  This just sets the learning rate generically for all layers.  I use this call already to change the learning rate for all layers from epoch to epoch.  However, I need certain layers to have a different learning rate.  I’m porting models from caffe and they seem to be very sensitive to this.  In caffe, it was a per layer learning rate multiplier, mentioned in the post that I linked to.  In python you can send in parameters for each layer, I cannot figure out how to create that same functionality in c++.</p>

username: ptrblck
cooked: <p>You should still be able to use the posted code to set the learning rate for each parameter group (which could be separate layers):</p>
<pre><code class="lang-auto">for (auto param_group : optimizer.param_groups()) {
  static_cast&lt;torch::optim::AdamOptions &amp;&gt;(param_group.options()).lr(new_lr);
}
# or alternatively via the index in the first code snippet
</code></pre>

username: melspec
cooked: <p>So I should push back a unique param_group for each layer of my model?  The optimizer by default only has one param_group, not one per layer.</p>

username: ptrblck
cooked: <p>Yes. This would be the same approach used in the Python API (<a href="https://pytorch.org/docs/stable/optim.html#per-parameter-options">docs for per-parameter options</a>):</p>
<pre><code class="lang-python">optimizer = torch.optim.SGD([
    {'params': [nn.Parameter(torch.zeros(1))]},
    {'params': [nn.Parameter(torch.ones(1))], 'lr': 1e-3}
], lr=1e-2, momentum=0.9)

for param_group in optimizer.param_groups:
    print(param_group)

&gt; {'params': [Parameter containing:
  tensor([0.], requires_grad=True)], 'lr': 0.01, 'momentum': 0.9, 'dampening': 0, 'weight_decay': 0, 'nesterov': False}
  {'params': [Parameter containing:
  tensor([1.], requires_grad=True)], 'lr': 0.001, 'momentum': 0.9, 'dampening': 0, 'weight_decay': 0, 'nesterov': False}
</code></pre>

username: melspec
cooked: <p>Getting very close, I will post a complete solution once I get it.  The new parameter groups created are going onto the CPU (the model was moved to the GPU prior to optimizer construction per the documentation) instead of the GPU.  So when I run the model, it complains that the weights are on the CPU.  How can I create the new parameters on the GPU instead?</p>
<p>As a side not for anyone interested I ended up using the call optimizer.add_param_group since it handles the creation of the options automatically.  The code posted above with “params_groups.push_back” actually fails when you try to index into it with “Expected has_options() to be true, but got false”</p>

username: scirocc19900305
cooked: <p>thanks for the share. But in my case is a little different.<br>
I create a tensor not a module that need to track the grad,and wanna set different lr of that tensor.But I got problem when “params_groups.push_back(tensor)”,how can I fix it?</p>
<pre><code class="lang-auto">    torch::Tensor actionLogSTD;//that is the problem
    actionLogSTD = register_parameter("actionLogSTD", torch::zeros((1, actionDim)) - 0.5);//require_grad=true

    auto&amp; params_groups = p_optimizer-&gt;param_groups();
    p_optimizer-&gt;parameters().push_back((*p_actor)-&gt;actionLogSTD);
    params_groups.push_back(torch::optim::OptimizerParamGroup((*p_sharedBILSTM)-&gt;parameters()));
    params_groups.push_back(torch::optim::OptimizerParamGroup((*p_actor)-&gt;actor_fc_net1-&gt;parameters()));
    params_groups.push_back(torch::optim::OptimizerParamGroup((*p_critic)-&gt;critic_fc_net1-&gt;parameters()));
    params_groups.push_back(torch::optim::OptimizerParamGroup((*p_critic)-&gt;actionLogSTD));//this lead to error,tensor type not matching OptimizerParamGroup type
    for (auto param_group : p_optimizer-&gt;param_groups())
        static_cast&lt;torch::optim::AdamOptions &amp;&gt;(param_group.options()).lr(lr);
</code></pre>

username: ptrblck
cooked: <p>I don’t know what the error message is, but based on the <a href="https://pytorch.org/cppdocs/api/classtorch_1_1optim_1_1_optimizer_param_group.html">docs</a> I guess you might need to wrap the parameter into a <code>std::vector&lt;Tensor&gt;</code> first before passing it to the <code>OptimizerParamGroup</code>.</p>

username: scirocc19900305
cooked: <p>thanks man!that really works!<br>
I simply change<br>
" params_groups.push_back(torch::optim::OptimizerParamGroup((*p_actor)-&gt;actionLogSTD));"<br>
to<br>
" params_groups.push_back(torch::optim::OptimizerParamGroup({(*p_actor)-&gt;actionLogSTD}));"</p>

username: Abhishek_Shivakumar
cooked: <pre><code>    // Create the first parameter group with drive_db_
    std::vector&lt;torch::Tensor&gt; params1 = { drive_db_ };

    // Create custom options for this parameter group
    auto options1 = std::make_unique&lt;torch::optim::AdamOptions&gt; (learning_rate);

    // Initialize the Adam optimizer with the parameter group
    std::vector&lt;torch::optim::OptimizerParamGroup&gt; param_groups;
    param_groups.emplace_back (torch::optim::OptimizerParamGroup ({drive_db_}, std::make_unique&lt;torch::optim::AdamOptions&gt; (learning_rate)));

    // Construct the optimizer without moving param_groups
    torch::optim::Adam optimizer (param_groups, torch::optim::AdamOptions (learning_rate)); // base learning rate
</code></pre>
<p>Unsure how I’d fix this… let’s see</p>