Build Custom param groups for optimizer

username: Jaideep_Valani
cooked: <p>In the optmizer we have param groups methods.</p>
<ol>
<li>
<p>What are the default param groups if we dont give any in the optimizer?</p>
</li>
<li>
<p>How to build the custom param groups ?</p>
</li>
</ol>

username: MariosOreo
cooked: <p>Hello,</p>
<p>You could get some references from <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/optimizer.py#L44" rel="nofollow noopener">here.</a><br>
If you don’t give any to the optimizer, it will raise an error of empty parameter list.</p>
<p>You could pass a list of parameters or a dict of parameters with other hyperparameters (such as lr, momentum …) to the oprimizer.</p>

username: Jaideep_Valani
cooked: <p>Thanks it looks like<br>
add_param_Groups could be of some help.<br>
Is it possilble to give eg.</p>
<p>Assume we have nn.Sequential( L1,l2,l3,l4,l5)<br>
i want three groups<br>
(L1) ,(l2,l3,l4),(l5)</p>
<p>High level logic i can think about is we may need to build the param dictionary  for each layer groups and pass as param to add param groups.<br>
Please help build code if you can<br>
THanks in advance…</p>

username: MariosOreo
cooked: <p>Sure, I wrote a demo for your use case. I assume the Sequential is considered as Seq in your network.</p>
<pre><code class="lang-auto">model = MyNetwork()
optimizer = torch.optim.SGD(
    [
        {'params': model.Seq[0].parameters(), 'lr': 0.1},
        {'params': model.Seq[1:3].parameters(), 'lr': 0.01},
        {'params': model.Seq[4].parameters(), 'lr': 0.001},
    ])
print(optimizer.param_groups)
</code></pre>

username: Jaideep_Valani
cooked: <p>thanks so we have to pass list of param dict where dict represents a group here.<br>
few more things</p>
<ol>
<li>
<p>if there are sub children like in resnet ,how do i check if layer has sub children<br>
Resnet is made up of multiple layers where each layer itself is a sequence of models.<br>
so if i send some layer and want to validate if it has children or not ?</p>
</li>
<li>
<p>If below is my model</p>
</li>
</ol>
<p>Sequential(<br>
(0): Lambda()<br>
(1): Conv2d(1, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2))<br>
(2): ReLU()<br>
(3): Conv2d(8, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))<br>
(4): ReLU()<br>
(5): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))<br>
(6): ReLU()<br>
(7): Conv2d(32, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))<br>
(8): ReLU()<br>
(9): AdaptiveAvgPool2d(output_size=1)<br>
(10): Lambda()<br>
(11): Linear(in_features=32, out_features=10, bias=False)<br>
)<br>
Is there a way to check  by name of  layer that i have reached adaptiveavgpool2d<br>
i intend to form group dynamically  ,user will give some indices  &amp; by default  if no indices given then default groups would be Input and all layers after input  till last classification layer .If i give indices then three group minimum</p>
<ol>
<li>input</li>
<li>after input to till a layer before Avg layer </li>
<li>from Avg to last layer</li>
</ol>
<p>some more peace of excercise for you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: MariosOreo
cooked: <p>If you want to check a specific layer by name in Sequential, you could try on this:</p>
<pre><code class="lang-auto">for module in model.children():
    if isinstance(module, nn.AdaptiveAvgPool2d):
        # your use case
</code></pre>
<p>There are many ways to check <code>modules</code>, <code>params</code> and <code>buffers</code> in network, you could check them on docs and play with them to meet your use case.</p>

username: sudonto
cooked: <p>Hi, do you think the order of param matters?</p>

username: Katerina_K
cooked: <p>I understand how we can split parameters into groups layer-wise, but what if we want to hand pick e.g some parameters from the 1st layer, some from the 2nd, etc. I am having trouble doing that.  I have a dictionary in the same form as the network parameters with True or False flags for each parameter. I want to group all those parameters whose corresponding flag is true together and those with a false flag together in a different group. Any ideas on how to do that? Thank you in advance for any suggestions!</p>