Trouble with Param grouping

Jaideep_Valani: <p>Hi All<br>
I am using below code to build in param groups depending upon the break points given<br>
say i have 5 seq layers in a model and these seq in turn have sub children. as shown below.<br>
. After I built the groups i pass it on to optimizer ,problem is my accuracy is getting too bad compared to having just one group by default which is having all parameters of the model.<br>
Can some one let me know if m doing groupings correctly which can logically work<br>
<strong>Model details</strong><br>
Sequential(<br>
(0): Conv2d(3, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)<br>
(1): GeneralRelu()<br>
(2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
torch.Size([128, 16, 128, 128])</p>
<p>Sequential(<br>
(0): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(1): GeneralRelu()<br>
(2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
torch.Size([128, 32, 64, 64])</p>
<p>Sequential(<br>
(0): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(1): GeneralRelu()<br>
(2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
torch.Size([128, 64, 32, 32])</p>
<p>Sequential(<br>
(0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(1): GeneralRelu()<br>
(2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
torch.Size([128, 64, 16, 16])</p>
<p>Sequential(<br>
(0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(1): GeneralRelu()<br>
(2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
torch.Size([128, 64, 8, 8])</p>
<p>Sequential(<br>
(0): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(1): GeneralRelu()<br>
(2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
torch.Size([128, 128, 4, 4])</p>
<p>Sequential(<br>
(0): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)<br>
(1): GeneralRelu()<br>
(2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
)<br>
torch.Size([128, 256, 2, 2])</p>
<p>AdaptiveAvgPool2d(output_size=1)<br>
torch.Size([128, 256, 1, 1])</p>
<p>Lambda()<br>
torch.Size([128, 256])</p>
<p>Linear(in_features=256, out_features=10, bias=True)<br>
torch.Size([128, 10])</p>
<p>Code for building groups</p>
<pre><code class="lang-auto">groups=[]
param={}
break_points=[0,7] # also tried [0,1,7]
for i,j in enumerate(break_points):
  if j!=0 and i==0:
    param['momentum']=0.9
    param['params']=model[0:j].parameters()
    groups.append(param)
    param={}
  if i+1==len(break_points):
    param['params']=model[j:].parameters()
    param['momentum']=0.9
    groups.append(param)
    break 
    
  param['params']=model[j:break_points[i+1]].parameters()
  param['momentum']=0.9
  groups.append(param)
  param={}
</code></pre>

ptrblck: <p>The first condition will never be met as far as I see it, since <code>i==0</code> only in the first iteration, where your <code>break_points</code> list contains a 0, so that <code>j!=0</code> will be <code>False</code>.</p>

Jaideep_Valani: <p>hi ptrblck<br>
thanks for reply …first condition is for case when break points are say 3,7 …so first group has to be 0 to 3 ,secomd 3 to 7 and last 7 till end .so if i pass 0 ,7 ,then groups wld be 0 to 7 and 7 till end …<br>
here m not sure if m grouping param correctly as my model performs worse with these grouping compared to having one default group of all params.</p>