Multiple parameter optimization in Multi GPU

srv902: Hello, I am using pre-trained VGGNet-16 model where the layers skipping the FC part are wrapped in torch.nn.DataParallel. The optimizer I used is: <SCODE>optimizer = optim.SGD([{'params': model.pretrained_model[0][24].parameters()},
                           {'params': model.pretrained_model[0][26].parameters()},
                           {'params': model.pretrained_model[0][28].parameters()},
                           {'params': model.regressor[0][1].parameters()},
                           {'params': model.regressor[0][4].parameters()}], lr=0.001, momentum=0.9)
<ECODE>

apaszke: 

srv902: Following is the model summary: <SCODE>Sequential (
  (0): Sequential (
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU (inplace)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU (inplace)
    (4): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU (inplace)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU (inplace)
    (9): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU (inplace)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU (inplace)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU (inplace)
    (16): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU (inplace)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU (inplace)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU (inplace)
    (23): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU (inplace)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU (inplace)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU (inplace)
    (30): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
  )
  (1): Sequential (
    (0): Dropout (p = 0.5)
    (1): Linear (25088 -> 4096)
    (2): ReLU (inplace)
    (3): Dropout (p = 0.5)
    (4): Linear (4096 -> 4096)
    (5): ReLU (inplace)
    (6): Linear (4096 -> 1000)
  )
)
<ECODE> and the class for model creation: <SCODE>  class MyModel(nn.Module):
      def __init__(self, pretrained_model):
          super(MyModel, self).__init__()
          self.pretrained_model = nn.Sequential(*list(pretrained_model.children())[:-1])
          self.pretrained_model = torch.nn.DataParallel(self.pretrained_model)
          self.regressor = nn.Sequential(net1)
  
      def forward(self, x):
          x = self.pretrained_model(x)
          x = x.view(-1,35840)
          x = self.regressor(x)
          x = x.view(-1,57,77)
          return x
<ECODE>

apaszke: 

srv902: model.pretrained_model[0][24].parameters() model.pretrained_model.module[0][24].parameters() it didn’t give any indexing error. Thank you.