Multilabel classifier's last layer is not tracked by pytorch

Suleymanzade: <p>Hello. I created the multilabel classifier for the tabular data where the number of the targets (outputs) can be increased - so, I cannot hardcode the output layer part, it must be flexible. The code of the model is shown below:</p>
<pre><code class="lang-auto">class MultiHeadBinaryModel(nn.Module):
    def __init__(self, input_layer, output_layer):
        super(MultiHeadBinaryModel, self).__init__()
        self.input_layer = input_layer
        self.output_layer = output_layer
        self.fc1 = nn.Linear(input_layer, 290) # 12 is the number of features
        self.bn1 = nn.BatchNorm1d(290)
        self.fc2 = nn.Linear(290, 320)
        #self.bn2 = nn.BatchNorm1d(320)
        
        self.fc3 = nn.Linear(320, 370)
        #self.bn3 = nn.BatchNorm1d(370)
        
        self.fc4 = nn.Linear(370, 420)
        #self.bn4 = nn.BatchNorm1d(420)
        self.dropout = nn.Dropout(0.15)
        # we will treat each head as a binary classifier ...
        # ... so the output features will be 1
        self.pre_out = []
        for i in range(self.output_layer):
            self.pre_out.append(nn.Linear(420, 1).to(device))
            
    def update_output_layer(self, new_output_layer):
        if new_output_layer &gt; self.output_layer:
            diff_layer = new_output_layer - self.output_layer 
            for _ in range(diff_layer):
                self.pre_out.append(nn.Linear(420, 1).to(device))
            self.output_layer = new_output_layer       
        
        
    def forward(self, x):
        x = F.relu(self.bn1(self.fc1(x)))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.dropout(x)
        x = F.relu(self.fc4(x))
        
        # each binary classifier head will have its own output
        out = [None] * self.output_layer
        for i in range(self.output_layer):
            out[i] = torch.sigmoid(self.pre_out[i](x))       
        return torch.cat(out, dim=1)
</code></pre>
<p>Initially I have 23 categories that can be recognized at once. the problem appears when I try to see the architecture of my model, let say <code>print(model)</code> that must print the all the layers - but the output layer is not seen by PyTorch tracker.</p>
<pre><code class="lang-auto">MultiHeadBinaryModel(
  (fc1): Linear(in_features=216, out_features=290, bias=True)
  (bn1): BatchNorm1d(290, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (fc2): Linear(in_features=290, out_features=320, bias=True)
  (fc3): Linear(in_features=320, out_features=370, bias=True)
  (fc4): Linear(in_features=370, out_features=420, bias=True)
  (dropout): Dropout(p=0.15, inplace=False)
)
</code></pre>
<p><strong>How to change the code for the last layer to be able to see it in the output?</strong> And I’m not sure, if it’s not tracked does it mean that their gradients are not computed? Thank you.</p>

ksmdanl: <p>Hi, depends from what kind of instance or object <code>input_layer</code> and <code>output_layer</code> are, they are not listed or registered in your <code>nn.Module</code>.</p>
<p>In your code snippet, I assume that <code>self.output_layer</code> is just some integer and not a layer, which doesn’t it part of the <code>nn.Module</code>, hence not listed.</p>
<p>Otherwise, if by output layers you mean the <code>self.pre_out</code>, it is not listed in your module either. By initializing <code>self.pre_out = nn.ModuleList()</code>, it puts the layers inside <code>self.pre_out</code> into the <code>nn.Module</code>.</p>

Suleymanzade: <p>Yes,  <code>input_layer</code> and <code>output_layer</code> are integers that represent the number of input features and output targets. When I wrote output layers I mean the <code>nn.Linear</code> layers that are inside the <code>self.pre_out</code> list. Later, I just calculate the out parameters with respect to each output: How to add them into the <em>autograd</em> by <code>nn.ModuleList()</code>?</p>

ksmdanl: <p>Instead of initializing <code>self.pre_out</code> with a normal list, initialize it with <code>nn.ModuleList()</code>.<br>
But I’m not sure I understand what you mean by adding the output to the autograd.</p>
<p>Do you want to make sure that your outputs are taken as leaf tensors? In that case, they should already be leaf tensors according to your <code>forward()</code> call.</p>
<p>Or do you want to make sure that your trainable parameters are listed?</p>

Suleymanzade: <p>Yes, I want the outputs to be taken as leaf tensors. It’s like the code of the model from this website  <a href="https://debuggercafe.com/deep-learning-architectures-for-multi-label-classification-using-pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Deep Learning Architectures for Multi-Label Classification using PyTorch - DebuggerCafe</a>  but I don’t want hardcode 23 output layers -  instead, I used the list, but the last output layers are not tracked.</p>

ksmdanl: <p>I missed your second question on your post, sorry:)<br>
The parameters are updated if they are leaf tensors. In your case <code>out</code> is a result of something in your <code>forward()</code> call, so to my understanding they should be leaf tensors and tracked inside the graph</p>

Suleymanzade: <p>Yes, exactly. And I don’t know, how to add them into the common architecture? It’s like multichannel part, but only for the output. Imagine that you have the code from this website, <a href="https://debuggercafe.com/deep-learning-architectures-for-multi-label-classification-using-pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Deep Learning Architectures for Multi-Label Classification using PyTorch - DebuggerCafe</a>  how to change it in such way to use it for any number of the outputs?</p>

ksmdanl: <p>It’s exactly how you did it with one change. As I mentioned before, if you replace <code>self.pre_out = []</code> with <code>self.pre_out = nn.ModuleList()</code>, you will be able to see the layers.</p>
<p>Example for <code>model = MultiHeadBinaryModel(1, 1)</code></p>
<pre><code class="lang-auto">MultiHeadBinaryModel(
  (fc1): Linear(in_features=1, out_features=290, bias=True)
  (bn1): BatchNorm1d(290, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (fc2): Linear(in_features=290, out_features=320, bias=True)
  (fc3): Linear(in_features=320, out_features=370, bias=True)
  (fc4): Linear(in_features=370, out_features=420, bias=True)
  (dropout): Dropout(p=0.15, inplace=False)
  (pre_out): ModuleList(
    (0): Linear(in_features=420, out_features=1, bias=True)
  )
)
</code></pre>

Suleymanzade: <p>Thank you, I will check it )</p>

Suleymanzade: <p>Thank you very much. Now, it works <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>