One Vs rest (multi-class classification) with Fully connected network and sigmoid PyTorch

username: Mohamed_Ragab
cooked: <p>Hi, all I am trying to implement the one-vs-all classification scheme without using softmax. I have 10 classes. So, I did 10 binary classifications, but it doesn’t work properly and I don’t know why? I will be grateful and thankful if someone can help me with this. Here is a snapshot:</p>
<pre><code class="lang-auto">for i in range(num_classes):
                binary_cross_entropy =  nn.BCELoss() #
                tgt_class_label = torch.eq(labels, i).float()
                print(f"before_sigmoid: {pred_out[:,i]}")
                print(f"normalized: {pred[:,i]}")
                print(f"True Labels: {tgt_class_label}")
                print(tgt_class_label)
                class_loss = binary_cross_entropy(pred[:,i],tgt_class_label)
                Loss+=class_loss
</code></pre>

username: Chirath_Pansilu
cooked: <p>It’s hard to tell from this snapshot what’s really going on . Can you provide at least the training procedure (or more) . It will be helpful .</p>

username: Mohamed_Ragab
cooked: <p><a class="mention" href="/u/chirath_pansilu">@Chirath_Pansilu</a>. This is the training part of the model</p>
<pre><code class="lang-auto">   for inputs, labels in train_dl:
        # send data and labels to the gpu
        src = inputs.to(device)
        labels = labels.to(device)
        # reset the gradients
        optimizer.zero_grad()
        # output predictions 
        pred_out,feas = model(src)
        #apply sigmoid to the output predictions of each class
        pred = sigmoid_fun(pred_out) 
        for i in range(num_classes):
                # define binary cross entropy loss
                binary_cross_entropy =  nn.BCELoss() #
                # make the ith class as 1 and all others as 0
                tgt_class_label = torch.eq(labels, i).float()
                # compute cross entropy between the target label and the ith class
                class_loss = binary_cross_entropy(pred[:,i],tgt_class_label)
                Loss+=class_loss
        # Loss = criterion(pred , labels)
        Loss.backward(retain_graph=True)
        optimizer.step()
</code></pre>

username: Chirath_Pansilu
cooked: <p>Hey , It looks like you have got a theory part wrong. When you do One vs Rest classification you have to make, in this case, 10 different models as there are 10 different classes to distinguish between .</p>
<p>But you have only created a 1 model and calculating loss for different labels. That’s not how One vs Rest works. You have to create 10 different Models.And then train each model for each class label. One for 1st class another for 2nd class and so on. Then train each model separately . After that when you want to predict a outcome you have to run <code>input</code> via each model and pick the model with the highest probability.</p>

username: Mohamed_Ragab
cooked: <p>Thank you so much <a class="mention" href="/u/chirath_pansilu">@Chirath_Pansilu</a> for this clear explanation. Its now clear to me. But I have additional concern: in my architecture my model has two parts: CNN-feature extractor and Linear layer as classier. This classifier has 10 output neurons, I specified each neuron to predict the corresponding class. If this still not correct and I have to use single model can I freeze the CNN feature extractor and make 10 classifier models i.e., 1 for each class?</p>

username: Chirath_Pansilu
cooked: <p>Hi, I’m not sure I understand what you meant by this.</p>
<aside class="quote no-group" data-username="Mohamed_Ragab" data-post="5" data-topic="85311">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohamed_ragab/48/8127_2.png" class="avatar"> Mohamed_Ragab:</div>
<blockquote>
<p>This classifier has 10 output neurons, I specified each neuron to predict the corresponding class.</p>
</blockquote>
</aside>
<p>You said that you are doing one vs rest classification and you have mentioned above that there are 10 output neurons. This is a normal multi class classifier .</p>
<p>But if you want to train one vs rest classifiers, yes you can freeze CNN after it has trained one time because it is only used as a feature extractor.</p>
<p>If I have misunderstood something please tell.<br>
Thank You</p>

username: Mohamed_Ragab
cooked: <p>Yes, you are right. Following your suggestion I have done the following, please see if I am doing that right:<br>
1- I created a module list of linear layers equal to the number of classes, i.e. 10 linear layers<br>
2- Each linear layer linear( feature_dim, 1)<br>
3- I pass the features to all the modules, and calculate the loss with respect to each independently.<br>
the model:</p>
<pre><code class="lang-auto">class one_vs_all(nn.Module):
    def __init__(self, input_dim=32,out_dim=1, num_classes=10, dropout=0):
        super(one_vs_all, self).__init__()
        # not the best model...
        self.input_dim = input_dim
        self.out_dim =out_dim
        self.sigmoid = nn.Sigmoid()
        self.num_classes=num_classes
        self.base_model= nn.Linear(self.input_dim,self.out_dim) 
        self.classifiers =  nn.ModuleList([self.base_model for i in range(self.self.num_classes)])
    def forward(self,class_index,x):
        predictions = self.classifiers[class_index](x)
        logits =self.sigmoid(predictions)
        return logits
</code></pre>
<p>and here is the training function</p>
<pre><code class="lang-auto">def train(model, train_dl, optimizer, criterion,config,device):
    model.train()
    epoch_loss = 0
    class_accuracy = {f'class{k+1}': [] for k in range(len(num_classes))}
    features =[];train_labels=[];total_pred= []
    Loss=0
    for inputs, labels in train_dl:
        src = inputs.to(device)
        labels = labels.to(device)
        #loss and score
        # one_vs_all
        for idx in range(num_classes):
            optimizer.zero_grad()
            tgt_class_label = torch.eq(labels, idx).float()
            logits        = model(idx,inputs)
            class_loss = binary_cross_entropy(logits,tgt_class_label)
            Loss+=class_loss
            class_accuracy[f'D{idx+1}'].append() = accuracy_score(tgt_class_label.cpu().detach().numpy(),pred.argmax(axis=1).cpu().detach().numpy())
            class_loss.backward()
            optimizer.step()
        epoch_loss += Loss.item()
        epoch_accuracy += Accuracy
        total_pred.append(pred.argmax(axis=1))
        features.append(feas)
        train_labels.append(labels)
    return epoch_loss / len(train_dl), epoch_accuracy/len(train_dl), torch.cat(features),torch.cat(total_pred),torch.cat(train_labels)```</code></pre>