Multi-class classification

Rojin: <p>I am trying to do a multi-class classification in pytorch. The code runs fine, but the accuracy is not good. I was wondering if my code is correct?<br>
The input to the model is a matrix of 2000x100 and the output is a 1D tensor with the index of the label ex: tensor([2,5,31,…,7]) =&gt; 2000 elements</p>
<pre><code class="lang-auto"># another multi-class classification
class MultiClass(nn.Module):
    
    def __init__(self, x_dim, z_dim):
        super(MultiClass, self).__init__()

        self.cf1  = nn.Linear(z_dim, z_dim)
        self.cf1N = nn.BatchNorm1d(num_features = z_dim)
        self.cf1D = nn.Dropout(0.5) 
        
        self.cf2  = nn.Linear(z_dim, z_dim)
        self.cf2N = nn.BatchNorm1d(num_features = z_dim)
        self.cf2D = nn.Dropout(0.5)
        self.cf3  = nn.Linear(z_dim, 32)

        
    def classifier(self,z):
        return F.softmax((self.cf3(self.cf2D(F.relu(self.cf2N(self.cf2(self.cf1D(F.relu(self.cf1N(self.cf1(z)))))))))))
            

    def forward(self, z):
        return self.classifier(z)
    

def MultiClass_loss_function(out, label):

    secLoss = F.cross_entropy(out, label.long())

    return secLoss
</code></pre>

tymokvo: <p>You might consider inheriting from the <code>Sequential</code> module and using a layer list to make your module a bit easier to reason about:</p>
<p>(I have <code>nn.functional</code> imported as <code>nnf</code>)</p>
<pre><code class="lang-auto">class MultiClass(nn.Sequential):
    
    def __init__(self, z_dim):
        super(MultiClass, self).__init__()

        layers = [
            ("cf1" , nn.Linear(z_dim, z_dim)),
            ("cf1N", nn.BatchNorm1d(num_features = z_dim)),
            ("r1"  , nn.ReLU()),
            ("cf1D", nn.Dropout(0.5) ),
            ("cf2" , nn.Linear(z_dim, z_dim)),
            ("cf2N", nn.BatchNorm1d(num_features = z_dim)),
            ("r2"  , nn.ReLU()),
            ("cf2D", nn.Dropout(0.5)),
            ("cf3" , nn.Linear(z_dim, 32)),
        ]
        
        [self.add_module(name, layer) for (name, layer) in layers]

        
    def classifier(self,z):
        return nnf.softmax(self(z))
</code></pre>
<p><code>x_dim</code> also does not seem to be used anywhere?</p>
<pre><code class="lang-auto">MultiClass(
  (cf1): Linear(in_features=2000, out_features=2000, bias=True)
  (cf1N): BatchNorm1d(2000, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (cf1D): Dropout(p=0.5)
  (cf2): Linear(in_features=2000, out_features=2000, bias=True)
  (cf2N): BatchNorm1d(2000, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (cf2D): Dropout(p=0.5)
  (cf3): Linear(in_features=2000, out_features=32, bias=True)
)
</code></pre>
<p>How is your data structured?</p>
<p>Most classification tasks are accomplished by predicting probabilities over a tensor with the rank of the number of classes rather than predicting the index of the class directly. E.g. <a href="https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a" rel="nofollow noopener">ImageNet</a> has 1000 categories so a label for <code>gold fish</code> would have the value <code>1.0</code> at the index <code>1</code> on a rank <code>1000</code> tensor.</p>
<p>Sorry if I’m telling you what you already know! But some more detail about the code/task would be helpful. <img src="https://discuss.pytorch.org/images/emoji/apple/smile_cat.png?v=9" title=":smile_cat:" class="emoji" alt=":smile_cat:"></p>
<p>Another thing to remember:</p>
<p>The domain of the softmax function is [0, 1]. So the result of your <code>.classifier()</code> method on your example label would be something like:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; nnf.softmax(torch.tensor([2, 5, 31, 7]).float())
tensor([2.5437e-13, 5.1091e-12, 1.0000e+00, 3.7751e-11])
</code></pre>

Oli: <p>Heeello,</p>
<p>I agree with <a class="mention" href="/u/tymokvo">@tymokvo</a> that more background into the problem helps (or an explanation on why this isn’t needed).</p>
<p>Luckily I found one possible error in your code. Your forward/classifier function squeezes the output into a softmax whilst your loss function,  cross_entropy, doesn’t want this. See the docs about <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="nofollow noopener">cross entropy and NLLLoss</a></p>

tymokvo: <aside class="quote no-group" data-username="Oli" data-post="3" data-topic="47565">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/48/9773_2.png" class="avatar"> Oli:</div>
<blockquote>
<p>a softmax whilst your loss function, cross_entropy, doesn’t want this. See the docs about <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">cross entropy and NLLLoss</a></p>
</blockquote>
</aside>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/unamused.png?v=12" title=":unamused:" class="emoji" alt=":unamused:" loading="lazy" width="20" height="20"> I totally missed the loss function using <code>F.cross_entropy</code> at the bottom of the code snippet. Sorry about that! Disregard pedantic note about label shapes above.</p>
<p>I do think the function is a bit confusing in this case. The docs say:</p>
<blockquote>
<p>input has to be a Tensor of size either (minibatch,C)(minibatch, C)(minibatch,C) or (minibatch,C,d1,d2,…,dK)(minibatch, C, d_1, d_2, …, d_K)(minibatch,C,d1​,d2​,…,dK​) with K≥1K \geq 1K≥1 for the K-dimensional case (described later).</p>
</blockquote>
<p>and</p>
<blockquote>
<p>This criterion expects a class index in the range [0,C−1][0, C-1][0,C−1] as the target for each value of a 1D tensor of size minibatch;</p>
</blockquote>
<p>So the network is still supposed to be returning probabilities over a dimension with rank = (your number of classes). As opposed to what you described:</p>
<aside class="quote no-group" data-username="Rojin" data-post="1" data-topic="47565">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rojin/48/11062_2.png" class="avatar"> Rojin:</div>
<blockquote>
<p>the output is a 1D tensor with the index of the label ex: tensor([2,5,31,…,7])</p>
</blockquote>
</aside>
<p>But the <em>label</em> is as you describe, a tensor of integer indices, where each entry is the integer class label that corresponds to the ground truth for the matching input.</p>
<p>Since the shapes all seem to match (32 on the output), it sounds as <a class="mention" href="/u/oli">@Oli</a> mentioned an activation function problem.</p>

Rojin: <p><a class="mention" href="/u/oli">@Oli</a> <a class="mention" href="/u/tymokvo">@tymokvo</a></p>
<p>Thanks for the replies, I removed the softmax layer, not sure if that is the right thing to do because I know that softmax is used for multi-class classification.</p>
<p>Basically I am trying to build a super simple multi-class classification in pytorch! I have done this in Keras easily but I’m  not sure what I’m doing wrong here.</p>
<p>Here is the new model, and I am sure it is wrong because the prediction is always the same.</p>
<pre><code class="lang-auto">class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(NeuralNet, self).__init__()
        self.layer1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.layer2 = nn.Linear(hidden_size, num_classes)
        
    def forward(self, x):
        out = self.layer1(x)
        out = self.relu(out)
        out = self.layer2(out)
        return out
    

train_x = Variable(torch.from_numpy(np.asanyarray(x_train[x_train.columns[:-1]]))).float()
train_y = Variable(torch.LongTensor(x_train_label)).long()

test_x = Variable(torch.from_numpy(np.asanyarray(x_test[x_test.columns[:-1]]))).float()
test_y = Variable(torch.LongTensor(x_test_label)).long()

</code></pre>
<p>The input and label are like this:</p>
<pre><code class="lang-auto">train_x, x_train_label
</code></pre>
<pre><code class="lang-auto">(tensor([[ 4.3734, 14.7227, 14.0051,  ..., 10.8181, 10.1554, 10.5231],
         [ 1.3698, 16.1158, 16.2395,  ..., 10.0263, 10.1859, 11.0192],
         [ 5.4781, 13.8524, 13.4969,  ..., 10.2149, 10.1836, 11.0694],
         ...,
         [ 0.9918, 15.1710, 12.5128,  ...,  9.5816,  9.3345, 10.6840],
         [ 0.0000, 11.7931, 11.7571,  ...,  9.9917, 10.2152, 11.6365],
         [ 0.8513, 13.1537, 11.8680,  ...,  9.6107, 10.6265, 10.2632]]),
 tensor([28, 22, 19,  ..., 26, 29,  5]))

</code></pre>
<pre><code class="lang-auto">tr_latent_X = data_utils.TensorDataset(train_x, train_y)
te_latent_X = data_utils.TensorDataset(test_x,  test_y)
</code></pre>
<pre><code class="lang-auto">train_loader_X = torch.utils.data.DataLoader(dataset=tr_latent_X,
                                             batch_size=bs,
                                             shuffle=False)

test_loader_X = torch.utils.data.DataLoader(dataset=tr_latent_X,
                                             batch_size=bs,
                                             shuffle=False)
</code></pre>
<pre><code class="lang-auto">vae = NeuralNet(5000 ,1000, 32)

loss_function = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(vae.parameters(), lr = 0.02)
</code></pre>
<pre><code class="lang-auto">def train(epoch):
    vae.train()

    train_loss = 0

    for batch_idx, (data,label) in enumerate(train_loader_X):

        optimizer.zero_grad()

        out = vae(data)
        loss = loss_function(out, label)

        loss.backward()
        train_loss += loss.item()
        optimizer.step()

        if batch_idx % 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader_X.dataset),
                100. * batch_idx / len(train_loader_X), loss.item() / len(data)))
    print('====&gt; Epoch: {} Average loss: {:.4f}'.format(epoch, train_loss / len(train_loader_X.dataset)))

</code></pre>
<pre><code class="lang-auto">for epoch in range(1, 10):
    train(epoch)
</code></pre>
<pre><code class="lang-auto">Train Epoch: 1 [0/6483 (0%)]	Loss: 0.093940
Train Epoch: 1 [6400/6483 (98%)]	Loss: 0.051790
====&gt; Epoch: 1 Average loss: 3.8580
Train Epoch: 2 [0/6483 (0%)]	Loss: 0.052003
Train Epoch: 2 [6400/6483 (98%)]	Loss: 0.051288
====&gt; Epoch: 2 Average loss: 0.0511
Train Epoch: 3 [0/6483 (0%)]	Loss: 0.051706
Train Epoch: 3 [6400/6483 (98%)]	Loss: 0.051227
====&gt; Epoch: 3 Average loss: 0.0508
Train Epoch: 4 [0/6483 (0%)]	Loss: 0.051529
Train Epoch: 4 [6400/6483 (98%)]	Loss: 0.051222
====&gt; Epoch: 4 Average loss: 0.0507
Train Epoch: 5 [0/6483 (0%)]	Loss: 0.051466
Train Epoch: 5 [6400/6483 (98%)]	Loss: 0.051220
====&gt; Epoch: 5 Average loss: 0.0507
Train Epoch: 6 [0/6483 (0%)]	Loss: 0.051450
Train Epoch: 6 [6400/6483 (98%)]	Loss: 0.051218
====&gt; Epoch: 6 Average loss: 0.0507
Train Epoch: 7 [0/6483 (0%)]	Loss: 0.051448
Train Epoch: 7 [6400/6483 (98%)]	Loss: 0.051215
====&gt; Epoch: 7 Average loss: 0.0507
Train Epoch: 8 [0/6483 (0%)]	Loss: 0.051449
Train Epoch: 8 [6400/6483 (98%)]	Loss: 0.051212
====&gt; Epoch: 8 Average loss: 0.0507
Train Epoch: 9 [0/6483 (0%)]	Loss: 0.051451
Train Epoch: 9 [6400/6483 (98%)]	Loss: 0.051210
====&gt; Epoch: 9 Average loss: 0.0507
</code></pre>
<pre><code class="lang-auto">out = []
k = []

for data,label in test_loader_X:
    out.append(vae(data))
    k.append(label)
</code></pre>
<p>and  here is how the prediction on the test looks like:</p>
<pre><code class="lang-auto">[tensor([[ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857],
         [ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857],
         [ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857],
         ...,
         [ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857],
         [ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857],
         [ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857]],
        grad_fn=&lt;AddmmBackward&gt;),
 tensor([[ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857],
         [ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857],
         [ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857],
         ...,
         [ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857],
         [ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857],
         [ 0.3506,  0.4214,  0.6405,  ..., -0.5726,  0.3580, -1.2857]],
        grad_fn=&lt;AddmmBackward&gt;),
</code></pre>

god_sp33d: <p>The network and the flow seems ok to me. Can you try the following things?</p>
<ol>
<li>Reduce the learning rate to 0.0005 or so and see the result ?</li>
<li>Why don’t you use a simple optimizer initially like SGD and see the output ?</li>
<li>Can you provide more information about the input data, I see values ranging from 0 - 12 (may be) did you perform any normalization before passing the data to the neural network ?</li>
</ol>

Rojin: <p>I switched the optimizer to SGD, the result is still the same. the learning rate is 0.0001.</p>
<p>The input data is log2(gene expression matrix)</p>
<p>Do you thing that I need softmax layer at the end?</p>

god_sp33d: <p>No, cross entropy loss has softmax built in it.</p>
<p>I am not aware of gene expression data but may be normalizing the data to vary between 0 - 1 might help.</p>

Oli: <aside class="quote no-group" data-username="Rojin" data-post="5" data-topic="47565">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rojin/48/11062_2.png" class="avatar"> Rojin:</div>
<blockquote>
<pre><code class="lang-auto">for data,label in test_loader_X:
    out.append(vae(data))
    k.append(label)
</code></pre>
</blockquote>
</aside>
<p>Hi Rojin. So you shouldn’t <em>always</em> use a softmax layer in your model since you will apply it once in your loss function. However, when you are doing your testing, your loss function isn’t used. That’s why you have to apply softmax “manually”. I suggest that you do a predict function in your model that looks something like below. (Could be a typo or something in there, never tested it)</p>
<pre><code class="lang-auto">def predict(self, x):
  with torch.no_grad():
    outp = self.forward(x)
    return F.softmax(outp)
</code></pre>