Getting integer labels as model output

ritesh313: <p>I am working on a classification problem and right now I’ve coded the network as a classifier which has n (n = number of classes) nodes in the output layer (and then I use softmax and get the label prediction through maximum index). I’m wondering if I can add one more layer in the end with single output and apply softmax to the previous (n nodes) layer and get the label as a model output. Is this possible or is there any other way to get integer labels as model output?</p>

Dwight_Foster: <p>You could do that but you would have to train the last linear layer to give the correct outputs. Also the range on the linear layers is infinite so it could give a number more than the number of layers that is why people use softmax.</p>

ritesh313: <blockquote>
<p>but you would have to train the last linear layer to give the correct outputs.</p>
</blockquote>
<p>You mean the integer output layer that I want to add?</p>
<blockquote>
<p>the range on the linear layers is infinite</p>
</blockquote>
<p>Yeah, that’s the problem, it either goes to infinity or stays 0 if I apply sigmoid activation layer.</p>
<p>This is my current network:</p>
<pre><code class="lang-auto">x = self.activation(nn.Linear(16*16, 512)(x))
x = self.activation(nn.Linear(512, 256)(x))
x = self.activation(nn.Linear(256, 128)(x))
x = torch.log_softmax(nn.Linear(128, 65)(x).squeeze(), dim = 1)
# x = nn.Linear(65,1)(x)
 _, max_val = torch.max(x, dim = 1)
x = max_val.to(torch.float32)
# x = nn.Linear(1,1)(max_idx.to(torch.float32))
</code></pre>
<p>The last layer on which I’ve applied softmax, was my original last layer, I’ve 65 classes and it performs really well. But I am trying to convert that to integer output from the model, I tried a few things (commented lines) but none of them works. I just wanted to know if this can be done (and produce accurate results) at all or not.</p>

Dwight_Foster: <p>Yes I mean the integer output layer that you would add you would have to train it like all the other layers. You could freeze the rest of your model and just train that layer and it might work. But you would have to train it to see. One possibility is that you could apply a sigmoid function to that layer which would make it in the range of 0 to 1. Then you could multiply it by the number of labels. That would ensure the range is between 0 and 19 but then you would have to train it using something like an MSE loss to make sure it is outputting the correct classes. Overall it is not worth it softmax is a simpler solution in my opinion.</p>

KFrank: <p>Hi Ritesh!</p>
<aside class="quote no-group" data-username="ritesh313" data-post="1" data-topic="113153" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ritesh313/48/29351_2.png" class="avatar"> ritesh313:</div>
<blockquote>
<p>is there any other way to get integer labels as model output?</p>
</blockquote>
</aside>
<p>If by “model output” you mean the model predictions that you feed<br>
in to your loss function, then you don’t want to do this.  If you do<br>
manage to have your model output values that are exactly integers<br>
(even if they are encoded as floating-point numbers) those outputs<br>
won’t be usefully differentiable so your model won’t train.</p>
<aside class="quote no-group">
<blockquote>
<p>(and then I use softmax and get the label prediction through maximum index).</p>
</blockquote>
</aside>
<p>As an aside, you don’t need the <code>softmax()</code> before the “maximum<br>
index” (<code>argmax()</code>) to get the label prediction.  <code>softmax()</code> doesn’t<br>
change the <em>order</em> of its arguments, so you can leave it out.</p>
<p>Best.</p>
<p>K. Frank</p>

ritesh313: <p><a class="mention" href="/u/dwight_foster">@Dwight_Foster</a><br>
Alright, I’ll try that, I too think its not worth it but this is required in this particular case. Thanks for your time!</p>
<p><a class="mention" href="/u/kfrank">@KFrank</a></p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="113153">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>those outputs won’t be usefully differentiable so your model won’t train.</p>
</blockquote>
</aside>
<p>That’s exactly what happened when I tried it, I just wanna try a few things and see if I can work around it.</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="113153">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>As an aside, you don’t need the <code>softmax()</code> before the “maximum<br>
index” (<code>argmax()</code>) to get the label prediction. <code>softmax()</code> doesn’t<br>
change the <em>order</em> of its arguments, so you can leave it out.</p>
</blockquote>
</aside>
<p>I am doing that to get the index of the maximum value. Maybe I didn’t understand what you said?</p>

KFrank: <p>Hi Ritesh!</p>
<aside class="quote no-group quote-modified" data-username="ritesh313" data-post="6" data-topic="113153" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ritesh313/48/29351_2.png" class="avatar"> ritesh313:</div>
<blockquote>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="113153">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>As an aside, you don’t need the <code>softmax()</code> before the “maximum<br>
index” (<code>argmax()</code>) to get the label prediction. <code>softmax()</code> doesn’t<br>
change the <em>order</em> of its arguments, so you can leave it out.</p>
</blockquote>
</aside>
<p>I am doing that to get the index of the maximum value. Maybe I didn’t understand what you said?</p>
</blockquote>
</aside>
<p>Let’s call the output of your model <code>logits</code>.  What I mean is:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.argmax (torch.nn.functional.softmax (logits)) == torch.argmax (logits)
</code></pre>
<p>That is, even though the values returned by <code>softmax()</code> are different<br>
than its inputs, the <em>order</em> of the returned values is the same as the<br>
<em>order</em> of the inputs.  So leaving out <code>softmax()</code> won’t change the<br>
result fo <code>argmax()</code>.  (I’m just noting that while using <code>softmax()</code> is<br>
valid, it’s not necessary.)</p>
<p>Best.</p>
<p>K. Frank</p>

ritesh313: <aside class="quote no-group" data-username="KFrank" data-post="8" data-topic="113153">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p><code>torch.argmax (torch.nn.functional.softmax (logits)) == torch.argmax (logits)</code></p>
</blockquote>
</aside>
<p>Ohhh, I just realized that. Yeah, that makes sense, Thank you for letting me know.</p>