Text generator lstm without softmax

username: cevvalu
cooked: <p>Hi, i am trying to get next predicted word in the sequence. Is it possible to get maximum value from dense layer’s output without softmax since i am not doing any classification</p>

username: vdw
cooked: <p>Well, you do classification. The vocabulary represents your classes, and the predicted class is the next word.</p>

username: cevvalu
cooked: <p>i see…But i do not use a softmax in output dense layer. I am picking maximum value from output which equals vocabulary size.</p>

username: ptrblck
cooked: <p>Yes, that would be possible, if you return the class logits directly.<br>
The <code>argmin</code> and <code>argmax</code> values of the logits would be the same as after applying a softmax.</p>

username: cevvalu
cooked: <p>so we can say “dense layer without softmax can return argmax values of the logits and this works just like softmax” ?</p>

username: cevvalu
cooked: <p>"yhat = np.argmax(yhat.to(“cpu”).detach().numpy()) "<br>
i am using argmax for converting  probability to integer and i think this is what you said right?<br>
since this integer value maps a word in my vocabulary i am able to predict the next word this way without a sofmax i guess.</p>

username: ptrblck
cooked: <p>It depends what your use case is. The actual values will of course be different, but the maximal index will be the same, so you can use it to get the predicted class (most likely class).</p>
<p>However, you need to be careful, if you want to calculate a loss with these values and should use logits for e.g. <code>nn.CrossEntropyLoss</code>.</p>
<aside class="quote no-group" data-username="cevvalu" data-post="7" data-topic="82624">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cevvalu/48/20139_2.png" class="avatar"> cevvalu:</div>
<blockquote>
<p>"yhat = np.argmax(yhat.to(“cpu”).detach().numpy()) "</p>
</blockquote>
</aside>
<p>This should work, if you specify the right dimension for the <code>argmax</code> operation. Usually the class dimension would be dim1.</p>

username: cevvalu
cooked: <p>yes i  use crossEntropyLoss. i didnt understand what you mean by class dimension but my dense layer  takes input a vector of size 256 which is output of a LSTM and output is<br>
1 dimensional vocabulary sized vector</p>

username: ptrblck
cooked: <p>Your output should have the shape <code>[batch_size, nb_classes]</code>, right?<br>
If that’s the case, you should use <code>np.argmax(yhat.to('cpu').detach().numpy(), axis=1)</code> to get the predicted class for each sample in the batch.<br>
Otherwise <code>np.argmax</code> will only return a single value for the entire array.</p>

username: cevvalu
cooked: <p>output shape is : torch.Size([47, 7579]) 7579 is my vocabulary size and 47 is total number of predicted next words. For a given image and a first word of a description, model predicts second word than use seconds as an input for third and it goes like that.( Model generates captions for images and there are 5 description for each image in training text split.) . For example: these integers are encoded words from my vocabulary. And their total is 47. 2 means end of a sentence.<br>
[42, 3, 87, 169, 6, 117, 55, 393, 11, 394, 3, 27, 4474, 640, 2, 18, 313, 64, 195, 118, 2, 39, 18, 117, 64, 195, 2057, 2, 39, 18, 117, 4, 394, 19, 60, 2057, 2, 39, 18, 3, 87, 169, 313, 64, 195, 2915, 2]<br>
torch.Size([47, 7579])<br>
Sorry if its not well explained  but i tried to explain as i know <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>

username: ptrblck
cooked: <p>I assume you are using a batch size of 1, since the batch dimension is missing in the output?<br>
If so, you could still use <code>axis=1</code> to get the predicted word from the vocabulary.</p>

username: cevvalu
cooked: <p>i see. Thank you ! Is there a special name for “using a dense like this without a softmax”?<br>
Maybe i can read more about it to make it clear. I saw something  similar to it called “multi parameter regression” but not sure</p>

username: ptrblck
cooked: <p>I’ve probably phrased it wrong.<br>
The output of a linear layer, which can be used for a multi-class classification use case, will return the logits, which are raw prediction values in the range <code>[-Inf, +Inf]</code>.<br>
Internally <code>nn.CrossEntropyLoss</code>, which is usually used for this classification use case, will apply <code>F.log_softmax</code> and <code>nn.NLLLoss</code>, so the raw prediction values will be transformed to log probabilities.</p>

username: cevvalu
cooked: <p>If i understood correctly, with CrossEntropyLoss there wont be need of an extra softmax  because it applies it already during loss calculation.</p>

username: ptrblck
cooked: <p>That is correct. It will apply <code>F.log_softmax</code> internally, so you shouldn’t add a <code>softmax</code> layer in your model.</p>

username: cevvalu
cooked: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a> helped me a  lot</p>