VGG output layer - no softmax?

username: gaurub
cooked: <p>This may seem like an extremely stupid question, but I was curious about something:</p>
<ol>
<li>In other implementations of VGG, the last layer is always put through softmax; however, in the torchvision implementation <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py" rel="nofollow noopener">here</a>, the last layer is the following:</li>
</ol>
<p>nn.Linear(4096, num_classes),</p>
<p>There is no softmax layer after this and the VGG documentation <a href="http://here" rel="nofollow noopener">https://gist.githubusercontent.com/ksimonyan/211839e770f7b538e2d8/raw/0067c9b32f60362c74f4c445a080beed06b07eb3/VGG_ILSVRC_16_layers_deploy.prototxt</a> also states that the last layer “prob” is a softmax layer.</p>
<p>I was wondering how this was possible and why this works. I realize this might be a more general math question, but I couldn’t think of anywhere else to post it, so I posted here.</p>
<p>Thanks!</p>

username: SimonW
cooked: <p>I didn’t double check, but it feels to me that it is trained with softmax. It might is just removed in the model zoo because the 4096 vector is more interesting in most use cases of a pretrained model. I still can be totally wrong though. <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: gaurub
cooked: <p>Hi,</p>
<p>Thanks for your reply. I figured that it’s trained with softmax and that it’s probably mostly used for transfer learning, so the last layer(s) will be thrown away anyway.</p>
<p>However, what confused me is it’s total absence. If what you say is true and it’s been left off intentionally with the knowledge that it will mostly be used for transfer learning, then that does, at the very least, tell me that my understanding is correct and that I can stop thinking I’m an idiot :).</p>

username: SimonW
cooked: <p>Upon digging into the details, it appears that the models are trained with <code>CrossEntropyLoss</code> which has <code>SoftMax</code> built in.</p>
<p>Imagenet training script:<br>
Criterion defined at: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L91">https://github.com/pytorch/examples/blob/master/imagenet/main.py#L91</a><br>
Loss defined at: <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L195">https://github.com/pytorch/examples/blob/master/imagenet/main.py#L195</a><br>
<code>CrossEntropyLoss</code> doc: <a href="http://pytorch.org/docs/master/nn.html#torch.nn.CrossEntropyLoss">http://pytorch.org/docs/master/nn.html#torch.nn.CrossEntropyLoss</a></p>

username: gaurub
cooked: <p>Hi,</p>
<p>Again, thank you very much for your response. I feel there’s still a gap in my understanding, however. I understand that they are trained with CrossEntropyLoss which includes Softmax, but I’m trying to understand the output from the network when a test image is passed through it. In that particular case, wouldn’t the output be different from VGG output, which has the softmax layer as part of the architecture? Yes, we will throw this away in the transfer learning use case, but as a hypothetical, say I wanted to just use VGG as a classifier; in that case, I wouldn’t get pseudo-probabilities that sum to 1, correct? If that is the case, is the main reason the softmax layer was left off merely because no one would be using it?</p>
<p>And once more, thank you for the help in understanding this.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>In case you want to use the VGG network to classify new samples, you can just call <code>argmax</code> on the logits to get the most likely class. The softmax won’t change the classification.<br>
However, if you need the probabilities, you can always call <code>softmax</code> on the net’s output.</p>

username: gaurub
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you very much! That clears up everything for me.</p>

username: SpandanMadan
cooked: <p>The reason why this is done is because you only need the softmax layer at the time of inferencing. While training, to calculate the loss you don’t need to softmax and just calculate loss without it. This way the number of computations get reduced!</p>

username: Santhosh_Kumar
cooked: <p>I try to apply softmax on output it returns probability for the single record only instead of returning the probability for whole test data,<br>
it returns probability correctly for the single record torch.softmax(output[0], dim=0).<br>
but this one not working torch.softmax(output, dim=1) its returning some other value instead of probability<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: ptrblck
cooked: <p>If you output is returned as <code>[batch_size, nb_classes]</code> (which would be the default for a classification use case), then <code>softmax(output, dim=1)</code> is the right approach, since the sum in dim1 will be 1.<br>
Each row (which corresponds to a sample in the batch) will contain the probabilities for each class.</p>

username: Santhosh_Kumar
cooked: <p>Yeah understood, Thank u so much👍 <a class="mention" href="/u/ptrblck">@ptrblck</a></p>