Regarding softmax layer

Ganga: <ul>
<li>how can i find whether my layer has softmax or not?</li>
</ul>

ptrblck: <p><code>nn.CrossEntropyLoss</code> applies <code>F.log_softmax</code> internally on the input. The usual “layers” such as <code>nn.ConvXd</code>, <code>nn.Linear</code> etc. are not applying any non-linearity for you.<br>
The same does of course not apply for custom user-defined layers.<br>
If you are unsure about a specific layers, please refer to the docs, which would mention if an activation function is applied internally.</p>

Ganga: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> , I read the docs of my github repo they didnt mention about non-linearity function applied internally.<br>
Can you please once go through my github repo code to have a glance whether my softmax function applied to last layer</p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">
      <a href="https://github.com/jiecaoyu/XNOR-Net-PyTorch" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars3.githubusercontent.com/u/10011346?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/jiecaoyu/XNOR-Net-PyTorch" target="_blank" rel="nofollow noopener">jiecaoyu/XNOR-Net-PyTorch</a></h3>

<p>PyTorch Implementation of XNOR-Net. Contribute to jiecaoyu/XNOR-Net-PyTorch development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I am using Imagenet dataset

ptrblck: <p>In the ImageNet example I cannot find the usage of a softmax and <code>nn.CrossEntropyLoss</code> is used, which looks right.</p>

Ganga: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your reply, from this conversation I came to know that <strong>softmax</strong> is used only for calculating the <strong>CrossEntropyLoss</strong>  rather than classification</p>
<p>Can I print or get the softmax output which is used in <strong>CrossEntropyLoss</strong> just for curiosity</p>

ptrblck: <p>Yes, you can add a <code>softmax</code> or <code>log_softmax</code> operation and e.g. print the output values.<br>
As long as you don’t feed these values into <code>nn.CrossEntropyLoss</code> there won’t be a problem.</p>

Ganga: <ul>
<li>
<p>I have sent my output to softmax function  and I am getting positive value in range(0,1)</p>
</li>
<li>
<p>I have inference code for taking multiple images in a folder, there I don’t have <strong>loss</strong> <strong>function</strong>(<strong>Cross entropy</strong>) neither <strong>Softmax</strong> Do I want to add cross entropy in my inference code? or it is ok</p>
</li>
<li>
<p>By adding <strong>softmax</strong> in my inference code will there be any change in my <strong>prediction</strong> or its(softmax) just used to convert logit into probability?</p>
</li>
</ul>
<p><strong>here is the code for Inference</strong></p>
<pre><code class="lang-auto">    import torchvision
    import torchvision.transforms as transforms
    import torchvision.datasets as datasets
    normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                std=[1./255., 1./255., 1./255.])

    torchvision.set_image_backend('PIL')

    data_transforms = {
    'predict': transforms.Compose([
        transforms.Resize((256, 256)),
        transforms.CenterCrop(input_size),
        transforms.ToTensor(),
        normalize,
        ])
    }

    dataset = {'predict' : datasets.ImageFolder("/content/XNOR-Net-PyTorch/ImageNet/networks/", data_transforms['predict'])}
    dataloader = {'predict': torch.utils.data.DataLoader(dataset['predict'], batch_size=args.batch_size, shuffle=False, num_workers=args.workers,pin_memory=True)}
batch_time = AverageMeter()
    losses = AverageMeter()
    top1 = AverageMeter()
    top5 = AverageMeter()

    model.eval()
    end = time.time()
    global bin_op
    bin_op = util.BinOp(model)
    bin_op.binarization()
    #input=img

    for input, labels in dataloader['predict']:
      with torch.no_grad():
        input_var = torch.autograd.Variable(input)
        
        
    # compute output
      output = model(input_var)</code></pre>

ptrblck: <ol>
<li>
<p>That sounds right.</p>
</li>
<li>
<p>If you want to calculate the “test loss” (and have targets during inference), you can use the criterion to calculate it. Otherwise, if you just want to predictions, it’s not necesary.</p>
</li>
<li>
<p>The predictions won’t change and you can get the predicted class index via <code>torch.argmax(output)</code>, where <code>output</code> can be the logits or the probabilities.</p>
</li>
</ol>

Ganga: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, for your all replies</p>

prya: <p>Referring to the example by <a class="mention" href="/u/ganga">@Ganga</a>, I understand that during training,<br>
the <strong>cross entropy loss</strong> is obtained and in that function <strong>softmax</strong> is also calculated before getting the loss.</p>
<p>However, during <strong>model inferencing</strong>, there is <em>no explicit</em> usage of softmax in the code but it output from <code>model(input)</code> gives probabilities?</p>
<p>How does the model give probabilities? Where is the softamx declared that when the model is called (for inferencing) softmax is executed on the logits?</p>

ptrblck: <p>You are correct that <code>nn.CrossEntropyLoss</code> will internally apply <code>F.log_softmax</code> and thus no <code>softmax</code> activation is used in the model.<br>
The model thus outputs logits which have values in the range <code>[-Inf, +inf]</code>.<br>
If you want to calculate the probability for each class during inference, you can apply <code>F.softmax</code> on the ouput and process them further (just don’t calculate the loss with them).<br>
However, to get the predicted classes, you could use <code>torch.argmax(output, dim=1)</code>, which will return the same predicted class index using the logits or probabilities, since the <code>softmax</code> will not change the order of logits and probabilitiies, and the highest logit will get the highest probability.</p>

pdc_87: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> Is there any advantage to activate the last layer with softmax if I already used CrossEntropyLoss as loss function? Internallly, it uses log_softmax. What do you think?</p>

ptrblck: <p>No, using <code>softmax</code> on your outputs and passing it to <code>nn.CrossEntropyLoss</code> is wrong and might stall your training. As described before: you could still use it in case you want to print the probabilities or use them in another way, but don’t calculate the loss with the <code>softmax</code> output using <code>nn.CrossEntropyLoss</code> or <code>nn.NLLLoss</code>.</p>