Where in the model to place torch.nn.Softplus or torch.softmax

username: MikeTensor
cooked: <p>I am wondering where to place functions like <code>nn.Softplus</code> or <code>torch.Softmax</code> in the speicfic models exactly and why.</p>
<p>In model-descriptions I saw that usually people apply these kind of functions <strong>at the end of a model</strong> (after the last <code>Linear</code>-Layer.)</p>
<p>For my problems it seems that it doesn’t work.<br>
I get an output from my <code>vgg_feature_extractor</code>; extract some <code>roi</code>s (region of interests) and send these to my two models.</p>
<p>Model A: Classifier. (one single <code>Linear</code>-Layer; nothing else).<br>
with Loss: <code>criterion_classification = nn.CrossEntropyLoss()</code></p>
<p>Model B: Distance_regressor (three <code>Linear</code>-Layers; nothing else).<br>
with loss: <code>criterion_distance = nn.SmoothL1Loss(reduction='mean')</code></p>
<p>If I add the <code>Softmax</code> to the end of model A and train it: The <code>train_acc</code> stucks at 0,57x (not moving anymore).</p>
<p>Right code seems to be (only classifier)…</p>
<pre><code class="lang-auto">        output_class_pred = classifier(rois_out)

        # 2. Calculate  and accumulate loss
        loss = loss_fn(output_class_pred, class_label_dl)
        train_loss += loss.item()

        # 3. Optimizer zero grad
        optimizer.zero_grad()

        # 4. Loss backward
        loss.backward()

        # 5. Optimizer step
        optimizer.step()

        y_pred_class_no = torch.argmax((torch.softmax(output_class_pred, dim=1)), dim=1)
        train_acc += (y_pred_class_no == class_label_dl).sum().item()/len(output_class_pred)

    # Adjust metrics to get average loss and accuracy per batch
    train_loss = train_loss / len(dataloader)
    train_acc = train_acc / len(dataloader)
</code></pre>
<ul>
<li>(with <code>Linear</code>-Layer is the output of <code>modelA</code> in this scenario)</li>
</ul>
<p>If I do it this way like quoted it seems to work. After a couple of epochs I get 87–89 % train_acc (with 9 classes).</p>
<p>So I am wondering why some people add the <code>Softmax</code> to the end of their <code>model</code>?<br>
Or where I should place it correctly …</p>
<p>For my <code>distance_regressor</code> I am still not sure where to place the <code>Softplus</code> (which I want to use to only get positive distance_values).</p>
<p>If placed at the end of <code>modelB</code> (the <code>distance_regressor</code>) –and before the <code>loss_dist_fn</code> it seems that the <code>predicted_distance</code> remains 0 (since the <code>models_output</code> is negative.).<br>
– But which seems for me more logic, since I want to get positive values only.</p>
<p>If I place it after the <code>loss_dist_fn</code> the values seems to quicker adjust to positive value-predictions.</p>
<p>Any hint where the two functions shall be usually placed? (end of model, or after loss_fn, …)</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="MikeTensor" data-post="1" data-topic="174190">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/5daacb/48.png" class="avatar"> MikeTensor:</div>
<blockquote>
<p>So I am wondering why some people add the <code>Softmax</code> to the end of their <code>model</code>?</p>
</blockquote>
</aside>
<p>This is usually wrong, as <code>nn.CrossEntropyLoss</code> expects raw logits as the model output and <code>nn.NLLLoss</code> expects log-probabilties (so <code>F.log_softmax</code> would be needed) for multi-class classification use cases.<br>
This also explains why your training is stuck when the softmax activation is added.</p>

username: MikeTensor
cooked: <p>Thanks a lot!!</p>
<ol>
<li>So where shall I place the <code>Softplus</code> for <code>modelB</code> (regression) and the <code>nn.SmoothL1Loss()</code>?</li>
</ol>
<p><strong>And by the way:</strong><br>
2) Is there a smooth way to exclude a  <code>prediction_output</code> –of a specific class– from the distance_regression loss_calculation?<br>
So the loss for the <code>distance_regressor</code> should be only calculated for <code>class</code>: <code>0–7</code>, but not for <code>class</code> <code>8</code>.</p>