Two output nodes for binary classification

shirui-japina: <p>I refer to the model in the paper <a href="https://arxiv.org/abs/1611.06651" rel="nofollow noopener">‘Deep Learning for the Classification of Lung Nodules’</a>.<br>
The model is like this:</p>
<pre><code class="lang-auto">    def forward(self, input_image):
        # (channels, height, width)
        ##
        out = self.conv_1(input_image) # 20 * 44 * 44
        out = self.bn_conv_1(out)

        out = self.pooling_1(out) # 20 * 22 * 22
        out = self.bn_pooling_1(out)

        ##
        out = self.conv_2(out) # 50 * 16 * 16
        out = self.bn_conv_2(out)

        out = self.pooling_2(out) # 50 * 8 * 8
        out = self.bn_pooling_2(out)

        ##
        out = self.conv_3(out) # 500 * 2 * 2
        out = self.bn_conv_3(out)

        out = self.pooling_3(out) # 500 * 1 * 1
        out = self.bn_pooling_3(out)

        out = F.relu(out)
        out = self.conv_4(out) # 2 * 1 * 1

        out = F.softmax(out)
</code></pre>
<p>My doubt is that:</p>
<ol>
<li>
<p>This is a <strong>binary classification</strong> model, but <strong>the output has two nodes</strong>.<br>
(Generally, there is only one output node in the binary classification model, and the prediction result is judged by <strong>greater than or less than 0.5</strong>.)</p>
</li>
<li>
<p>Although I don’t know if this is a key consideration, there is no fully connected layer in the model.</p>
</li>
</ol>
<p>Considering the above, how should I design the code for the loss function?<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

ptrblck: <ol>
<li>For a binary classification use case, you could use a single output and a threshold (as you’ve explained) or alternatively you could use a multi-class classification with just two classes, so that each class gets its output neuron. The loss functions for both approaches would be different.<br>
In the first case (single output), you would use e.g. <code>nn.BCEWithLogitsLoss</code> and the output tensor shape should match the target shape.<br>
In the latter case, you would use e.g. <code>nn.CrossEntropyLoss</code> and the target tensor shape should contain the class indices in the range <code>[0, nb_classes-1]</code> and miss the “class dimension” (usually the channel dim).</li>
</ol>
<p>Both approaches expect logits, so you should remove your softmax layer and just pass the last output to the criterion.</p>
<ol start="2">
<li>A final linear layer is not strictly necessary, if you make sure to work with the right shapes of your output and target.</li>
</ol>

shirui-japina: <p>Thanks for your reply.</p>
<blockquote>
<p>In the latter case, you would use e.g.  <code>nn.CrossEntropyLoss</code>  and the target tensor shape should contain the class indices in the range  <code>[0, nb_classes-1]</code>  and miss the “class dimension” (usually the channel dim).</p>
</blockquote>
<p>I got it.</p>
<blockquote>
<p>Both approaches expect logits, so you should remove your softmax layer and just pass the last output to the criterion.</p>
</blockquote>
<p>Thanks for your suggestion, <em><strong>‘so you should remove your softmax layer and just pass the last output to the criterion.’</strong></em>, that’s really the point.</p>
<p>In addition to these, there are questions:</p>
<blockquote>
<p>Both approaches expect logits, so you should remove your softmax layer and just pass the last output to the criterion.</p>
</blockquote>
<ol>
<li>Does above means the description <a href="https://pytorch.org/docs/master/nn.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">here ( CrossEntropyLoss)</a> like below?</li>
</ol>
<blockquote>
<p>The input is expected to contain raw, unnormalized scores for each class.</p>
</blockquote>
<ol start="2">
<li>But why is it? In my opinion, the input of <strong>CrossEntropyLoss</strong> is:<br>
(1) the prediction (the output of model); (2) the label.<br>
And  the prediction data <em><strong>should be converted to probability</strong></em>, which means that I need to place a layer of <strong>soft-max layer</strong> at the end of the model.</li>
</ol>
<p>Is that my wrong understanding that the input data of CrossEntropyLoss should be probability?<img src="https://discuss.pytorch.org/images/emoji/apple/open_mouth.png?v=12" title=":open_mouth:" class="emoji" alt=":open_mouth:" loading="lazy" width="20" height="20"><br>
(So  that put the model output data to the CrossEntropyLoss directly is <strong>correct</strong> from mathematical theory?)<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<p>Or alternatively the CrossEntropyLoss function in PyTorch will do that (computes the probability of prediction of model) <strong>automatically</strong>?<img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=12" title=":upside_down_face:" class="emoji" alt=":upside_down_face:" loading="lazy" width="20" height="20"></p>

shirui-japina: <p>I looked at the <a href="https://pytorch.org/docs/master/_modules/torch/nn/functional.html#cross_entropy" rel="noopener nofollow ugc">source code</a> and came to the conclusion.</p>
<pre><code class="lang-auto">def cross_entropy(input, target, weight=None, size_average=None, ignore_index=-100,
                  reduce=None, reduction='mean'):
    if size_average is not None or reduce is not None:
        reduction = _Reduction.legacy_get_string(size_average, reduce)
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
</code></pre>
<p>From the source code, I’m sure that</p>
<blockquote>
<p>Or alternatively the CrossEntropyLoss function in PyTorch will do that (computes the probability of prediction of model)  <strong>automatically</strong> ?<img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=12" title=":upside_down_face:" class="emoji" alt=":upside_down_face:" loading="lazy" width="20" height="20"></p>
</blockquote>
<p>This is the right answer.<img src="https://discuss.pytorch.org/images/emoji/apple/100.png?v=12" title=":100:" class="emoji" alt=":100:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=12" title=":tada:" class="emoji" alt=":tada:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/upside_down_face.png?v=12" title=":upside_down_face:" class="emoji" alt=":upside_down_face:" loading="lazy" width="20" height="20"></p>