Softmax + Cross-Entropy Loss

username: vsant
cooked: <p>Hello,</p>
<p>My network has Softmax activation plus a Cross-Entropy loss, which some refer to Categorical Cross-Entropy loss. See:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b818ab685d1eaea37f849134d9b22a2d8e5b619c.png" data-download-href="https://discuss.pytorch.org/uploads/default/b818ab685d1eaea37f849134d9b22a2d8e5b619c" title="Softmax_Cross-Entropy"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b818ab685d1eaea37f849134d9b22a2d8e5b619c_2_690x140.png" alt="Softmax_Cross-Entropy" data-base62-sha1="qgAHWPHqnC9Q0CR19OpiqNE2JBy" width="690" height="140" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b818ab685d1eaea37f849134d9b22a2d8e5b619c_2_690x140.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/8/b818ab685d1eaea37f849134d9b22a2d8e5b619c_2_1035x210.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/8/b818ab685d1eaea37f849134d9b22a2d8e5b619c.png 2x" data-dominant-color="E9EEF3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Softmax_Cross-Entropy</span><span class="informations">1162×236 48.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In binary classification, do I need one-hot encoding to work in a network like this in PyTorch? I am using Integer Encoding. Just as matter of fact, here are some outputs WITHOUT Softmax activation (batch = 4):</p>
<p>outputs:  tensor([[ 0.2439,  0.0890],<br>
[ 0.2258,  0.1119],<br>
[-0.2149,  0.2282],<br>
[ 0.0222, -0.1259]]</p>
<p>And here are some outputs WITH Softmax (Softmax activation before Cross-Entropy):</p>
<p>outputs:  tensor([[0.3662, 0.6338],<br>
[0.4209, 0.5791],<br>
[0.4611, 0.5389],<br>
[0.5497, 0.4503]]</p>
<p>As expected, the elements of the n-dimensional output Tensor lie in the range [0,1] and sum to 1. Honestly, I see no remarkable in the loss functions in both situations.</p>
<p>But, the whole point is to know whether I can use integer encoding with Softmax + Cross-Entropy in PyTorch.</p>
<p>Thanks.</p>

username: Epoching
cooked: <p>Do keep in mind that <code>CrossEntropyLoss</code> does a softmax for you. (It’s actually a <code>LogSoftmax</code> + <code>NLLLoss</code> combined into one function, see <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 1.9.0 documentation</a>). Doing a Softmax activation before cross entropy is like doing it twice, which can cause the values to start to balance each other out as so:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30b2199d63ee48d41455d788f6b2540eb3299ab1.png" alt="image" data-base62-sha1="6WMuLnjFv9wOvJyphmoXqdnEmQN" width="428" height="332"></p>
<p>As for your question, are you saying there’s no point in using softmax since your outputs without it are already between 0-1? I mean technically I guess you don’t, but using an activation function is a nice way to ensure bounds in case your network spits out something unexpected</p>

username: vsant
cooked: <p>Hello,</p>
<p>Thank you for your response. Yes, I do know that <code>Cross-EntropyLoss</code> has a softmax “embedded”. Without the Softmax, the outputs are not necessarily between 0 an 1. See below:</p>
<p>outputs: tensor([[ 0.2439, 0.0890],<br>
[ 0.2258, 0.1119],<br>
[-0.2149, 0.2282],<br>
[ 0.0222, -0.1259]]</p>
<p>The piece of code related to this post is like that:</p>
<pre><code class="lang-auto">def train_model(model, criterion, ...)
...
   outputs = model(inputs)
   _, preds = torch.max(outputs, 1)  
   loss = criterion(outputs, labels)
...


# main code

my_model = my_model.to(device)
my_criterion = nn.CrossEntropyLoss()
...
my_model = train_model(my_model, my_criterion,...)


</code></pre>
<p>Above, I use integer encoding. I am just wondering whether I can use integer encoding with Softmax + Cross-Entropy in PyTorch. The point is that some authors, by using other frameworks rather than PyTorch, state that we MUST use one-hot encoding for binary classification, because, eventually, we may have all outputs equal. For instance, see this Stack Overflow post (Keras): <a href="https://stackoverflow.com/questions/44322611/keras-cnn-same-output" class="inline-onebox" rel="noopener nofollow ugc">python - keras CNN same output - Stack Overflow</a></p>
<p>Hence, the explanation here is the incompatibility between the softmax as output activation and binary_crossentropy as loss function. To solve this, we must rely on one-hot encoding otherwise we will get all outputs equal (this is what I read). But I used Cross-Entropy here. In my case, as shown above, the outputs are not equal. Hence, it seems there is no problem in using integer encoding in PyTorch in a situation like that. But, I would like to be sure.</p>
<p>Thanks.</p>

username: Epoching
cooked: <p>Oh sorry I misunderstood what you said. Yea from my experience I always use Integer Encoding (e.g. <code>[2, 3, 1, 1]</code> vs <code>[[0 0 1 0], [0 0 0 1], [0 1 0 0], [0 1 0 0]</code> for <code>CrossEntropyLoss</code> and <code>BCELoss</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>The docs for <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">CrossEntropyLoss</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html#torch.nn.BCELoss" rel="noopener nofollow ugc">BCELoss</a> / <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#torch.nn.BCEWithLogitsLoss" rel="noopener nofollow ugc">BCEWithLogitsLoss</a> both show examples using integer encoding the labels rather than one-hot encoding</p>
<p>The predictions that go into these loss functions are just the raw logits from the model</p>

username: vsant
cooked: <p>Thank you again for your reply.</p>

username: Epoching
cooked: <p>No problem! I learned something new too haha</p>