Why squeeze() is working for cross_entropy but showing error for logits?

username: akib62
cooked: <p>Hello, I am using <code>squeeze()</code> before calculating <code>loss</code>. My code is given below and giving an error<br>
<code>ValueError: Target size (torch.Size([1, 256, 383])) must be the same as input size (torch.Size([1, 1, 256, 383]))</code></p>
<pre><code class="lang-auto">    def training_step(self, batch, batch_nb):
        x, y = batch
        y_hat = self.forward(x)
        y_label = y.squeeze(axis = 1)
        loss = F.cross_entropy(y_hat, y_label) if self.n_classes &gt; 1 else \
            F.binary_cross_entropy_with_logits(y_hat, y_label)
        tensorboard_logs = {'train_loss': loss}
        return {'loss': loss, 'log': tensorboard_logs}
</code></pre>
<p>But, when I am changing the <code>y_label</code> to <code>y</code> only for <code>logist</code> <code>binary_cross_entropy_with_logitsy_hat, y)</code><br>
Working fine!</p>
<p>Why <code>y_label</code> is showing the error for <code>logits</code> but working fine for <code>cross_entropy</code>.</p>
<p>However, if I do not use <code>squeeze()</code> it gives the same error for both loss functions.</p>
<p>For your kind information<br>
x torch size <code>torch.Size([1, 3, 256, 383])</code><br>
y torch size <code>torch.Size([1, 3, 256, 383])</code><br>
y_label torch size <code>torch.Size([1, 256, 383])</code></p>
<p>Thanks in advance.</p>

username: KFrank
cooked: <p>Hi Akib!</p>
<aside class="quote no-group quote-modified" data-username="akib62" data-post="1" data-topic="99706" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3bc359/48.png" class="avatar"> akib62:</div>
<blockquote>
<p><code>ValueError: Target size (torch.Size([1, 256, 383])) must be the same as input size (torch.Size([1, 1, 256, 383]))</code><br>
…<br>
But, when I am changing the <code>y_label</code> to <code>y</code> only for <code>logist</code> <code>binary_cross_entropy_with_logitsy_hat, y)</code><br>
Working fine!</p>
<p>Why <code>y_label</code> is showing the error for <code>logits</code> but working fine for <code>cross_entropy</code>.</p>
</blockquote>
</aside>
<p><code>cross_entropy()</code> and <code>binary_cross_entropy_with_logits</code> take<br>
<code>target</code>s (your <code>y_label</code>) with differing shapes, and, in fact, with<br>
different numbers of dimensions.</p>
<p>If <code>input</code> (your <code>y_hat</code>) has shape <code>[nBatch, nClass, height, width]</code>,<br>
<code>cross_entropy_loss()</code> expects a <code>target</code> with shape<br>
<code>[nBatch, height, width]</code> (no <code>nClass</code> dimension), whose values<br>
are integer class labels that run from <code>0</code> to <code>nClass - 1</code>, while<br>
<code>binary_cross_entropy_with_logits()</code> expects a <code>target</code> of shape<br>
<code>[nBatch, nClass, height, width]</code> (the same shape as the <code>input</code>),<br>
whose values are probabilities, that is floats that run from <code>0.0</code> to <code>1.0</code><br>
(and can be exactly <code>0.0</code> and <code>1.0</code> in the most common use case).</p>
<p>(Note, using <code>binary_cross_entropy_with_logits()</code> with an <code>nClass</code><br>
dimension is appropriate for the <em>multi-label</em>, multi-class use case.  In<br>
the more common single-label, two-class (binary) use case, you would<br>
not use an <code>nClass</code> dimension, neither <code>nClass = 1</code> nor <code>nClass = 2</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>y torch size <code>torch.Size([1, 3, 256, 383])</code><br>
y_label torch size <code>torch.Size([1, 256, 383])</code></p>
</blockquote>
</aside>
<p>As a side comment, this isn’t consistent with:</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">        y_label = y.squeeze(axis = 1)
</code></pre>
</blockquote>
</aside>
<p>In your <code>y</code>, <code>axis = 1</code> (usually called <code>dim</code>) has size <code>3</code>, so <code>squeeze()</code><br>
does nothing, and <code>y_label</code> will have shape <code>[1, 3, 256, 383]</code>.</p>
<p>(If <code>y</code> had, instead, shape <code>[1, 1, 256, 383]</code>, <code>squeeze (axis = 1)</code><br>
would, in fact, remove the <code>axis = 1</code> singleton dimension.)</p>
<p>A comment on your approach:  What I <em>think</em> you’re trying to do is<br>
switch to <code>binary_cross_entropy_with_logits</code> when you have<br>
a two-class (binary) problem.  While, in principle, your scheme could<br>
be made to work with some complexity and attention to detail, I<br>
don’t think it’s worth the bother.</p>
<p>If you <em>know</em> that you have a binary problem, just build that into your<br>
code.  The output of your model (<code>y_hat</code>) and your <code>target</code> will both<br>
have shape <code>[nBatch, height, width]</code> (note no <code>nClass</code> dimension),<br>
and you will use <code>binary_cross_entropy_with_logits()</code>.</p>
<p>If you want to write code that can be used for a multi-class problem,<br>
but will work for various numbers of classes, including <code>nClass = 2</code>,<br>
just implement the general multi-class solution using<br>
<code>cross_entropy_loss()</code>.  (Again, in this case, the output of your<br>
model should have shape <code>[nBatch, nClass, height, width]</code>,<br>
and your target should have shape <code>[nBatch, height, width]</code>.)</p>
<p><code>cross_entropy_loss()</code> works just fine for the <code>nClass = 2</code> (binary)<br>
case, and any minor efficiency gains you might get by switching<br>
to <code>binary_cross_entropy_with_logits()</code> when <code>nClass = 2</code><br>
just aren’t worth the hassle.</p>
<p>Best.</p>
<p>K. Frank</p>

username: akib62
cooked: <p><a class="mention" href="/u/kfrank">@KFrank</a> Thank you very much.</p>
<p>I learned a lot from your comment!</p>

username: akib62
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>, if I want to use the <code>MSE loss</code> instead of <code>cross_entropy</code> then how I have to write the code?</p>
<p>More specifically, are there any variants for <code>MSE loss</code>. For example, we are using <code>cross_entrpoy</code> for <code>binary-class classification</code> and <code>logits</code> for <code>multi-class classification</code>.</p>
<p>So, <code>MSE loss</code> has 2 types (one for binary class nad another for multi-class)?</p>
<p>Sorry, for this kind of silly questions!</p>

username: KFrank
cooked: <p>Hi Akib!</p>
<aside class="quote no-group" data-username="akib62" data-post="4" data-topic="99706" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3bc359/48.png" class="avatar"> akib62:</div>
<blockquote>
<p>if I want to use the <code>MSE loss</code> instead of <code>cross_entropy</code> then how I have to write the code?</p>
</blockquote>
</aside>
<p>It is a BAD idea to use <code>MSELoss</code> for classification.  Here is the<br>
reason: <code>MSELoss</code> cares about how far the prediction is from the<br>
target.  But in a classification problem you generally don’t have<br>
that notion of distance.</p>
<p>If a tree is 60 years old, I have done better if I predict it to be 55<br>
years old than if I predict it to be 40 years old.  Both predictions<br>
are wrong, but the 40-years-old prediction is worse.  <code>MSELoss</code><br>
takes this into account.</p>
<p>But if I classify a bird as fish, that’s wrong, but no better or worse<br>
than classifying the bird as a reptile.  There’s no sense of distance<br>
or some concept of a class being closer to some classes and farther<br>
from others.  (Conceptually there could be, but there is not in the<br>
typical pytorch classification use case.)</p>
<p>However …</p>
<p>If you <em>wan</em>t to use <code>MSELoss</code> for classification, you would convert<br>
your integer class labels to floats (so <code>target</code> would be a <code>FloatTensor</code><br>
of shape <code>[nBatch]</code>, with no <code>nClass</code> dimension).  Your model would<br>
have a single output so it would output batch predictions of shape<br>
<code>[nBatch, 1]</code> which you would <code>squeeze()</code> to get shape <code>[nBatch]</code>.</p>
<p>Your (floating-point) <code>target</code> would have values running from <code>0.0</code><br>
to <code>nClass - 1</code>.  If your <code>target</code> value were, say, <code>2.0</code>, then a predicted<br>
value of <code>2.0</code> would be perfect, and <code>MSELoss</code> would return <code>0.0</code>.  A<br>
prediction of <code>1.95</code> or <code>2.05</code> would be quite good, while a prediction<br>
of <code>0.0</code> (or even <code>-1.0</code>) or <code>4.0</code> would be worse.</p>
<p>But for a <em>classification</em> problem, for a <code>target</code> of <code>2.0</code>, would you<br>
really consider a prediction of <code>0.0</code> to be worse than a prediction of<br>
<code>1.0</code>, or are they both just wrong because they both predict the wrong<br>
class?</p>
<p>Best.</p>
<p>K. Frank</p>

username: akib62
cooked: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a> thank you very much. No, I am not thinking to use <code>MSE</code> for classification problems. I am thinking it to use for regression related problem.</p>