Semantic Segmentation : How to train?

username: abhi8569
cooked: <p>I am using a custom model based on SegNet to do segmentation for 3 different objects. Model is returning a tensor of size “[5, 3, 120, 160] (batch size, channel, height, width)” and I have a label where each channel contain information of a particular object, something like this:</p>
<pre><code class="lang-auto">[[      [1, 0, 0, 0],
        [1, 1, 0, 1],
        [0, 0, 0, 0],
        [1, 0, 0, 0]],

       [[0, 1, 0, 0],
        [0, 0, 1, 0],
        [0, 1, 1, 0],
        [0, 0, 0, 1]],

       [[0, 0, 1, 1],
        [0, 0, 0, 0],
        [1, 0, 0, 1],
        [0, 1, 1, 0]]]
</code></pre>
<p>Label:  torch.Size([5, 3, 120, 160])</p>
<p>I am using CrossEntropy loss function to calculate the loss but I am getting this error “1only batches of spatial targets supported (non-empty 3D tensors) but got targets of size: : [5, 3, 120, 160]”.</p>
<p>As far as I can understand, I need to have a one-hot encoded vector for every pixel to calculate the loss. In my case, the label does represent one hot encoding (for a pixel, only one channel in the label will have the value ‘1’). May I know what can be done to resolve the issue?</p>

username: KFrank
cooked: <p>Hello Abishek!</p>
<p>I’m not sure that you’re describing what you are doing consistently.<br>
However, here are some comments based on some of the things<br>
you say.</p>
<aside class="quote no-group quote-modified" data-username="abhi8569" data-post="1" data-topic="74298">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abhi8569/48/21951_2.png" class="avatar"> abhi8569:</div>
<blockquote>
<p>I am using CrossEntropy loss function to calculate the loss<br>
…<br>
As far as I can understand, I need to have a one-hot encoded vector for every pixel to calculate the loss.</p>
</blockquote>
</aside>
<p>This is backwards.  For <code>CrossEntropyLoss</code> you do <em>not</em> use a<br>
one-hot-encoded label vector.  Rather, you use a single integer<br>
class label (per pixel, per etc.).</p>
<aside class="quote no-group" data-username="abhi8569" data-post="1" data-topic="74298" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abhi8569/48/21951_2.png" class="avatar"> abhi8569:</div>
<blockquote>
<p>segmentation for 3 different objects.</p>
</blockquote>
</aside>
<p>Let me say what I think you are saying in slightly different words</p>
<p>You are training a model to perform 3-class segmentation<br>
(classification).  Your input is a 120x160-pixel image (or, more<br>
precisely, a batch of <code>nBatch = 5</code> such samples).  Your output<br>
is supposed to predict which of three classes each pixels is in.</p>
<aside class="quote no-group">
<blockquote>
<p>Model is returning a tensor of size “[5, 3, 120, 160] (batch size, channel, height, width)”</p>
</blockquote>
</aside>
<p>This looks correct, but I would use slightly different words, and<br>
say that your model returns a tensor of shape<br>
<code>[nBatch, nClass, height, width]</code>.</p>
<p>That is, I understand that what you are calling <code>channel = 3</code><br>
is the number of classes into which you classify pixels.</p>
<aside class="quote no-group">
<blockquote>
<p>and I have a label where each channel contain information of a particular object, something like this:</p>
<pre><code class="lang-auto">[[      [1, 0, 0, 0],
        [1, 1, 0, 1],
        [0, 0, 0, 0],
        [1, 0, 0, 0]],

       [[0, 1, 0, 0],
        [0, 0, 1, 0],
        [0, 1, 1, 0],
        [0, 0, 0, 1]],

       [[0, 0, 1, 1],
        [0, 0, 0, 0],
        [1, 0, 0, 1],
        [0, 1, 1, 0]]]
</code></pre>
<p>Label:  torch.Size([5, 3, 120, 160])</p>
</blockquote>
</aside>
<p>This looks wrong for use with <code>CrossEntropyLoss</code>.</p>
<p>I’m assuming that you are giving a simplified version of your<br>
label tensor where you are leaving out the <code>nBatch</code> dimension,<br>
<code>nClass = 3</code>, <code>height = 4</code>, and <code>width = 4</code> (for a shape of<br>
<code>[3, 4, 4]</code>).  This looks like a one-hot-encoded version of your<br>
class labels, where the labels are one-hot encoded along the<br>
<code>nClass = 3</code> dimension of your label tensor.</p>
<p>You should instead have a label tensor of integer class labels:</p>
<pre data-code-wrap="python"><code class="lang-python">[       [0, 1, 2, 2],
        [0, 0, 1, 0],
        [2, 1, 1, 2],
        [0, 2, 2, 1]]
</code></pre>
<p>This is tensor of shape <code>[height, width]</code> (again, leaving out<br>
the batch dimension), which, in this simplified example is <code>[4, 4]</code>.<br>
The number of classes, <code>nClass = 3</code> is not part of the shape of the<br>
label tensor, but shows up in the values of the labels – they range<br>
over <code>(0, 1, 2)</code> (that is, <code>[0, nClass - 1]</code> inclusive).</p>
<p>To recap:</p>
<p><code>CrossEntropyLoss</code> takes (for the “multi-dimensional case”) a<br>
prediction of shape <code>[nBatch, nClass, height, width]</code><br>
(what it calls <code>input</code>), whose entries are <em>logits</em> that run from<br>
<code>-inf</code> to <code>+inf</code>.  The label has shape <code>[nBatch, height, width]</code><br>
(what it calls <code>target</code>), and its entries are integer class labels<br>
that range over <code>[0, nClass - 1]</code>, inclusive.</p>
<p>Good luck!</p>
<p>K. Frank</p>

username: abhi8569
cooked: <p>Hi Frank,</p>
<p>Sorry for not being clear with my question and thanks a ton for explaining everything in depth. What you interpreted from my question is exactly what I wanted to convey. And yes, you answered my question. Instead of using one-hot encoding I used integer class labels and it worked. I’ll make sure to be clear with my question from my next post.<br>
Again, Thank you!!</p>