RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4

copperwiring: <p><strong>Issue</strong>: RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4</p>
<p><strong>Problem Statement</strong>: I have an image and a pixel of the image can belong to only(either) one of <code>Band5','Band6', 'Band7'</code> (see below for details). Hence, I have a pytorch multi-class problem but I am unable to understand how to set the targets which needs to be in form <code>[batch, w, h]</code></p>
<p>My dataloader return two values:</p>
<pre><code>x = chips.loc[:, :, :, self.input_bands]     
y = chips.loc[:, :, :, self.output_bands]        
x = x.transpose('chip','channel','x','y')
y_ohe = y.transpose('chip','channel','x','y')
return x, y_ohe
</code></pre>
<p>Also, I have defined:</p>
<pre><code>input_bands = ['Band1','Band2', 'Band3', 'Band3', 'Band4']  # input classes
output_bands = ['Band5','Band6', 'Band7'] #target classes

model = ModelName(num_classes = 3, depth=default_depth, in_channels=5, merge_mode='concat').to(device)
loss_new = nn.CrossEntropyLoss()
</code></pre>
<p>In my training function:</p>
<pre><code>        #get values from dataloader
        X = normalize_zero_to_one(X) #input
        y = normalize_zero_to_one(y) #target

        images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
        masks = Variable(torch.from_numpy(y)).to(device) 
        optim.zero_grad()        
        outputs = model(images) 

        loss = loss_new(outputs, masks) # (preds, target)
        loss.backward()         
        optim.step() # Update weights  
</code></pre>
<p>I know the the target (here <code>masks</code>) should be <code>[batch_size, w, h]</code>. However, it is currently <code>[batch_size, channels, w, h]</code>.</p>
<p>I read a lot of posts including <a href="https://discuss.pytorch.org/t/bceloss-for-multiclass-problem/36675">1</a>, <a href="https://discuss.pytorch.org/t/for-crossentropy-loss-function-why-output-and-target-size-cannot-be-the-same/46026">2</a> and they say <code>the target should only contain the target class indices</code>. I don’t understand how can I concatenate indices of three classes and still set target as <code>[batch_size, w, h]</code>.</p>
<p>Right now, I get the error:</p>
<pre><code>RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4
</code></pre>
<p>To the best of my understanding, I don’t need to do any one hot encoding. Similar errors and explanation I found on the internet are here:’</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/only-batches-of-spatial-targets-supported-non-empty-3d-tensors-but-got-targets-of-size-1-1-256-256/49134">Reference 1</a></li>
<li><a href="https://discuss.pytorch.org/t/unable-to-understand-the-runtime-error/19906">Reference 2</a></li>
<li><a href="https://discuss.pytorch.org/t/nll-loss-error-only-batches-of-spatial-targets-supported-but-got-targets-of-dimension-1/57047">Reference 3</a></li>
<li><a href="https://stackoverflow.com/questions/50896412/channel-wise-crossentropyloss-for-image-segmentation-in-pytorch" rel="nofollow noopener">Reference 4</a></li>
</ul>
<p>Any help will be appreciated! Thank you.</p>

copperwiring: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I have seen you have given most responses to similar questions in the past. I have even linked them in the references. You mention that how the <code>target</code> shape should look like but I didn’t get how to achieve that change in shape.</p>
<p>I’d really appreciate if you can help.</p>
<p>Thank you!</p>

ptrblck: <p>Assuming your current target is one-hot encoded in the channel dimension, i.e. it uses a 1 for the “active” class in that channel while all other channels contain zeros, you could use:</p>
<pre><code class="lang-python">target = torch.argmax(target, dim=1)
</code></pre>
<p>to create the target with the expected class indices.</p>
<p>If that doesn’t work for you, could you post an example target with its values?</p>
<p>Also, don’t use <code>Variables</code> anymore, as they are deprecated since PyTorch <code>0.4</code>.</p>

copperwiring: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thank you ! Two questions:</p>
<ol>
<li>In one of your past posts <a href="https://discuss.pytorch.org/t/multi-class-classifier/29901">here</a> :</li>
</ol>
<p><code>No, for multi-class classification (one target class for each sample), the targets should hold the class indices. Other frameworks often use one-hot encoded target vectors, which is not necessary in PyTorch. Have a look at the docs for more information.</code></p>
<p>I was wondering if the vectors need to be one-hot encoded or not?</p>
<ol start="2">
<li>When you say <code>Assuming your current target is one-hot encoded in the channel dimension, i.e. it uses a 1 for the “active” class in that channel while all other channels contain zeros</code>, how can I encode the data for three different classes.</li>
</ol>
<p>Currently my target (<code>masks</code> in the code snippet in question) looks like this:</p>
<blockquote>
<p><span class="hashtag-raw">#print</span>(masks)<br>
tensor([[[[0., 0., 0.,  …, 0., 0., 0.],<br>
[0., 0., 0.,  …, 0., 0., 0.],<br>
[0., 0., 0.,  …, 0., 0., 1.],<br>
…,<br>
[1., 1., 0.,  …, 0., 0., 1.],<br>
[1., 0., 1.,  …, 0., 0., 0.],<br>
[1., 1., 0.,  …, 0., 0., 0.]],<br>
[[1., 1., 1.,  …, 0., 0., 0.],<br>
[1., 1., 1.,  …, 0., 0., 0.],<br>
[1., 1., 1.,  …, 0., 0., 0.],<br>
…,<br>
[0., 0., 0.,  …, 1., 1., 0.],<br>
[0., 0., 0.,  …, 1., 1., 1.],<br>
[0., 0., 1.,  …, 1., 1., 1.]]]])</p>
</blockquote>
<p>Also, the shape of <code>X</code> , <code>y</code> and 'masks` respectively are as below:</p>
<blockquote>
<p>(1, 5, 256, 256)<br>
(1, 3, 256, 256)<br>
torch.Size([1, 3, 256, 256])</p>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="copperwiring" data-post="4" data-topic="82098">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/48.png" class="avatar"> copperwiring:</div>
<blockquote>
<p>I was wondering if the vectors need to be one-hot encoded or not?</p>
</blockquote>
</aside>
<p>For a multi-class classification or segmentation, the target should not be one-hot encoded.<br>
Based on your <code>masks</code> output, it seems that you might be dealing with a multi-label segmentation, where each pixel might belong to zero, one, or multiple classes.<br>
Some pixels have no active class (e.g. row0, last column). Is that your use case?</p>

copperwiring: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="82098">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Based on your <code>masks</code> output, it seems that you might be dealing with a multi-label segmentation, where each pixel might belong to zero, one, or multiple classes.</p>
</blockquote>
</aside>
<p>Sorry if my code may be wrong somwhere, but in my case, each pixel can only belong to one class i.e. to either of one bands of the <code>output</code> bands (shown in code snippet).</p>
<p>I also edited my previous code to show more data with shapes.</p>

ptrblck: <p>Ah OK, in that case each pixel should have one valid <code>1</code> value in the channel dimension and you can use:</p>
<pre><code class="lang-python">target = torch.argmax(target, dim=1)
</code></pre>
<p>to create the valid target tensor for <code>nn.CrossEntropyLoss</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

copperwiring: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="82098">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>target = torch.argmax(target, dim=1)</code></p>
</blockquote>
</aside>
<p>Ok! But currently my <code>target</code> shape is <code>(1, 3, 256, 256)</code>, isn’t it expecting something like <code>(1, 1, 256, 256)</code> for me to apply <code>target = torch.argmax(target, dim=1)</code>?</p>
<p>I’ll try the code snippet in the meanwhile.</p>

copperwiring: <p>May be I just don’t understand how `</p>
<pre><code>target = torch.argmax(target, dim=1)
</code></pre>
<p>`</p>
<p>encoded the data for all the 3 output channels!</p>

copperwiring: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for the help! I see the loop works for the loss but how would we get the prediction values for each class from it?</p>
<p>Edit 1:</p>
<p>My current test code set up looks like this but how to get the right prediction:</p>
<blockquote>
<pre><code>        X = normalize_zero_to_one(X)
        y = normalize_zero_to_one(y)
        
        images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
        masks = Variable(torch.from_numpy(y)).to(device)
        
        masks = torch.argmax(masks, dim=1)
        
        outputs = model(images)            
        loss = loss_new(outputs, masks)          
        output_sigmoid = torch.sigmoid(outputs)
        preds = output_sigmoid &gt; output_sigmoid.cpu().numpy().mean()
</code></pre>
</blockquote>
<p>I don’t think I’ll need sigmoid here, isn’t it?</p>

ptrblck: <p>Assuming that you are working with 3 classes (based on the last edit of the post with the shape information), your model output should have the shape <code>[batch_size, nb_classes=3, height, width]</code>.<br>
To get the predicted class indices, you can use the same method:</p>
<pre><code class="lang-python">preds = torch.argmax(output, dim=1)
</code></pre>

copperwiring: <aside class="quote no-group" data-username="ptrblck" data-post="11" data-topic="82098">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>preds = torch.argmax(output, dim=1)</code></p>
</blockquote>
</aside>
<p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Yes, I do have the shape <code>shape [batch_size, nb_classes=3, height, width].</code>which seems I am in the right direction.</p>
<p><code>preds = torch.argmax(output, dim=1)</code> will give me the prediction values (or probabilties). Will I need to use a softmax  or a sigmoid for thresholding?</p>

ptrblck: <aside class="quote no-group" data-username="copperwiring" data-post="12" data-topic="82098">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/ad7895/48.png" class="avatar"> copperwiring:</div>
<blockquote>
<p><code>preds = torch.argmax(output, dim=1)</code> will give me the prediction values (or probabilties). Will I need to use a softmax or a sigmoid for thresholding?</p>
</blockquote>
</aside>
<p>This will give you the predicted class indices, no probabilities.<br>
<code>output</code> should contain the raw logits. If you need to see probabilities, you should apply <code>softmax</code> on it.<br>
However, you don’t need to use the <code>softmax</code> to get the most likely predicted class, as <code>argmax(logits, 1)</code> will yield the same result as <code>argmax(softmax(logtis, 1) 1)</code>.</p>
<p>A threshold is not necessary for a multi-class prediction.</p>

copperwiring: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. One more question (sorry for troubling. I am working with multi-class problem for the first time)</p>
<p>I am looking at the output:</p>
<blockquote>
<pre><code>        X = normalize_zero_to_one(X)
        y = normalize_zero_to_one(y)
        
        images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
        masks = Variable(torch.from_numpy(y)).to(device)
        
        masks = torch.argmax(masks, dim=1)
        outputs = model(images)            
        loss = loss_new(outputs, masks)          
        
        preds = torch.argmax(outputs, dim=1)
        
        print(preds.shape)
        print(preds)
</code></pre>
</blockquote>
<blockquote>
<p>torch.Size([1, 256, 256])<br>
tensor([[[1, 1, 1,  …, 1, 1, 1],<br>
[1, 1, 1,  …, 1, 1, 1],<br>
[1, 1, 1,  …, 1, 1, 0],<br>
…,<br>
[1, 1, 1,  …, 1, 1, 1],<br>
[1, 1, 1,  …, 1, 1, 0],<br>
[1, 0, 0,  …, 1, 1, 0]]])</p>
</blockquote>
<p>Can you help me understand how to read this output? When I see the first value as 1, how do I read which output band is it talking about</p>
<p><strong>P.S.</strong> Note, for certain simplicity, I changed my output classes to 2 classes ( <code> Band5','Band6</code>) and got output shape <code>[batch_size, nb_classes=2, height, width]</code>. Print result in the answer reflects the same.</p>

ptrblck: <p>The <code>preds</code> tensor represents the predicted class index for each pixel.<br>
I.e. you could interpret the tensor as:</p>
<pre><code class="lang-python">tensor([[[class1, class1, class1, …, class1, class1, class1],
          ...
         [class1, class1, class1, …, class1, class1, class0],
          ...,
</code></pre>
<p>For the original problem with 3 classes, you should see the values <code>[0, 1, 2]</code> in the <code>pred</code> tensor.</p>

copperwiring: <p>Ah! Interesting. So, if I want to extract the pixel ‘values’ of a certain class where it is predicted, say <code>class 1</code>, I should extract the idices where I see <code>class 1</code> as 1 and I can use them to plot the values.</p>
<p>I hope I got it right this time!</p>

ptrblck: <p>Yes, you could get the pixel locations via:</p>
<pre><code class="lang-python">class0_loc = (preds==0).nonzero()
class1_loc = (preds==1).nonzero()
</code></pre>
<p>You can also plot the prediction directly e.g. using <code>matplotlib</code> via:</p>
<pre><code class="lang-python">plt.imshow(preds[0])
</code></pre>

copperwiring: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> That was really helpful!</p>

copperwiring: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your help in working with Cross Entropy Loss. However, the predictions (for three classes) seem to be too off.</p>
<p>When I use the code below:</p>
<pre><code class="lang-auto">            print("shape of model input images:", images.shape)
            print("shape of masks_input:", masks_input.shape)
            print('masks input:')
            print(masks_input)
            outputs = model(images)  #model doesn't have softmax and images are normalized
            masks = torch.argmax(masks_input, dim=1)
            loss = loss_new(outputs, masks) 
            preds = torch.argmax(outputs, dim=1)
            print("shape of preds:", preds.shape)
            print('preds output')
            print(preds)
</code></pre>
<p>I get the following pred output. Basically, 95% of it is being predicted of Class 1 -</p>
<pre><code class="lang-auto">shape of model input images: torch.Size([1, 5, 256, 256])
shape of masks_input: torch.Size([1, 3, 256, 256])
masks input:
tensor([[[[0., 0., 0.,  ..., 1., 1., 1.],
          [0., 0., 0.,  ..., 1., 1., 1.],
          [0., 0., 0.,  ..., 1., 1., 0.],
          ...,
          [0., 0., 1.,  ..., 0., 0., 0.],
          [0., 1., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]],

         [[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 1.],
          ...,
          [1., 1., 0.,  ..., 0., 0., 1.],
          [1., 0., 1.,  ..., 0., 0., 0.],
          [1., 1., 0.,  ..., 0., 0., 0.]],

         [[1., 1., 1.,  ..., 0., 0., 0.],
          [1., 1., 1.,  ..., 0., 0., 0.],
          [1., 1., 1.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 1., 1., 0.],
          [0., 0., 0.,  ..., 1., 1., 1.],
          [0., 0., 1.,  ..., 1., 1., 1.]]]])
shape of preds: torch.Size([1, 256, 256])
preds output
tensor([[[0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 2],
         [0, 0, 0,  ..., 0, 0, 0],
         ...,
         [0, 0, 0,  ..., 0, 0, 0],
         [0, 0, 0,  ..., 0, 0, 2],
         [0, 0, 0,  ..., 0, 0, 0]]])
</code></pre>

ptrblck: <p>Your model might be overfitting to the majority class.<br>
Could you check the class distribution in your masks, i.e. how many pixels belong to which class?<br>
If you have a lot of background (class0), your model might only lean this class and you could use e.g. focal loss to counter it.</p>