The size of tensor a (21) must match the size of tensor b (3) at non-singleton dimension 1

username: Alef93xx
cooked: <p>Hey guys!</p>
<p>I’m trying to train a neural network for multiclass semantic segmentation (FCN) using the CrossEntropyLoss function, but the following error occurs:</p>
<p>Preds: torch.Size([1, 21, 160, 240])<br>
Target: torch.Size([1, 3, 160, 240])<br>
The size of tensor a (21) must match the size of tensor b (3) at non-singleton dimension 1</p>
<p>As I am using a dataset with 21 classes (RGB 8 bit), my input should be correct. But the dimension of my target is different, this is a little confusing, I don’t know how to solve it.</p>
<p>Should I do some kind of transformation?</p>
<p>Any help or guidance on this will be greatly appreciated!</p>
<p>Part of the code where the error occurs:</p>
<pre><code class="lang-auto">def check_accuracy(loader, model, device="cuda"):
    num_correct = 0
    num_pixels = 0
    dice_score = 0
    model.eval()

    with torch.no_grad():
        for x, y in loader:
            x = x.to(device)
            y = y.to(device).squeeze(1)
            preds = nn.Softmax(dim=1)(model(x))
            preds = (preds &gt; 0.5).float()
            y = y.permute(0, 3, 1, 2) 
            print(f'Preds: {preds.shape}')
            print(f'Target: {y.shape}')
            num_correct += (preds == y).sum()         ####ERROR#####
            num_pixels += torch.numel(preds)
            dice_score += (2 * (preds * y).sum()) / (
                (preds + y).sum() + 1e-8
            )

    print(
        f"{num_correct}/{num_pixels} accuracy {num_correct/num_pixels*100:.2f}"
    )
    print(f"Dice Score : {dice_score/len(loader)}")
    model.train()
</code></pre>

username: KFrank
cooked: <p>Hi Alisson!</p>
<aside class="quote no-group" data-username="Alef93xx" data-post="1" data-topic="126954" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/97f17d/48.png" class="avatar"> Alef93xx:</div>
<blockquote>
<p>Preds: torch.Size([1, 21, 160, 240])<br>
Target: torch.Size([1, 3, 160, 240])<br>
The size of tensor a (21) must match the size of tensor b (3) at non-singleton dimension 1</p>
</blockquote>
</aside>
<p>The short answer is pretty much what the error message says:  You’re<br>
trying to perform an element-wise equality test between two tensors<br>
whose dimensions don’t match.  (If you were comparing two tensors<br>
that had the same shape, you wouldn’t get this error.)</p>
<p>However, I think this is a symptom of some more general structural<br>
things you are are doing wrong …</p>
<aside class="quote no-group">
<blockquote>
<p>I’m trying to train a neural network for multiclass semantic segmentation (FCN) using the CrossEntropyLoss function</p>
</blockquote>
</aside>
<p>You don’t show your training code, and I’m a little surprised that you’re<br>
not reporting similar errors with that, given what it looks like you’re<br>
doing wrong.</p>
<p>Anyway, I would expect that you would be using targets (and<br>
inputs) that have the same format for both training and your<br>
accuracy evaluation.</p>
<p>The input to your model, <code>x</code> – the batch of images you wish to<br>
perform semantic segmentation on – would typically have shape<br>
<code>[nBatch, nChannel, height, width]</code>.</p>
<p>In your case it appears that <code>nBatch = 1</code>.  (A batch size of 1 is fine.)</p>
<p>It seems that <code>nChannel = 3</code> – RGB?  But you say that you are using<br>
“RGB 8 bit” which would mean that you would have a single 8-bit<br>
channel with <em>palettized</em> color – that is, you have a look-up table<br>
(<em>LUT</em>) that translates the 8-bit value of each pixel into an RGB color<br>
by using it as an index into the LUT.  So it’s unclear what is going on<br>
with your channel dimension.</p>
<p><code>height = 160</code> and <code>width = 240</code>, which is fine.</p>
<p>This line:</p>
<aside class="quote no-group">
<blockquote>
<pre data-code-wrap="python"><code class="lang-python">y = y.permute(0, 3, 1, 2)
</code></pre>
</blockquote>
</aside>
<p>suggests that your images – or at least your target “images” –  are<br>
stored (individually) with shape <code>[height, width, nChannel]</code>, so<br>
you permute the dimensions to put them in the order preferred by<br>
pytorch.  This is also fine.  (Why you have the <code>.squeeze(1)</code>, I don’t<br>
know.)</p>
<p>The core problem – and this is hard to know for sure because you<br>
don’t show the code or shapes that you use for training – is that<br>
your <code>target</code> (your <code>y</code>) is not formatted correctly for <code>CrossEntropyLoss</code>.</p>
<p>The output of your model (your <code>preds</code>) has shape<br>
<code>[nBatch = 1, nClass = 21, height = 160, width = 240]</code>.<br>
This is fine.  But you would then want the <code>target</code> (the annotated<br>
class labels for the image) to have shape <code>[nBatch, height, width]</code>,<br>
with <em>no</em> <code>nClass</code> dimension.  Each “pixel” in your <code>target</code> “image”<br>
should be an integer class label that runs from <code>0</code> to<br>
<code>nClass - 1 = 20</code>.</p>
<p>It is sometimes the case that <code>target</code> images have the class labels<br>
color coded – that is, all of the class-3 pixels might be colored blue,<br>
all of the class-7 pixels might be colored gray, and so on.  If this is the<br>
case, you would have to pre-process your <code>target</code> images so that<br>
they use integer class labels, as described above.</p>
<p>Now, on to your accuracy calculation:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            preds = nn.Softmax(dim=1)(model(x))
</code></pre>
</blockquote>
</aside>
<p>This is okay, but you don’t actually need the <code>Softmax</code> here.<br>
<code>preds = model (x)</code> is simpler, and – for this purpose – equivalent.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            preds = (preds &gt; 0.5).float()
</code></pre>
</blockquote>
</aside>
<p>This is incorrect (for the multi-class problem you say you have).<br>
(This would make sense for <em>binary</em> segmentation.)</p>
<p>You want to convert the “soft,” probabilistic prediction made by<br>
your model into a “hard” prediction for the single best class.</p>
<p>For this you want <code>preds = preds.argmax (dim = 1)</code>.  This<br>
produces a batch of prediction “images” of shape<br>
<code>[nBatch, height, width]</code> (no <code>nClass</code> dimension) where each<br>
pixel is the integer class label for the class the model predicts<br>
as being the most probable.</p>
<p>If your <code>target</code> (<code>y</code>) is also composed of integer class labels and has<br>
shape <code>[nBatch, height, width]</code>, <code>(preds == y).sum()</code> will<br>
count the number of correct predictions.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Alef93xx
cooked: <p>Hi K. Frank, thanks so much for your notes.</p>
<p>Sorry, I forgot to enter the training code, it’s below:</p>
<pre><code class="lang-auto">import torch
import albumentations as A
from albumentations.pytorch import ToTensorV2
from tqdm import tqdm
import torch.nn as nn
import torch.optim as optim
from model import UNet
from utils import (
    load_checkpoint,
    save_checkpoint,
    get_loaders,
    check_accuracy,
    save_predictions_as_imgs,
)
print(f'PyTorch: {torch.__version__}')
print(f'CUDA: {torch.version.cuda}')

if torch.cuda.is_available() == True:
   DEVICE = "cuda"
   print('GPU')
   #torch.manual_seed(12)
   #torch.cuda.manual_seed(12)
   #np.random.seed(12)
else: 
   DEVICE = "cpu"
   print('CPU')

if(torch.backends.cudnn.is_available() == True):
    torch.backends.cudnn.enabled = True
    torch.backends.cudnn.benchmark = True
    print('cuDNN activated')
else:
    print('cuDNN not available!')

# Hyperparameters etc.
LEARNING_RATE = 3e-4
BATCH_SIZE = 1
NUM_EPOCHS = 10
NUM_WORKERS = 0
IMAGE_HEIGHT = 160
IMAGE_WIDTH = 240
PIN_MEMORY = True
LOAD_MODEL = False
TRAIN_IMG_DIR = "DATASET/train/imagens/"
TRAIN_MASK_DIR = "DATASET/train/mascaras/"
VAL_IMG_DIR = "DATASET/val/imagens/"
VAL_MASK_DIR = "DATASET/val/mascaras/"

#torch.cuda.empty_cache()

def train_fn(loader, model, optimizer, loss_fn, scaler):
    loop = tqdm(loader)

    for batch_idx, (data, targets) in enumerate(loop):
        data = data.to(device=DEVICE)
        targets = targets.long().squeeze(1).to(device=DEVICE)

        # forward
        with torch.cuda.amp.autocast():
            predictions = model(data)
            loss = loss_fn(predictions, targets)
            
        # backward
        optimizer.zero_grad()
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

        # update tqdm loop
        loop.set_postfix(loss=loss.item())


def main():
    train_transform = A.Compose(
        [
            A.Resize(height=IMAGE_HEIGHT, width=IMAGE_WIDTH),
            A.Rotate(limit=35, p=1.0),
            A.HorizontalFlip(p=0.5),
            A.VerticalFlip(p=0.1),
            A.Normalize(
                mean=[0.0, 0.0, 0.0],
                std=[1.0, 1.0, 1.0],
                max_pixel_value=255.0,
            ),
            ToTensorV2(),
        ],
    )

    val_transforms = A.Compose(
        [
            A.Resize(height=IMAGE_HEIGHT, width=IMAGE_WIDTH),
            A.Normalize(
                mean=[0.0, 0.0, 0.0],
                std=[1.0, 1.0, 1.0],
                max_pixel_value=255.0,
            ),
            ToTensorV2(),
        ],
    )

    
    model = UNet(21).to(DEVICE)
    loss_fn = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)

    train_loader, val_loader = get_loaders(
        TRAIN_IMG_DIR,
        TRAIN_MASK_DIR,
        VAL_IMG_DIR,
        VAL_MASK_DIR,
        BATCH_SIZE,
        train_transform,
        val_transforms,
        NUM_WORKERS,
        PIN_MEMORY,
    )

    if LOAD_MODEL:
        load_checkpoint(torch.load("my_checkpoint.pth.tar"), model)


    check_accuracy(val_loader, model, device=DEVICE)
    scaler = torch.cuda.amp.GradScaler()

    for epoch in range(NUM_EPOCHS):
        train_fn(train_loader, model, optimizer, loss_fn, scaler)

        # save model
        checkpoint = {
            "state_dict": model.state_dict(),
            "optimizer":optimizer.state_dict(),
        }
        save_checkpoint(checkpoint)

        # check accuracy
        check_accuracy(val_loader, model, device=DEVICE)

        # print some examples to a folder
        save_predictions_as_imgs(
            val_loader, model, folder="Results\\", device=DEVICE
        )


if __name__ == "__main__":
    main()

</code></pre>
<p>My code may seem meaningless at times, sorry about that, but I’m a beginner. I will try to explain some points better.</p>
<ol>
<li>
<p>I was also surprised that the training code didn’t report a bug. But if I apply argmax function to predictions and target to remove dimension 1, the error in check_accuracy function disappears. But then it causes a similar error in the training code.</p>
</li>
<li>
<p>I forgot to comment, but I convert the images to RGB, because if I don’t, the colors of the masks are changed. When this happens, some different classes are given the same color. I don’t know why this happens. I’ll leave the dataset code here:</p>
</li>
</ol>
<pre><code class="lang-auto">import os
from PIL import Image
from torch.utils.data import Dataset
import numpy as np
import time
import matplotlib
import matplotlib.pyplot as plt
import cv2
import torch

class MeuDataset(Dataset):
    def __init__(self, image_dir, mask_dir, transform=None):
        self.image_dir = image_dir
        self.mask_dir = mask_dir
        self.transform = transform
        self.images = os.listdir(image_dir)

    def __len__(self):
        return len(self.images)

    def __getitem__(self, index):
        img_path = os.path.join(self.image_dir, self.images[index])
        mask_path = os.path.join(self.mask_dir, self.images[index].replace(".jpg", ".png"))
        image = np.array(Image.open(img_path).convert("RGB"))
        mask = np.array(Image.open(mask_path).convert("RGB"), dtype=np.uint8)
        #mask = np.transpose(mask,(2, 0, 1))        
        #debug
        #print(f'Normal: {mask.shape}')
        #plt.imshow(mask)
        #plt.show()
        #mask = mask/classes
        #mask[mask == 255.0] = 1.0

        if self.transform is not None:
            augmentations = self.transform(image=image, mask=mask)
            image = augmentations["image"]
            mask = augmentations["mask"]

        return image, mask
</code></pre>
<ol start="3">
<li>Target formatting is really my biggest issue here, as you said. I’ve seen in other threads that we must convert RGB colors to class indices, but I have no idea how to do that. I’ve even tried it, but nothing has worked yet.</li>
</ol>
<p>After changing the accuracy calculation as you suggested, I got the following error:</p>
<p>1only batches of spatial targets supported (3D tensors) but got targets of size: : [1, 160, 240, 3].</p>
<p>In this case, swapping the dimensions as required by PyTorch and removing the dimension corresponding to 3 would be sufficient?</p>

username: KFrank
cooked: <p>Hi Alisson!</p>
<aside class="quote no-group" data-username="Alef93xx" data-post="3" data-topic="126954" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/97f17d/48.png" class="avatar"> Alef93xx:</div>
<blockquote>
<ol start="3">
<li>Target formatting is really my biggest issue here, as you said. I’ve seen in other threads that we must convert RGB colors to class indices, but I have no idea how to do that. I’ve even tried it, but nothing has worked yet.</li>
</ol>
</blockquote>
</aside>
<p>Let me make a number of comments before outlining how one<br>
might convert RGB colors to integer class labels,</p>
<p>First, just to use a clear-cut word, let me call your ground-truth,<br>
annotated (labeled) images “masks.”  So your dataset consists<br>
of a bunch of images that will be input to your model (for both<br>
training and validation) each with a corresponding mask that is<br>
used either to train your model or evaluate how well your trained<br>
model is performing its segmentation task.</p>
<p>First, are your training and validation datasets in the same format?<br>
Do your training and validation datasets come from one larger<br>
combined dataset that was split (perhaps randomly) into the two<br>
separate training and validation datasets?</p>
<p>What do your input images look like?  Are they color images?<br>
Grayscale?  Do they look like sensible images that appear to the<br>
eye to have structure that could be segmented?</p>
<p>What do your masks look like?  Do they have blobs of different<br>
colors in them that line up sensibly with the segments in the<br>
corresponding input images?</p>
<p>Before processing, it sounds like your masks  are some sort of color<br>
images.  What format are they in?  Do they have three separate RGB<br>
channels?  Are they 32-bit RGB images (that might only be using<br>
24 bits under the hood)?  Are they 8-bit palettized images?</p>
<p>What format are your masks in after any processing that may occur<br>
in your dataloader?</p>
<p>I would suggest that you print out the shapes of your input-image<br>
and mask batches both immediately after they are delivered by<br>
the dataloader, and then again immediately before they are fed<br>
to your model and loss function.  (The reason to print out the<br>
shapes at these two different locations in your code is to verify<br>
that your <code>squeeze()</code>s and <code>permute()</code>s are doing what you want.)</p>
<p>In the typical case (It doesn’t <em>have</em> to be this way, but it would be<br>
typical.), a batch of input images would have shape<br>
<code>[nBatch, nChannel, height, width]</code>.  <code>nChannel</code> would typically<br>
<code>1</code> for grayscale images and <code>3</code> for color images.  (<code>nChannel</code> could be<br>
<code>3</code> for grayscale images that happen to be formatted as color images.)</p>
<p>Note, if you are passing color images to your model, it will only work<br>
if your input images (by the time they get passed to your model) are<br>
multi-channel (almost certainly three-channel RGB) images.  Your<br>
model won’t work with 32-bit color nor 8-bit palettized color images.</p>
<p>The output of your model will typically have shape<br>
<code>[nBatch, nClass, height, width]</code>.  You should also print out<br>
the shape of your model output images to verify this.</p>
<p>When used for training with <code>CrossEntropyLoss</code> your masks must<br>
have shape <code>[nBatch, height, width]</code> (with no <code>nClass</code> nor <code>nChannel</code><br>
dimension), and the values of the mask pixels must be integer class<br>
labels that run over <code>0</code> to <code>nClass - 1</code>.  This is what <code>CrossEntropyLoss</code><br>
expects.</p>
<p>(It would be possible to have masks of a different shape or format<br>
for your accuracy calculation – you don’t <em>have</em> to follow the<br>
requirements of <code>CrossEntropyLoss</code>, but there is likely no benefit<br>
to using a different format, and doing so would only confuse the issue.)</p>
<p>I would recommend making a simplified “test-bed” version of your<br>
code – get rid of as much extraneous stuff as you can.  Get rid of<br>
the checkpoint stuff, cuda, amp, scaler, etc.  I would keep the transforms<br>
as they could modify the shape / format of the images and masks<br>
that are being passed to your model, loss function, and accuracy<br>
calculation, and that’s part of what you need to check.</p>
<p>Then try passing a single batch of <code>train_loader</code> images through<br>
your model and loss function (printing out the shapes, as discussed<br>
above) and try passing a single <code>val_loader</code> batch of images through<br>
your model and accuracy calculation (again printing out shapes).</p>
<p>Make sure that everything lines up the way it should, and if it doesn’t<br>
make sure you understand the <em>details</em> of what isn’t right, and what<br>
you need to do, conceptually, to fix it.</p>
<p>Now to the question creating masks that consist of integer class<br>
labels.</p>
<p>I will assume that your masks start out being some sort of color<br>
images where each of a discrete set of colors (in your case,<br>
presumably <code>nClass = 21</code> different colors) indicates a specific<br>
class.</p>
<p>Converting your masks to consist of integer class labels could be<br>
done in your dataloader or in a separate processing step after<br>
your dataloader.  I would probably put this processing step in the<br>
dataloader, but it doesn’t really matter.</p>
<p>You need to write the code that reads through all (or at least a<br>
representative sample) of your masks and makes a table of all<br>
the colors that show up in the pixels.  (Conceptually, it doesn’t<br>
matter whether the colors are encoded with three RGB channels<br>
or as 32-bit or 8-bit colors – you just need to keep track of the<br>
distinct colors that show up.)  Each distinct color corresponds to<br>
a class.  Because you say that you have 21 classes, you had better<br>
have no more that 21 distinct colors.  For example, if you discover<br>
that your training masks have 25 distinct colors, then your masks<br>
are telling you that you have 25 classes, not 21, so you would have<br>
to sort that out.</p>
<p>Now make a dictionary (consider it to be a look-up table) whose<br>
keys are the 21 distinct colors and values are the integers 0<br>
through 20.  The assignment of integer class label to mask color<br>
can be arbitrary.  If your classes have some “meaning,” you are<br>
welcome to assign integer class labels that somehow reflect that<br>
meaning for your own convenience, but the choice won’t affect<br>
how the model training progresses.  (The class-label assignment<br>
does have to be consistent across your dataset – both your training<br>
and validation data jointly.)</p>
<p>(As an aside, for this kind of semantic segmentation, we often think<br>
of an image being segmented as having 'foreground" objects that<br>
form the classes and an “uninteresting” background.  For training,<br>
however, the background pixels still form a class, and are treated<br>
no differently than the “foreground” pixels.  For your convenience<br>
(It doesn’t affect the training.), you might choose to use 0 as the<br>
integer class label for the background pixels.)</p>
<p>Now for each mask read through the pixels in the mask tensor and<br>
create a new mask tensor of <code>dtype = torch.int64</code> (the same as<br>
<code>torch.long</code>) and typically of shape <code>[height, width]</code>.  As you write<br>
your new mask tensor on a pixel-by-pixel basis, translate the<br>
corresponding color-valued pixel from your original mask into the<br>
proper integer class label by passing it through the look-up table<br>
you built.</p>
<p>Good luck.</p>
<p>K. Frank</p>