Question about output and label channels in semantic segmentation

mshmoon: <p>I use some nets,FCN8 ,SegNet for semantic segmentation .The trouble follow:<br>
all of the nets I used,The last layers of this net output the  feature maps is (1,22,256,256),why not (1,3,256,256)? and another question is that the labels size is (1,1,256,256),why not(1,3,256,256)?<br>
please help me, I a new gay</p>

ptrblck: <p>The output and label shape depend on your classification problem. If the last layer outputs an activation of <code>[batch_size, 22, width, height]</code>, it probably means that the model was used to classify each pixel into one of 22 classes.<br>
You should change the number of output channels to your use case.</p>
<p>Edit: I edited your title so that the question becomes clearer.</p>

zeakey: <p>22 = 21 (classes) + 1 (background) in PASCAL_VOC dataset.</p>

mshmoon: <p>Thank your for you help . I know it by debug</p>

mshmoon: <p>Thank you for you help.Thank you very much .I know it</p>

mshmoon: <p>I am sorry,I want to ask for you another question.The label size is (batch_size, 1, width, height),that means the lable have one channel,so it is a picture?<br>
And,the output size is (batch_size, class_num, width, height),then the cost function to compute loss by the every channel of the  output and the only channel of the label?</p>

ptrblck: <p>Yes, the label seems to be a picture. You could try to visualize it using something like:</p>
<pre><code class="lang-auto">import matplotlib.pyplot as plt
plt.imshow(label[0, 0, ...].data.numpy())
</code></pre>
<p>When you are dealing with a multi-class problem, you could use <code>NLLLoss</code> or <code>CrossEntropyLoss</code>. Have a look at the <a href="http://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss">doc</a>.</p>
<p>Here is a small example with 22 classes and a binary example:</p>
<pre><code class="lang-auto"># Multi-class example
x = Variable(torch.randn(1, 22, 10, 10))
y = Variable(torch.LongTensor(1, 10, 10).random_(22))

criterion = nn.NLLLoss()

loss = criterion(x, y)

# Binary example
x = F.sigmoid(Variable(torch.randn(1, 1, 10, 10)))
y = Variable(torch.FloatTensor(1, 1, 10, 10).random_(2))

criterion = nn.BCELoss()

loss = criterion(x, y)
</code></pre>

mshmoon: <p>Thank you for you help ,good friend</p>

Vishrut10: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="15828">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>doc</p>
</blockquote>
</aside>
<p>I am facing similar problems. What should be input dimension in case of multi-class labels. I have image with 3 channels and labels with 1 channel and 12 classes.<br>
I get output prediction with 12 channels. How do I tackle this ?<br>
Thanks in advance!</p>

ptrblck: <p>If you are dealing with a multi-class classification, you could use <code>nn.CrossEntropyLoss</code> with a model output of <code>[batch_size, nb_classes, h, w]</code> and a target of <code>[batch_size, h, w]</code> containing the class indices.<br>
How is your current target image defined?</p>

Vishrut10: <p>Thank you for your reply.<br>
Sorry for incomplete information</p>
<p>This is a semantic segmentation problem</p>
<p>Input data: Image is in .jpg format. I use transform.ToTensor() to convert this.<br>
Target: Annotated .png image with 12 classes. I convert it using “torch.from_numpy(np.array(mask)).long()”</p>
<p>Following is the code I am working on.</p>
<h1>data loader class</h1>
<p>class GetDataset(torch.utils.data.Dataset):<br>
def <strong>init</strong>(self, root, transformation=None):<br>
self.root = root<br>
self.transformation = transformation<br>
# load all image files, sorting them to<br>
# ensure that they are aligned<br>
self.imgs = list(sorted(os.listdir(os.path.join(root, “JPEGImages_test”))))<br>
self.masks = list(sorted(os.listdir(os.path.join(root, “SegmentationClass_test”))))</p>
<pre><code>def __getitem__(self, idx):
    # load images ad masks
    img_path = os.path.join(self.root, "JPEGImages_test", self.imgs[idx])
    mask_path = os.path.join(self.root, "SegmentationClass_test", self.masks[idx])
    print(idx)
    #import pdb
    #pdb.set_trace()
    img = Image.open(img_path)
    
    mask = Image.open(mask_path)
    
    img_t = self.transformation(img)
    mask_t = torch.from_numpy(np.array(mask)).long()
    return img_t, mask_t 

def __len__(self):
    return len(self.imgs)  
</code></pre>
<h1>==&gt; Tried both losses</h1>
<p><span class="hashtag">#criterion</span> = nn.CrossEntropyLoss()<br>
criterion = nn.NLLLoss()</p>
<h1>Train function</h1>
<p>def train_model(model, criterion, optimizer, dataloaders, scheduler, num_epochs=1):<br>
import copy</p>
<pre><code>best_model_wts = copy.deepcopy(model.state_dict())
print(num_epochs)
for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)
    model.train()  # Set model to training mode

    for inputs, labels in dataloaders:
        print("reached")
        
        inputs = inputs.to(device)
        type(input)
        labels = labels.to(device)
        optimizer.zero_grad()
        outputs = model(inputs)
        #import pdb
        #pdb.set_trace()
        loss = criterion(outputs['out'], labels)
        loss.backward()
        optimizer.step()

    best_model_wts = copy.deepcopy(model.state_dict())

# load best model weights
model.load_state_dict(best_model_wts)
return model
</code></pre>
<p><span class="hashtag">#Train</span> model<br>
model_ft = train_model(model_segdl, criterion, optimizer_ft,data_loader, exp_lr_scheduler, num_epochs=1)</p>
<p>model_ft.eval()<br>
x = torch.rand(1, 3, 300, 400)<br>
predictions = model_ft(x)</p>
<p>predictions[‘out’].size()<br>
Out[59]: torch.Size([1, 12, 300, 400])</p>
<p>Is this correct ? If not what I am missing</p>

ptrblck: <p>Could you print the shape of <code>mask_t</code>?<br>
How are your masks stored? Is each class corresponding to a certain color in an RGB image or do the image files already contain class indices?</p>

Vishrut10: <p>Yes I have indices for each class. mask_t are essentially labels, when I load them from dataloader, the shape is as follows.<br>
Although I used label inputs with dimension of [1, 512, 512](which is my .png file) and converted it to long using torch.from_numpy(np.array(mask)).long()</p>
<p>ipdb&gt; labels.size()<br>
torch.Size([3, 512, 512])</p>
<p>ipdb&gt; labels<br>
tensor([[[2, 2, 2,  …, 2, 2, 2],<br>
[2, 2, 2,  …, 2, 2, 2],<br>
[2, 2, 2,  …, 2, 2, 2],<br>
…,<br>
[2, 2, 2,  …, 2, 2, 2],<br>
[2, 2, 2,  …, 2, 2, 2],<br>
[2, 2, 2,  …, 2, 2, 2]],</p>
<pre><code>    [[2, 2, 2,  ..., 2, 2, 2],
     [2, 2, 2,  ..., 2, 2, 2],
     [2, 2, 2,  ..., 2, 2, 2],
     ...,
     [9, 9, 9,  ..., 2, 2, 2],
     [9, 9, 9,  ..., 2, 2, 2],
     [9, 9, 9,  ..., 2, 2, 2]],

    [[1, 1, 1,  ..., 2, 2, 2],
     [1, 1, 1,  ..., 2, 2, 2],
     [1, 1, 1,  ..., 2, 2, 2],
     ...,
     [1, 1, 1,  ..., 2, 2, 2],
     [1, 1, 1,  ..., 2, 2, 2],
     [1, 1, 1,  ..., 2, 2, 2]]])</code></pre>

ptrblck: <p>Based on the example output, it looks like each channel contains different values.<br>
Are you sure that the mask image does not contain a color code?<br>
Could you try to call <code>target.view(target.size(0), -1).unique(dim=1)</code> and post the unique color values here?</p>

Vishrut10: <p>I had converted image to grayscale, shall I use annotated .png image ?</p>

ptrblck: <p>I’m not sure what the annotated .png images are, but your image does not seem to be grayscale, as the channels contain different values.<br>
What did the <code>unique</code> call return?</p>

Vishrut10: <aside class="quote no-group" data-username="ptrblck" data-post="14" data-topic="15828">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>target.view(target.size(0), -1).unique(dim=1)</p>
</blockquote>
</aside>
<p>labels.view(labels.size(0), -1).unique(dim=1)<br>
tensor([[ 2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,<br>
2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,<br>
2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  3,  3,<br>
3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,<br>
3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  3,  5,  5,  5,  5,  5,  5,  5,<br>
5,  5,  5,  5,  5,  5,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,  6,<br>
6,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  7,  8,  8,  8,  8,<br>
8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,  8,<br>
8,  9,  9,  9,  9,  9,  9,  9,  9],<br>
[ 2,  2,  2,  2,  2,  2,  2,  2,  3,  3,  3,  3,  3,  3,  4,  4,  4,  5,<br>
5,  5,  5,  5,  6,  6,  6,  7,  7,  7,  7,  7,  8,  8,  8,  8,  8,  9,<br>
9,  9,  9,  9,  9,  9,  9, 10, 10, 10, 10, 11, 11, 11, 11, 11,  2,  2,<br>
2,  2,  2,  2,  2,  3,  3,  3,  3,  3,  3,  4,  6,  6,  7,  7,  7,  7,<br>
8,  8,  8,  8,  9,  9,  9,  9,  9, 11, 11,  2,  2,  2,  3,  3,  3,  9,<br>
9,  9, 10, 10, 11, 11,  2,  2,  2,  5,  6,  7,  7,  7,  9,  9, 10, 11,<br>
11,  2,  2,  4,  4,  7,  7,  9,  9, 10, 10, 10, 11, 11,  2,  2,  2,  2,<br>
2,  2,  3,  3,  3,  3,  3,  3,  6,  7,  7,  8,  8,  8,  8,  9,  9,  9,<br>
9,  2,  2,  3,  3,  7,  7, 10, 10],<br>
[ 1,  2,  3,  5,  6,  7,  8,  9,  2,  3,  6,  7,  8,  9,  2,  7,  9,  1,<br>
2,  3,  8,  9,  1,  2,  6,  2,  5,  6,  7,  9,  2,  3,  6,  8,  9,  1,<br>
2,  3,  5,  6,  7,  8,  9,  1,  2,  3,  5,  2,  3,  5,  6,  7,  2,  3,<br>
5,  6,  7,  8,  9,  2,  3,  6,  7,  8,  9,  7,  2,  6,  2,  5,  6,  7,<br>
2,  3,  6,  9,  2,  3,  5,  6,  7,  3,  7,  2,  3,  8,  2,  3,  8,  2,<br>
3,  8,  2,  3,  2,  3,  2,  5,  9,  2,  2,  2,  5,  9,  2,  5,  2,  2,<br>
5,  2,  3,  2,  3,  2,  5,  2,  3,  2,  3,  5,  2,  3,  2,  3,  6,  7,<br>
8,  9,  2,  3,  6,  7,  8,  9,  2,  2,  6,  2,  3,  6,  9,  2,  3,  6,<br>
7,  2,  9,  2,  9,  2,  9,  2,  3]])</p>
<p>I definitely sense some problem here</p>

ptrblck: <p>It looks a bit strange.<br>
I assumed your tensor has a shape of <code>[channels, height, width]</code>, which should yield a clean output of the different color values.<br>
Are you using an additional batch dimension?<br>
If so, could you slice the tensor and run the code again?</p>
<p>Anyway, the values look a bit fishy and something might be wrong.<br>
Could you also try to plot the image using e.g. <code>matplotlib</code>?</p>

Vishrut10: <p>Thats a sample .png image</p>
<p>plt.imshow(img)<br>
Out[10]: &lt;matplotlib.image.AxesImage at 0x1288d4da0&gt;<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/f/ffb557efca59a84988ed621ac56a713ef1f8744d.png" alt="image" data-base62-sha1="Au6hszCAjca33KO3DjtlZUIpNZb" width="262" height="252"></p>
<p>img.size<br>
Out[12]: (512, 512)</p>
<p>np.unique(np.array(img))<br>
Out[13]: array([2, 3, 5, 6, 7, 8, 9], dtype=uint8)</p>

ptrblck: <p>This looks perfectly fine.<br>
Could you use this target image and try to recreate the loading logic from your <code>Dataset</code> and check, which method creates these pseudo-values?</p>