Semantic Segmentation Classes

Giulia_Carvalhal: <p>Hi, community,</p>
<p>I am trying to train my Unet model with a multiclass dataset. My ground-truth are grayscale, not normalized and my masks are grayscale in range (0-3) with pixels equal to each class (0=background, 1=tissue1, 2=tissue2, 3=tissue3). In this case, should I make a mapping of the classes? How can I use the softmax and argmax in the evaluation?</p>
<p>I am asking this because my model is predicting very badly. With Adam optimizer, he is all black until epoch 900, at 900 he predicts the background and the perfect format of the tissue3 (but with the pixels value changed, as pixel=1). With SGD optimizer he stars with a reasonable segmentation, but in the 200 epoch, he is all background. I have no idea what is happening. I hope someone could help me.</p>
<p>Best Regards,<br>
Giulia</p>

ptrblck: <aside class="quote no-group" data-username="Giulia_Carvalhal" data-post="1" data-topic="134561">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/giulia_carvalhal/48/43104_2.png" class="avatar"> Giulia_Carvalhal:</div>
<blockquote>
<p>My ground-truth are grayscale, not normalized and my masks are grayscale in range (0-3) with pixels equal to each class (0=background, 1=tissue1, 2=tissue2, 3=tissue3). In this case, should I make a mapping of the classes?</p>
</blockquote>
</aside>
<p>It seems your target segmentation maps are already properly mapped.<br>
As a quick test could you check the unique values via <code>print(target.unique())</code> and make sure that only the valid class indices are returned?</p>
<aside class="quote no-group" data-username="Giulia_Carvalhal" data-post="1" data-topic="134561">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/giulia_carvalhal/48/43104_2.png" class="avatar"> Giulia_Carvalhal:</div>
<blockquote>
<p>How can I use the softmax and argmax in the evaluation?</p>
</blockquote>
</aside>
<p>Assuming your model outputs <code>[batch_size, nb_classes, height, width]</code> then <code>preds = torch.argmax(output, dim=1)</code> would give you the predicted class indices.</p>

Giulia_Carvalhal: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Thank you for replying. I check it, and it is incorrect (value= [0,1]). But this problem only occurs after transforming the targets to long tensor. Before this step the unique values are correct (0, 0.333, 0.666, 1). Observation: I divide the pixels by 3, to normalize them.</p>
<p>Why does this change when transform to long tensor is occurring? How can I solve it?</p>
<p>Best,<br>
Giulia</p>

ptrblck: <p>The output is expected to have the values <code>[0, 1]</code> since you are rounding <code>[0, 0.333, 0.666, 1]</code> to <code>long()</code>.<br>
Given that the target already seemed to contain the right class indices you should not normalize it additionally or why would you want to change/normalize its values?</p>

Giulia_Carvalhal: <p>I did this normalization process because the <code>ToPILImage</code> is creating the different output, as you are passing a <code>FloatTensor</code> , which is assumed to be in the range <code>[0, 1]</code> , while yours has the range <code>[0, 4*]</code> . <a href="https://discuss.pytorch.org/t/pixel-values-changes-when-apply-transform-totensor/78504/6" class="inline-onebox">Pixel values changes when apply transform.ToTensor - #6 by ptrblck</a></p>
<p>When I did not normalize, the ToPILImage set 0 to 0, and 1,2, and 3 to 1.</p>

Giulia_Carvalhal: <p>And the target unique values should not display like [0, 0.333, 0.666, 1]? Instead of [0,1]?</p>

ptrblck: <p>I might not fully understand the current use case.<br>
Are you using <code>ToPILImage</code> on the model output to visualize it or on the target tensors?<br>
In case it’s just to visualize the target, you might scale the values to <code>[0, 0.33,0.66, 1]</code>. However, don’t transform the target during training and keep the values as the class indices <code>[0, 1, 2, 3]</code>.</p>

Giulia_Carvalhal: <p>Okay, thank you very much! Now the targets look alright.</p>
<p>“Target Unique tensor([0, 1, 2, 3], device=‘cuda:0’)”</p>
<p>Best,<br>
Giulia</p>

Giulia_Carvalhal: <p>Hey one more doubt,</p>
<p>Based on this softmax example.</p>
<pre><code class="lang-auto">def check_accuracy(loader, model, num_labels, time=0, device=DEVICE):
    num_correct = 0
    num_pixels = 0
    dice_score = 0
    
    model.eval()

    with torch.no_grad():
        for x, y in loader:
            x = x.to(device)
            preds = model(x)
            y = y.to(device)

            lossfn = nn.CrossEntropyLoss()
            celoss = lossfn(preds, y.long())

            #y = y.unsqueeze(1)

            preds_labels = torch.argmax(preds, dim=1)
    
            num_correct += (preds_labels == y).sum()
            num_pixels += torch.numel(preds_labels)
     
            dice_score = dice_loss(y, preds_labels, num_labels)

    acc = evaluate_segmentation(preds_labels, y, 4, score_averaging='weighted')
    acc.append(dice_score)
    acc.append(celoss.cpu().numpy())

    print_and_save_results(num_correct, num_pixels, acc, time)  

    model.train()

def save_predictions_as_imgs(loader, model, epoch, folder="data/predictions/", device=DEVICE):
    print("=&gt; Saving predictions as images")
    model.eval()
    with torch.no_grad():    
        for idx, (x, _) in enumerate(loader):
            x = x.to(device)
            preds = model(x)
            preds_labels = torch.argmax(preds, dim=1)
           # preds = torch.log_softmax(model(x), 1)
           # preds_labels = torch.argmax(preds, 1)
            
            preds_labels = label_to_pixel(preds_labels)

            now = datetime.now().strftime("%Y%m%d_%H%M%S")

            save_image(preds_labels, f"{folder}{now}_pred_e{epoch}_i{idx}.png")

            # preds_labels = ToPILImage()(preds_labels).convert("L")
            # preds_labels.save(f"{folder}{now}_pred_e{epoch}_i{idx}.png")
            
    model.train()


def save_validation_as_imgs(loader, folder="data/predictions/", device=DEVICE):
    print("=&gt; Saving predictions as images")

    for idx, (_, y) in enumerate(loader):
        y = y.to(device)
        val = y.unsqueeze(1)

        now = datetime.now().strftime("%Y%m%d_%H%M%S")
        
        save_image(val, f"{folder}{now}_val_i{idx}.png")
        #val = ToPILImage()(y).convert("L")
        #val.save(f"{folder}{now}_val_i{idx}.png")

</code></pre>
<p>Would this code be correct?</p>

ptrblck: <p>I’m not seeing a <code>softmax</code> usage besides this commented out line:</p>
<pre><code class="lang-python">           # preds = torch.log_softmax(model(x), 1)
           # preds_labels = torch.argmax(preds, 1)
</code></pre>
<p>which would not be needed before applying <code>argmax</code>, so you could remove it.</p>
<p>Besides that the code looks good.<br>
I would probably not recreate the criterion inside the <code>DataLoader</code> loop and just reuse it instead.<br>
Also, I don’t know how the <code>dice_loss</code> etc. is implemented and you could check the loss and accuracy calculations with a known example just as a smoke test.</p>