How can I display a test image and display the mask for it based on my trained model?

username: Neda
cooked: <p>I am wondering how I can test the trained model for semantic segmentation and visualise the mask for the test image. There is an example for classification problem in Pytorch but couldn’t find any obvious example for the segmentation.</p>
<p>I found <a href="https://medium.com/@tsakunelsonz/loading-and-training-a-neural-network-with-custom-dataset-via-transfer-learning-in-pytorch-8e672933469" rel="nofollow noopener">this page</a> that test the network, but it’s for classification problem.</p>
<p>I did manipulate it for segmentation application like below but now sure am I doing it right?</p>
<pre><code class="lang-auto">  model.eval()
    total = 0
    correct = 0
    count = 0
    #iterate through test dataset
    for data in (test_loader):
                
                t_image, mask = data
                t_image, mask = t_image.to(device), mask.to(device)
                with torch.no_grad():

                    outputs = model.forward(t_image) 
                    ps = torch.exp(outputs)
                         
                    _, predicted = torch.max(outputs.data, 1)  # Find the class index with the maximum value.
                    # We can use the PyTorch.eq() function to do this, which compares the values in two tensors and if they match, returns a 1. If they don’t match, it returns a 0
                    # By summing the output of the .eq() function, we get a count of the number of times the neural network has produced a correct output,
                    # and we take an accumulating sum of these correct predictions so that we can determine the overall accuracy of the network on our test data set.
                    total += mask.size(0)
                    correct += predicted.eq(mask.data).sum().item()
                    count +=1                
                    print("Accuracy of network on test images is ... {:.4f}....count: {}".format(100*correct/total,  count ))
</code></pre>
<p>the first question is why it used only forward here <code>outputs = model.forward(t_image)</code>?</p>
<p>And the second question is how can I visualise the output of the test, for example, how can I display a test image and drow the mask for it based on my trained model? Is there any example for this? Thank you in advance.</p>

username: tom
cooked: <aside class="quote no-group" data-username="Neda" data-post="1" data-topic="33016">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neda/48/7779_2.png" class="avatar"> Neda:</div>
<blockquote>
<p>the first question is why it used only forward here <code>outputs = model.forward(t_image)</code> ?</p>
</blockquote>
</aside>
<p>Because they haven’t asked here to be told that you should use <code>output = model(t_image)</code>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"><br>
More seriously, you would not call <code>.forward</code> unless you have a very good reason to not call the model itself (i.e. you tried that, it failed, you know why it failed and why calling <code>.forward</code> is better and, personally, I’d ask here about why calling the model won’t work).</p>
<aside class="quote no-group">
<blockquote>
<p>And the second question is how can I visualise the output of the test, for example, how can I display a test image and drow the mask for it based on my trained model? Is there any example for this? Thank you in advance.</p>
</blockquote>
</aside>
<p>I use <code>pyplot</code> a lot</p>
<pre data-code-wrap="python"><code class="lang-python"># magic line only if you use Jupyter
%matplotlib inline
from matplotlib import pyplot
...
# assuming mask is batch x h x w and we want the first
pyplot.subplot(1, 2, 1) # have two plots in 1 row two columns, first plot
# assuming im is batch x channel x h x w and channel is RGB
pyplot.plot(im[0].detach().cpu().permute(1, 2, 0))
pyplot.subplot(1, 2, 2) # second plot
pyplot.plot(mask[0].detach().cpu())
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

username: Neda
cooked: <p><a class="mention" href="/u/tom">@tom</a> thank you. calling the model is working.</p>
<p>In <code>pyplot.plot(mask[0].detach().cpu())</code> I think you did plot the current mask (ground truth) which is already in the dataset. is that right?</p>
<p>Maybe I didn’t ask my question clearly. I wanted to test the model and display a test image from test_loader and visualise a predicted segmentation mask for this test image to find out is model learned anything or not. (for example like Predicted Mask(U-Net) in fig 3 in <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5977596/" rel="nofollow noopener">this paper</a>.</p>

username: tom
cooked: <p>Hi,</p>
<p>sorry, I didn’t adapt it well to your specific code. If just using <code>predict</code> in place of mask (based on accuracy involving <code>predicted.eq(mask.data)</code>) doesn’t work, what’s the shape of predict?</p>
<p>Best regards</p>
<p>Thomas</p>

username: Neda
cooked: <p><a class="mention" href="/u/tom">@tom</a> predict size is  <code>torch.Size([1, 240, 320])</code> and image size is <code>torch.Size([1, 1, 240, 320])</code></p>

username: tom
cooked: <p>Then the above recipe should work for predict in place of mask.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Neda
cooked: <p>I still couldn’t figure out how can I visualise the output of network after training. This code will visualize the raw output but I don’t know how can I display all dim of image, at the moment will display only one channel in <code>plt.imshow(outputs[0,0,:,:].detach().cpu())</code> while the shape is <code>#print(outputs.shape) # torch.Size([1, 2, 240, 320])</code> it is the same issue with <code>plt.imshow(t_image[0,0,:,:].detach().cpu())</code> while the shape is <code>#print(t_image.size()) # torch.Size([1, 1, 240, 320])</code></p>
<pre><code class="lang-auto">model.eval()
    total = 0
    test_loss = 0
    correct = 0
    count = 0
    #iterate through test dataset
    for vi, data in enumerate(test_loader):
                
                t_image, mask = data
                #print(t_image.shape) # torch.Size([1, 1, 240, 320])
                t_image, mask = t_image.to(device), mask.to(device)
                with torch.no_grad():

                    outputs = model(t_image)
                    #print(outputs.shape) # torch.Size([1, 2, 240, 320])
                    test_loss += criterion(outputs, mask).item() / len(test_loader)

                    pr = torch.exp(outputs) # get the exp of output and will give the probability map of outputs
                    # The outputs are energies for the 2 classes.
                    # Higher the energy for a class, the more the network thinks that the image is of the particular class. So, let’s get the index of the highest energy:     
                    _, predicted = torch.max(outputs.data, 1)  
                    
                    total += mask.size(0)
                    correct += predicted.eq(mask.data).sum().item()
                    accuracy = 100 * correct / total
                    predict = predicted.eq(mask.data)
                    #print(predict.shape) # torch.Size([1, 240, 320])
                    count +=1                
                    print(count, "Test Loss: {:.3f}".format(test_loss), "Test Accuracy: %d %%" % (accuracy))

    
    plt.figure()
    plt.subplot(1, 4, 1)
    #print(outputs.shape) # torch.Size([1, 2, 240, 320])
    plt.imshow(outputs[0,0,:,:].detach().cpu())
    plt.title('DL raw output')
    plt.subplot(1, 4, 2)
    plt.imshow(predict.detach().cpu().squeeze())
    plt.title('DL prediction')
    plt.subplot(1, 4, 3)
    plt.imshow(mask.detach().cpu().squeeze())
    plt.title('ground truth')
    plt.subplot(1, 4, 4)
    #print(t_image.size()) # torch.Size([1, 1, 240, 320])
    plt.imshow(t_image[0,0,:,:].detach().cpu())
    plt.title('original input image')
</code></pre>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could you please help to sort out this issue? I did try <code>permute(1, 2, 0))</code> but couldn’t solve the issue</p>

username: ptrblck
cooked: <p>I’m not sure, why it’s not working.<br>
If you would like to visualize both probability maps for the two classes, your code should work:</p>
<pre><code class="lang-python">plt.figure()
plt.imshow(torch.exp(outputs[0,0,:,:]).detach().cpu())  # plot class0
plt.figure()
plt.imshow(torch.exp(outputs[0,1,:,:]).detach().cpu())  # plot class1
</code></pre>
<p>What is your output currently showing?</p>

username: Neda
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you for the probability map. You helped me before for the probability map of output, and it’s done. I wanted to visualise the prediction results from the test_loader, the mask (ground truth), and an original input image. I am wondering why my original input image doesn’t look grayscale image. Am I doing something wrong?</p>
<pre><code class="lang-auto">
     #test model
    model.eval()
    total = 0
    test_loss = 0
    correct = 0
    count = 0
    #iterate through test dataset
    for vi, data in enumerate(test_loader):
                
                t_image, mask = data
                #print(t_image.shape) # torch.Size([1, 1, 240, 320])
                t_image, mask = t_image.to(device), mask.to(device)
                with torch.no_grad():

                    outputs = model(t_image)
                    #print(outputs.shape) # torch.Size([1, 2, 240, 320])
                    test_loss += criterion(outputs, mask).item() / len(test_loader)

                    pr = torch.exp(outputs) # get the exp of output and will give the probability map of outpus
                    # The outputs are energies for the 2 classes.
                    # Higher the energy for a class, the more the network thinks that the image is of the particular class. So, let’s get the index of the highest energy:     
                    _, predicted = torch.max(outputs.data, 1)  
                    
                    total += mask.size(0)
                    correct += predicted.eq(mask.data).sum().item()
                    accuracy = 100 * correct / total
                    predict = predicted.eq(mask.data)
                    #print(predict.shape) # torch.Size([1, 240, 320])
                    count +=1                
                    print(count, "Test Loss: {:.3f}".format(test_loss), "Test Accuracy: %d %%" % (accuracy))

    
    plt.figure(figsize=(11,4))
    plt.subplot(1, 3, 1)
    plt.imshow(predict.detach().cpu().squeeze())
    plt.title('DL prediction')
    plt.subplot(1, 3, 2)
    plt.imshow(mask.detach().cpu().squeeze())
    plt.title('ground truth')
    plt.subplot(1, 3, 3)
    #print(t_image.size()) # torch.Size([1, 1, 240, 320])
    plt.imshow(t_image.detach().cpu().squeeze())
    plt.title('original input image')
</code></pre>

username: InnovArul
cooked: <p>you could try to use grayscale colormap.<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/3823752/display-image-as-grayscale-using-matplotlib" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/166743/ryan" target="_blank" rel="nofollow noopener">
    <img alt="Ryan" src="https://www.gravatar.com/avatar/1ad88de83c3fab1c57034ffc4df8627a?s=128&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/3823752/display-image-as-grayscale-using-matplotlib" target="_blank" rel="nofollow noopener">Display image as grayscale using matplotlib</a>
</h4>

<div class="tags">
  <strong>python, matplotlib, grayscale</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/166743/ryan" target="_blank" rel="nofollow noopener">
    Ryan
  </a>
  on <a href="https://stackoverflow.com/questions/3823752/display-image-as-grayscale-using-matplotlib" target="_blank" rel="nofollow noopener">04:33PM - 29 Sep 10 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Neda
cooked: <p><a class="mention" href="/u/innovarul">@InnovArul</a> thank you. Yes cmap = ‘gray’ display the original image as grayscale.</p>

username: Neda
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> one more question here, if I visualize the probability of output same as your example</p>
<pre><code class="lang-auto">plt.figure()
plt.imshow(torch.exp(outputs[0,0,:,:]).detach().cpu())  # plot class0
plt.figure()
plt.imshow(torch.exp(outputs[0,1,:,:]).detach().cpu())  # plot class1
</code></pre>
<p>it return a figure which is different with the figure that visualize by following example</p>
<pre><code class="lang-auto">    output = model(t_image)
    prob = torch.exp(output)
    prob_imgs = make_grid(prob.permute(1, 0, 2, 3))
    prob_imgs = prob_imgs.detach()
    plt.imshow(prob_imgs.permute(1, 2, 0))
</code></pre>
<p>what is cause to be different in these example, is it because of make_grid? because permute is just swap the first (batch size) and second dimensions (number of classes).<br>
this image is from second example, for the first example the zeros value in figures will be 1 and 1 converted to zeros. (black and white will swap). which way is correct way to do this? thank you in advance<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/ae8a8c4e387b82ee5322e2aa2739984004a729e8.png" alt="train_exp3" data-base62-sha1="oU3UTJNPtybdbNdzTA55e5wmtUQ" width="640" height="478"></p>

username: ptrblck
cooked: <p>I guess the difference in the visualizations is because <code>make_grid</code> created a “color” image (i.e. a tensor with 3 channels), which <code>matplotlib</code> apparently treats differently than an image without any channel dimension.<br>
As long as the values and visualizations are consistent, you could just stick to the approach which you like more.</p>

username: noornk
cooked: <p>Why do we have to plot the mask in two figures? I don’t get this.<br>
And how are we supposed to calculate the precision of the mask? I mean the area of intersection.</p>

username: ptrblck
cooked: <p>The code plots the probability maps for both classes, not the mask.<br>
If you are only interested on a single class, you can of course just plot the corresponding prediction.</p>
<p>You could use the <a href="https://github.com/amdegroot/ssd.pytorch/blob/5b0b77faa955c1917b0c710d770739ba8fbff9b7/layers/box_utils.py#L50" rel="nofollow noopener">IoU implementation</a> or a dice score.</p>

username: noornk
cooked: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a> sorry for asking so many questions.<br>
I still have trouble understanding this. The input of the network, for a binary class, is a  [1, w, h] image. The prediction is [1, 2, w, h] (assuming the batch size is one), and the mask(ground truth) is [1, w, h]. I have to main questions:</p>
<ol>
<li>How is the loss computed? the loss should get each cell’s value (which 0 or 1), and compare it to the mask values(again 0 or 1).</li>
<li>How to get the final predicted mask, in terms of an image? the output is class probability, then how am I suppose to get 0 or 1?</li>
</ol>

username: ptrblck
cooked: <p>If you are using <code>nn.CrossEntropyLoss</code> (or <code>nn.NLLLoss</code>) for binary classification, each channel of the output will correspond to the logits (or log probabilities) of the class index.<br>
Have a look at the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="nofollow noopener">CrossEntropyLoss docs</a> to see the applied formula.<br>
As you can see, the target will be used as an index to get the output logit, which corresponds to this target.</p>
<p>In the case of a binary classification, you can also use <code>nn.BCEWithLogitsLoss</code>, which would then expect an output of <code>[batch_size, 1, h, w]</code> and a target of the same shape.</p>
<p>If you want to get the prediction in terms of class indices, you can call <code>preds = torch.argmax(output, 1)</code> in the former case or use a threshold, if you are using <code>nn.BCEWithLogitsLoss</code> e.g.:<br>
<code>preds = torch.sigmoid(output) &gt; 0.5</code>.</p>

username: noornk
cooked: <p>Thank you so much for your reply.<br>
My output is in the shape of [batch_size, 2, h, w], and target is [batch_size, g, w], so I tried to use nn.CrossEntropyLoss, but I’m getting this error:</p>
<blockquote>
<p>RuntimeError: “host_softmax” not implemented for ‘Long’</p>
</blockquote>
<p>And when my output is [batch_size, 2, h, w], take for example these two cells: [0, 0, 10, 10] and [0, 1, 10, 10]. Shouldn’t the values complement each other? because the cell is in either class?</p>

username: ptrblck
cooked: <p>Could you post the code snippet which throws this error?<br>
Are you passing the model output as float and targets as long to <code>nn.CrossEntropyLoss</code>?</p>
<aside class="quote no-group" data-username="noornk" data-post="19" data-topic="33016">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/noornk/48/12076_2.png" class="avatar"> noornk:</div>
<blockquote>
<p>And when my output is [batch_size, 2, h, w], take for example these two cells: [0, 0, 10, 10] and [0, 1, 10, 10]. Shouldn’t the values complement each other? because the cell is in either class?</p>
</blockquote>
</aside>
<p>Since they represent logits, you won’t see any pattern there.<br>
However, you could apply a softmax on dim1 and see the corresponding probabilities in each channel for the sake of debugging.</p>

username: noornk
cooked: <p>Yeah. fixed it! However, now in the middle of training all the outputs become zero.</p>