Semantic segmentation for binary classification issue

Nishanth_Sasankan: <p>I am very new to Pytorch and Deep Learning in general. I have a set of grayscale images which I convert to a 3 channel images by repeating the first channel two more times. I am using the fcn_resnet101 from the pytorch models. The model only predicts one class for all images. Am I missing something from the methodology below.</p>
<pre><code class="lang-auto">import torch
import torchvision
import loader
from loader import DataLoaderSegmentation
import torch.nn as nn
import torch.optim as optim
import numpy as np
from torch.utils.data.sampler import SubsetRandomSampler
batch_size = 1
validation_split = .2
shuffle_dataset = True
random_seed= 66

n_class    = 2
num_epochs = 1
#lr         = 1e-4
#momentum   = 0
w_decay    = 1e-5
step_size  = 50
gamma      = 0.5

traindata = DataLoaderSegmentation('/home/ubuntu/Downloads/Brain/test0716')
dataset_size = len(traindata)
indices = list(range(dataset_size))
split = int(np.floor(validation_split * dataset_size))
#if shuffle_dataset:
np.random.seed(random_seed)
np.random.shuffle(indices)
train_indices = indices[split:]
val_indices = indices[:split]

#train_sampler = SubsetRandomSampler(train_indices)
#valid_sampler = SubsetRandomSampler(val_indices)
trainloader = torch.utils.data.DataLoader(traindata, batch_size=batch_size)
validation_loader = torch.utils.data.DataLoader(traindata, batch_size=batch_size)

#trainloader = torch.utils.data.DataLoader(traindata, batch_size=1, shuffle=False, num_workers=2)

model = torchvision.models.segmentation.fcn_resnet101(pretrained=False, progress=True,aux_loss=None, num_classes=2).cuda()
criterion = nn.CrossEntropyLoss().cuda()
#optimizer = optim.RMSprop(model.parameters(), lr=lr, momentum=momentum, weight_decay=w_decay)
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.train()


for iter in range(num_epochs):
    print(iter)	
    for (i,l) in trainloader:
        i= i.to(device)
        l = l.to(device=device, dtype=torch.int64)
        outt = model(i)
        loss = criterion(outt['out'], l.squeeze(1))
        #print(loss)
        loss.backward()
        optimizer.step()
torch.save(model, '/home/ubuntu/Downloads/Brain/test0716/mod.pth)
</code></pre>
<p>Thanks for all your help</p>

ptrblck: <p>In your training loop you are not zeroing out the gradients, which will thus accumulate them in each iteration.<br>
Add <code>optimizer.zero_grad()</code> at the beginning of your <code>DataLoader</code> loop:</p>
<pre><code class="lang-python">for (i, l) in trainloader:
    optimizer.zero_grad()
    i = ...
</code></pre>
<p>Besides that the code looks fine. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Let me know, if that helps.</p>

Nishanth_Sasankan: <p>Thank you for your quick response. I included your suggestion and rerun the code. I am testing the model with an image I know that has 2 classes</p>
<pre><code class="lang-auto">import torchvision
import numpy as np
from torch.utils.data.sampler import SubsetRandomSampler
from PIL import Image
import matplotlib.pyplot as plt
import torchvision.transforms as T

def decode_segmap(image, nc=2):
    label_colors = np.array([(0, 0, 0),(255,255,255)])
    r = np.zeros_like(image).astype(np.uint8)
    g = np.zeros_like(image).astype(np.uint8)
    b = np.zeros_like(image).astype(np.uint8)

    for l in range(0, nc):
        idx = image == l
        r[idx] = label_colors[l, 0]
        g[idx] = label_colors[l, 1]
        b[idx] = label_colors[l, 2]

    rgb = np.stack([r, g, b], axis=2)
    return rgb
#import pic directly, webmethod check
import cv2
img = cv2.imread('/home/ubuntu/Downloads/Brain/test0716/train/slice_src_BN01002_032.png')
label = cv2.imread('/home/ubuntu/Downloads/Brain/test0716/trainannot/slice_src_BN01002_032.png')
plt.imshow(img); plt.show()
plt.imshow(label); plt.show()
trf = T.Compose([T.ToTensor()])
inp = trf(img).unsqueeze(0).cuda()
fcn = torch.load('/home/ubuntu/Downloads/Brain/test0716/mod.pth')
fcn.eval()
sam_out = fcn(inp)['out']
om = torch.argmax(sam_out.squeeze(), dim=0).cpu().numpy()
rgb = decode_segmap(om)
print(rgb.shape)
print (np.unique(rgb))
plt.imshow(rgb);

</code></pre>
<p>This is the code I use to visualize the result, The prediction gives me an image with only 1 label. I greatly appreciate your help. Please let me know if you have any suggestions.<br>
Thanks<br>
Nishanth</p>

ptrblck: <p>The code for visualization should generally work:</p>
<pre><code class="lang-python">output = torch.randn(1, 2, 224, 224)
pred = torch.argmax(output.squeeze(), 0).numpy()
seg_output = decode_segmap(pred)

plt.imshow(seg_output)
</code></pre>
<p>This code snippet gives a random segmentation output.</p>
<p>Did you observe your training and validation loss and accuracy?<br>
Your model might be overfitting to a single class, but that should already be visible during training in your validation loss.</p>

Nishanth_Sasankan: <p>Thank you for your continued response. I tried your code snippet and it did give me a random segmented output.</p>
<p>As for your question</p>
<p>“Did you observe your training and validation loss and accuracy?”</p>
<p>I am not sure how I can visualize training, validation loss and accuracy during training. Also, how do I implement validation and validation loss during the training epoch?</p>
<p>How do I know if the model is overfitting for one of the class? Also, how do I correct for it?</p>
<p>Thanks<br>
Nishanth</p>

ptrblck: <aside class="quote no-group" data-username="Nishanth_Sasankan" data-post="5" data-topic="50920">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nishanth_sasankan/48/13219_2.png" class="avatar"> Nishanth_Sasankan:</div>
<blockquote>
<p>I am not sure how I can visualize training, validation loss and accuracy during training. Also, how do I implement validation and validation loss during the training epoch?</p>
</blockquote>
</aside>
<p>Usually you would create a <code>train</code> function to train the model for a complete epoch using the training dataset. Afterwards you could run the evaluation on the validation dataset.<br>
Have a look at the <a href="https://github.com/pytorch/examples/blob/ace46c8223edd0fe0e857a6144326b9012ecf24d/imagenet/main.py#L258">ImageNet exmple</a> to see how <code>train</code> and <code>validate</code> are defined.<br>
The same code uses <a href="https://github.com/pytorch/examples/blob/ace46c8223edd0fe0e857a6144326b9012ecf24d/imagenet/main.py#L300">this code</a> to print the progress.</p>
<aside class="quote no-group" data-username="Nishanth_Sasankan" data-post="5" data-topic="50920">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nishanth_sasankan/48/13219_2.png" class="avatar"> Nishanth_Sasankan:</div>
<blockquote>
<p>How do I know if the model is overfitting for one of the class? Also, how do I correct for it?</p>
</blockquote>
</aside>
<p>One signal of overfitting is when you observe a decreasing training loss and a static or increasing validation loss.<br>
Further training at this point usually won’t give you any benefits and you would stop at this point.<br>
You would counter it by e.g.</p>
<ul>
<li>using (or collecting) more data</li>
<li>remove “capacity” of the model, e.g. by reducing the number of parameters</li>
<li>add regularization (e.g. dropout)</li>
</ul>
<p>You can find way more information and a better explanation in <a href="https://www.deeplearningbook.org/contents/regularization.html">Chapter 5, DeepLearningBook</a>.</p>

Nishanth_Sasankan: <p>Thank you for your response. I noticed that my Dataset is heavily imbalanced, there are way more pixels representing the backgrounds than the organ (label I am interested in). I have 18000 images of which only 600 have the organ class I am looking for. Is it possible that this is the reason why only one of the classes is learned?</p>
<p>Thanks<br>
Nishanth</p>

ptrblck: <p>Yes, that could be possible.<br>
You could try to oversample the minority class using e.g. a <code>WeightedRandomSampler</code>.</p>

Nishanth_Sasankan: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I tried working with a more even dataset with balanced classes. I also tried playing around with the weight_decay parameter in the optimizer. However, I am still running into the issue where my model is only outputting 1 class. I finally decided to just increase the epochs and overfit the model with training data and perform prediction on a training image. It turns out I am still only getting 1 class when I am supposed to get 2 classes. I tried scouring the internet for a similar issue, but couldn’t find one.  Any and all help is very much appreciated.</p>
<p>Thanks<br>
Nishanth</p>
<p>I have provided a detailed explanation in the following post</p><aside class="quote quote-modified" data-post="1" data-topic="52353">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nishanth_sasankan/48/13219_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/semantic-segmentation-using-deeplabv3-resnet101-from-torchvision-models/52353">Semantic Segmentation using deeplabv3+resnet101 from torchvision models</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I am using the Deeplab V3+ resnet 101 to perform binary semantic segmentation. 
import torch
import torchvision
import loader
from loader import DataLoaderSegmentation
import torch.nn as nn
import torch.optim as optim
import numpy as np
from torch.utils.data.sampler import SubsetRandomSampler

batch_size = 1
validation_split = .2
shuffle_dataset = True
random_seed= 66

n_class    = 2
num_epochs = 1
lr         = 1e-4
momentum   = 0.9
w_decay    = 1e-5
step_size  = 50
gamma      = 0.5
traindata = …
  </blockquote>
</aside>