Multi class segmentation

username: henry_Kang
cooked: <p>Hello. I am dealing with the multi-class segmentation.<br>
I used to handle the binary class for semantic segmentation.<br>
In the binary, I use the binary mask as the target.<br>
However in the multi-class, it looks like I need some change.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd7ce7e03b739f1ddb8a3f236a61d64b91cdaf1f.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd7ce7e03b739f1ddb8a3f236a61d64b91cdaf1f" title="DSC_3974"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd7ce7e03b739f1ddb8a3f236a61d64b91cdaf1f.png" alt="DSC_3974" data-base62-sha1="tjPuqFKqskRzTutdBODB2epkJdl" width="690" height="388" data-dominant-color="010101"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">DSC_3974</span><span class="informations">1280×720 1.62 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is my mask.<br>
I have 5 classes which are Red, Green,Blue, white and black.<br>
My model output is the 5 channel.<br>
self.fuse=conv_1(ch_in=112,ch_out=5)<br>
I use this 5 classes with 3 channel image as 1 channel image use below code.</p>
<pre><code>                r,g,b=cv2.split(target_img)
                r[r==255]=1
                g[g==255]=2
                b[b==255]=3
                mask=r+g+b
                mask[mask==6]=4
                target_img=mask
</code></pre>
<p>After that I run the below train loop.<br>
Because I read some article and they said I have to use one hot encoding for cross entropy loss.</p>
<pre><code class="lang-auto">
        self.criterion = nn.CrossEntropyLoss().to(self.device)

    def run_train_loop(self, epochs):
        # Run training
        for epoch in range(epochs):
            print('Epoch {}/{}'.format(epoch + 1, epochs))
            print('-' * 10)
            self.scheduler.step()
            for param_group in self.optimizer.param_groups:
                lr = param_group['lr']
            print("Learning rate: " + str(lr))
            running_loss = 0.0
            running_score = 0.0

            for batch_idx, (images, masks) in enumerate(self.train_data_loader):
                # Obtain batches
                images, masks = images.to(self.device), masks.to(self.device)
                masks=masks.squeeze(1)
                self.optimizer.zero_grad()
                model= self.model.to(self.device)
                output = model(images)
                loss = self.criterion(output, masks.long())
</code></pre>
<p>However, my loss is 0 after some epoch. I have 2000 images and mask. So I think it is not the lack of data problem. I am not using the softmax function but I read some answer in this forum that it has already included in cross entropy function. If not I have to include it of course.<br>
Also I am not sure that it is the  correct way that the ground truth channel is 1 channel only. Becuase output of the model is 5 channel. Do you think I should change the my mask as 5 channel with 0 and 1? Or should my model output is the 5 channels, because 0 is background.</p>
<p>Also I am not sure that what is class mapping? Does it just decode the segmentation output for visualization? Or how can  I put class information to the training process?<br>
I am little bit confuse about it.<br>
Thank you.</p>

username: ptrblck
cooked: <aside class="quote no-group quote-modified" data-username="henry_Kang" data-post="1" data-topic="82057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/henry_kang/48/23282_2.png" class="avatar"> henry_Kang:</div>
<blockquote>
<p>Because I read some article and they said I have to use one hot encoding for cross entropy loss.<br>
[…]<br>
Do you think I should change the my mask as 5 channel with 0 and 1? Or should my model output is the 5 channels, because 0 is background.</p>
</blockquote>
</aside>
<p>Nom you shouldn’t use one-hot encoded target tensors for <code>nn.CrossEntropyLoss</code>, as it expects a model output in the shape <code>[batch_size, nb_classes, height, width]</code> and a target tensor in the shape of <code>[batch_size, height, width]</code> containing the class indices in the range <code>[0, nb_classes-1]</code> for a multi-class segmentation use case.</p>
<aside class="quote no-group" data-username="henry_Kang" data-post="1" data-topic="82057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/henry_kang/48/23282_2.png" class="avatar"> henry_Kang:</div>
<blockquote>
<p>Or should my model output is the 5 channels, because 0 is background.</p>
</blockquote>
</aside>
<p>For 4 valid classes + a background class, the number of classes should be 5.</p>
<aside class="quote no-group" data-username="henry_Kang" data-post="1" data-topic="82057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/henry_kang/48/23282_2.png" class="avatar"> henry_Kang:</div>
<blockquote>
<p>Also I am not sure that what is class mapping?</p>
</blockquote>
</aside>
<p>A class mapping might be a method which maps the different discrete color codes from the target image to a class index, e.g. red → class0, green → class1, blue → class2, pink → class3, …</p>

username: henry_Kang
cooked: <p>Thank you for your answer. And sorry for the late response. My alarm just pop up after 6 days.<br>
I do not know why.<br>
for the first question. I use the code like Camvid dataset and solve the issue.</p>
<blockquote>
<p>def get_class_weights(num_classes, c=1.02):<br>
pipe = loader(‘./datasets/CamVid/train1/’, ‘./datasets/CamVid/trainannot1/’, batch_size=‘all’)<br>
_, labels = next(pipe)<br>
all_labels = labels.flatten()<br>
each_class = np.bincount(all_labels, minlength=num_classes)<br>
prospensity_score = each_class / len(all_labels)<br>
class_weights = 1 / (np.log(c + prospensity_score))<br>
return class_weights</p>
</blockquote>
<blockquote>
<p>criterion = nn.CrossEntropyLoss(weight=torch.FloatTensor(class_weights).to(device))</p>
</blockquote>
<p>I hope this is correct. It looks like somewhat work also.<br>
Thank you.</p>

username: ptrblck
cooked: <p>It seems as if the <code>class_weights</code> might only be calculated from a single batch?<br>
If that’s the case, the weights might be quite off and you should try to grab at least a few batches to approximate the valid class distribution.</p>

username: henry_Kang
cooked: <p>Thank you for point out. I do not think about that problem. I followed the example from ENet implementation  <a href="https://github.com/iArunava/ENet-Real-Time-Semantic-Segmentation" rel="nofollow noopener">https://github.com/iArunava/ENet-Real-Time-Semantic-Segmentation</a><br>
<a href="https://github.com/iArunava/ENet-Real-Time-Semantic-Segmentation/blob/master/ENet-Real_Time_Semantic_Segmentation.ipynb" rel="nofollow noopener">https://github.com/iArunava/ENet-Real-Time-Semantic-Segmentation/blob/master/ENet-Real_Time_Semantic_Segmentation.ipynb</a></p>
<p>If you do not mind how I can change the my code to yours?<br>
I mean for example if I use the batch size 4 in the 2 GPU in that cases, I have to connect 2 labels and calculate the class weights?</p>

username: ptrblck
cooked: <p>Assuming <code>pipe</code> is a <code>DataLoader</code> object, you could iterate it once and collect all targets via:</p>
<pre><code class="lang-python">targets = []
for _, target in pipe:
    targets.append(target)
targets = torch.stack(targets)
</code></pre>
<p>and calculate the class distribution later.</p>
<p>I hope that the <code>target</code> tensors are not too big to fit into your RAM.<br>
If that’s the case, you might need to add a <code>break</code> statement after a certain amount of targets.<br>
As long as you shuffle the targets and grab “enough” of them, the calculated estimation of the class distribution could come close to the real distribution.</p>

username: henry_Kang
cooked: <p>Oh, Now I understand what you say. it really needs to loop all batches.<br>
Thank you for your answer.</p>