How to pass semantic segmentation labels to FCN?

avalon1511: <p>I have the following data:</p>
<p>Ground truth image<br>
Segmentation mask image<br>
And a <code>.npy</code> file that contains the pixel wise labels for the ground truth.</p>
<p>I want to use this data to train a FCN from scratch. The structure of the FCN is as follows -</p>
<pre><code class="lang-auto">Conv2D
Dropout
BN
Activation
</code></pre>
<p>This block is repeated three times to finish the model.</p>
<p>How do I pass the <code>.npy</code> data to a FCN so that I can train it from scratch to generate segmentation masks?</p>

ptrblck: <p>You should be able to load the numpy array via <code>np.load</code>, transform it to a tensor via <code>torch.from_numpy(array)</code>, and pass the tensor to the PyTorch model.</p>

avalon1511: <p>Thanks this worked! However, my classification accuracy is really bad. My network is as follows:</p>
<pre><code class="lang-auto">class label_net_3c(nn.Module):

    def __init__(self):
        
        super(label_net_3c, self).__init__()


        self.conv1 = nn.Conv2d(3, 3, kernel_size = 1)
        self.bn1 = nn.BatchNorm2d(3)
        self.act1 = nn.ReLU(inplace = True)
        self.conv2 = nn.Conv2d(3,2, kernel_size = 1)
        self.bn2 = nn.BatchNorm2d(2)
        self.act2 = nn.ReLU(inplace = True)
        self.conv3 = nn.Conv2d(2, 34, kernel_size = 1)
        self.act3 = nn.Softmax(dim = 1)

    
    def forward(self, x):

        x = self.conv1(x)
        x = self.bn1(x)
        x = self.act1(x)
        x = self.conv2(x)
        x = self.bn2(x)
        x = self.act2(x)
        x = self.conv3(x)
        x = self.act3(x)

        return x
</code></pre>
<p>it’s a pixel wise classifier so I pass batches of pixels of size N such that the input is <code>N x 3 x 1 x 1</code> for a given RGB image. Any tips on how to improve the accuracy?</p>

ptrblck: <p>I guess you are using <code>nn.CrossEntropyLoss</code> as the loss function for your segmentation use case.<br>
If so, remove the <code>nn.Softmax</code> layer as <code>nn.CrossEntropyLoss</code> expects raw logits not probabilities.</p>

avalon1511: <p>Gotcha, I assume <code>nn.CrossEntropyLoss</code>. has a built in Softmax operation. Although I did what you suggested but my accuracy is still really bad.</p>

ptrblck: <p>Yes, <code>nn.CrossEntropyLoss</code> calls <code>F.log_softmax</code> and <code>nn.NLLLoss</code> internally.<br>
Could you try to overfit a small dataset (e.g. just 10 samples) by playing around with some hyper-parameters? If this still doesn’t work there might be another issue in the code which I missed.</p>

avalon1511: <p>Hi, I think my features were just too weak since I was trying to see if I could get some classification results with a few samples only. I am trying to now plot the intermediate features.</p>
<p>I have a tensor of size (1,32,256,256) - can I plot a t-sne for this? just want to visualize the data dont want to compare it to the ground truth.</p>

avalon1511: <p>I figured out the problem. I am doing</p>
<pre><code class="lang-auto">m = pixel_classifier()
pred = m(train_batch)
print(pred.grad_fn)
# &lt;AddmmBackward0 object at 0x7f865771f590&gt;
_, pred = torch.max(pred, dim = 1)
print(pred.grad_fn)
# None
</code></pre>
<p>Seems like <code>torch.max()</code> is breaking the graph and my loss function + optimizer parameters arent updating for this reason.</p>
<p>Is there a workaround for torch.max()? I need it to get the labels for the predictions.</p>

ptrblck: <p><code>torch.argmax</code> (i.e. the second return value from <code>torch.max</code>) is not differentiable as the gradients would be almost everywhere zero.<br>
You can use it to calculate the accuracy but not to train the model.<br>
For a multi-class classification use <code>nn.CrossEntropyLoss</code> and pass the logits to it.</p>

aml.rhmn: <p>Hi ptrblck,</p>
<p>I need a bit of help on the semantic segmentation task as well.<br>
So I followed your instructions in this thread and did the following:</p>
<ol>
<li>Removed the nn.softmax layer from the architecture</li>
<li>Last layer of my architecture is nn.Conv2D(in_channels, out_channels=4, …) since I have 4 classes</li>
<li>Passed the logits in nn.CrossEntropy(logits, groundtruth) and backpropagated the loss<br>
My groundtruth is of size (batch_size,1,256,256) but the logits is (batch_size, 4, 256, 256)<br>
so how can I convert the logits to predicted segmentation map of size (batch_size,1,256,256) for visualization purpose? I need the predicted classes to appear on one slice.<br>
Many Thanks !</li>
</ol>

aml.rhmn: <p>Never mind, I have resolved it. Thanks for the above discussion. :))</p>