Creating a simple Semantic Segmentation Network

mayool: <p>Hi everyone!<br>
I’m pretty new to pytorch and interested in Semantic Segmantion.<br>
I know there are severeal pretrained models included in pytorch, but i would like to build one from scratch to really understand what is going on.</p>
<p>What I want is a <strong>binary</strong> pixelwise prediction that says me for each pixel, if that pixel belongs to a car/human/whatever or whether it is background.</p>
<p>I’ve read the “common” papers like U-Net, FCN, etc. but they usually have more complex skip connections etc… What I would like to try is a very simple Network that receive a: [1,3,28,28] RGB picture and returns a binary pixelwise prediction of the same size. I don’t care if the performance is good or not at the moment.</p>
<p>My main problem is, how to upsample after sending the image through several convolution layers. Obviously the shape is changed due to pooling and the stride size. How do I make sure that the resulting output has the same dimensions like the input image? I’ve read that I can use nn.ConvTranspose2d() to upsample, but I have no idea what values the parameters should take in order to end up with a pixelwise prediction.</p>
<p>Can someone help me out?</p>

ptrblck: <aside class="quote no-group" data-username="mayool" data-post="1" data-topic="74785">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/41988e/48.png" class="avatar"> mayool:</div>
<blockquote>
<p>What I want is a <strong>binary</strong> pixelwise prediction that says me for each pixel, if that pixel belongs to a car/human/whatever or whether it is background.</p>
</blockquote>
</aside>
<p>Based on this description, it seems you would like to work on a multi-class segmentation rather than a binary segmentation, as you are dealing with multiple classes.<br>
Alternatively, if each pixel might belong to multiple classes, you would work on a multi-label segmentation.</p>
<aside class="quote no-group" data-username="mayool" data-post="1" data-topic="74785">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/41988e/48.png" class="avatar"> mayool:</div>
<blockquote>
<p>I’ve read that I can use nn.ConvTranspose2d() to upsample, but I have no idea what values the parameters should take in order to end up with a pixelwise prediction.</p>
</blockquote>
</aside>
<p>The parameters for upsampling depend on the number of layers you would like to use.<br>
You could “revert” the conv/pooling operation or use another setup with more or less transposed convs.</p>
<p>I’ve written quite a simple UNet implementation a while ago and you can find it <a href="https://github.com/ptrblck/pytorch_misc/blob/master/unet_demo.py">here</a>.<br>
Maybe it’s good as a starter code or you could have a look at the setup of the transposed conv layers.</p>

mayool: <p>Thank you so much! Very easy to understand implementation!</p>
<p>However, I do have 2 questions:</p>
<ol>
<li>
<p>your x (line 112) has shappe (1,3,96,96) (i assume (batch_size,nb_channels,heigth, width)), your y (line 113) has shape (1,96,96) and your model output has shape (batch_size, nb_classes, 96,96).<br>
–&gt; How is your loss function calculated. Shouldn’t y and output have the same dimensions? Or is that something special NLLLoss does?</p>
</li>
<li>
<p>How did you know that your model output will have the same height and width like your input image (<em>,</em>,96,96)?<br>
I mean all the parameters (stride, padding, etc.) are directly influencing the output shape. How did you know that the parameters you choose (e.g. when instantiating the model (line 115 - 120) or your upconv-layer (line 53)) will lead to the same output shape? Is it just try and error or is there some trick? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>
</li>
</ol>
<p>Thanks again for sharing the code with me!</p>

ptrblck: <aside class="quote no-group" data-username="mayool" data-post="3" data-topic="74785">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/41988e/48.png" class="avatar"> mayool:</div>
<blockquote>
<p>How is your loss function calculated. Shouldn’t y and output have the same dimensions? Or is that something special NLLLoss does?</p>
</blockquote>
</aside>
<p><code>nn.CrossEntropyLoss</code> (and thus also <code>nn.NLLLoss</code>) expect the model output to have the shape <code>[batch_size, nb_classes, height, width]</code> and the target <code>[batch_size, height, width]</code> containing the class indices for a segmentation use case.</p>
<aside class="quote no-group" data-username="mayool" data-post="3" data-topic="74785">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/41988e/48.png" class="avatar"> mayool:</div>
<blockquote>
<p>Is it just try and error or is there some trick? <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>For even inputs shapes and “simple” convolutions, you can easily create the transposed conv so that you’ll get the same output shape.<br>
However, if you are using odd input shapes, dilations, etc. you might need to use <code>output_padding</code> in your transposed convs.<br>
Usually, I add print statements after each layer and make sure I get the shape I need.</p>