# Classes, Accuracy, and Loss in Pixel-wise using Masks

username: RyanPatterson
cooked: <p>Im trying to train the model using a UNet framework to detect cracks in roads. My images are grayscale between 0-1.0 with shape (batchsize, <span class="hashtag">#classes</span>, image height, image width). My masks are binary with 0 for background(I dont care about) and 1 for the crack sections. My first question is can I get away with using only 1 class or do I need to use 2? I’m not sure exactly how to calculate accuracy. Also I’m a little stuck on getting useful information from the loss and accuracy.</p>
<pre><code class="lang-auto">loss_function = nn.NLLLoss(reduction='none')     # reduction of none returns the same size image
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
initial = torch.rand([b_size, image_width, image_height]).to(device)  # b_size might be wrong


def fwd_pass(X, y):
    if train:
        model.zero_grad()       # Clears the gradients
    output = model(X)
    logp = F.log_softmax(output, dim=1)
    loss = loss_function(logp, y)
    # acc = ???
    if train:
        loss.backward(gradient=initial) 
        optimizer.step()
    return loss, # acc
'''</code></pre>

username: KFrank
cooked: <p>Hello Ryan!</p>
<aside class="quote no-group" data-username="RyanPatterson" data-post="1" data-topic="81385" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/58f4c7/48.png" class="avatar"> RyanPatterson:</div>
<blockquote>
<p>My images are grayscale between 0-1.0 with shape (batchsize, <span class="hashtag-raw">#classes</span>, image height, image width). My masks are binary with 0 for background(I dont care about) and 1 for the crack sections. My first question is can I get away with using only 1 class or do I need to use 2? I’m not sure exactly how to calculate accuracy.</p>
</blockquote>
</aside>
<p>You’re basically on the right track.  Let me make some suggestions<br>
(but, note, you can usually do more-or-less the same thing in a<br>
couple of different ways).</p>
<p>You are working on a <em>binary segmentation</em> problem.  (This is a<br>
binary classifier, where you classify individual pixels.)  You could<br>
treat this as the two-class case of a general multi-class problem,<br>
but you’re marginally better off treating this as a binary problem.<br>
So don’t <em>explicitly</em> use two classes.</p>
<p>The images you input to your model should have shape<br>
<code>[batchsize, height, width]</code> – no class dimension,<br>
because your <em>input</em> images don’t know anything about class.<br>
The output of your model – which will be one of the inputs to<br>
your loss function – should have the same shape – also no<br>
class dimension (because were implementing this as a binary<br>
problem).  Your masks (sometimes called <em>labels</em> or <em>targets</em>) – the<br>
other input to your loss function – should also have the same<br>
shape, again no class dimension (the class information is contained<br>
in the binary values of the mask pixels).  And even though (in your<br>
case) binary, your masks should be <code>FloatTensor</code>s.</p>
<p>You should use <code>BCEWithLogitsLoss</code>.  (<code>NLLLoss</code> is reasonable,<br>
but is for the multi-class problem, and you’re better off going the<br>
pure binary route.)  The output of your model should be raw-score<br>
<em>logits</em> that run from <code>-inf</code> to <code>inf</code>, so you shouldn’t pass them<br>
through a non-linearity such as <code>sigmoid()</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>I’m not sure exactly how to calculate accuracy.</p>
</blockquote>
</aside>
<p>Conceptually, you convert a prediction logit to a <em>probability</em> – to be<br>
understood as the (predicted) probability of a pixel being in the<br>
“1” (crack) class – by passing it though a <code>sigmoid()</code>.  You would<br>
typically convert a prediction probability to a yes-no prediction by<br>
thresholding the probability – <code>p &gt; 0.5</code> means class-“1”.</p>
<p>Because <code>sigmoid()</code> maps a logit of <code>0.0</code> to a probability of <code>0.5</code>,<br>
you can simply compare your logits (the output of your model)<br>
with a threshold of <code>0.0</code>.  These are your yes-no predictions for<br>
your pixels, and your accuracy is typically taken to be the fraction<br>
of pixels for which these prediction are correct, i.e., are equal to<br>
your mask values.</p>
<p>One last comment:  It may be the case that you have many more<br>
“background” pixels than “crack” pixels.  If so, you might want to<br>
experiment with using a class weight in your loss function.  (Look at<br>
the <code>pos_weight</code> argument to <code>BCEWithLogitsLoss</code> constructor.)<br>
But you should try things first with an unweighted loss function just<br>
to see how well the plain-vanilla approach.</p>
<p>Best.</p>
<p>K. Frank</p>

username: RyanPatterson
cooked: <p>I’m having trouble using the input image with shape of (batchsize, image_height, image_width). I’m getting “RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1 3 3, but got 3-dimensional input of size [1, 256, 256] instead”. Probably something simple but I cant see it.</p>
<p>I deleted these two lines</p>
<p>image = np.reshape(image, (1, image_height, image_width))<br>
out = output.squeeze(1)</p>
<p>And changed this line</p>
<p>loss = loss_function(out, y)</p>
<p>To</p>
<p>loss = loss_function(output, y)</p>
<p>This is most of my code.</p>
<pre><code class="lang-auto">
k_size = 3
pad = 1
class UNetMini(Module):

    def __init__(self):
        super(UNetMini, self).__init__()

        self.block1 = Sequential(
            Conv2d(1, 32, kernel_size=k_size, padding=pad),
            ReLU(),
            Dropout2d(0.2),
            Conv2d(32, 32, kernel_size=k_size, padding=pad),
            ReLU(),
        )
        self.pool1 = MaxPool2d((2, 2))

        self.block2 = Sequential(
            Conv2d(32, 64, kernel_size=k_size, padding=pad),
            ReLU(),
            Dropout2d(0.2),
            Conv2d(64, 64, kernel_size=k_size, padding=pad),
            ReLU(),
        )
        self.pool2 = MaxPool2d((2, 2))

        self.block3 = Sequential(
            Conv2d(64, 128, kernel_size=k_size, padding=pad),
            ReLU(),
            Dropout2d(0.2),
            Conv2d(128, 128, kernel_size=k_size, padding=pad),
            ReLU()
        )

        self.up1 = UpsamplingNearest2d(scale_factor=2)
        self.block4 = Sequential(
            Conv2d(192, 64, kernel_size=k_size, padding=pad),
            ReLU(),
            Dropout2d(0.2),
            Conv2d(64, 64, kernel_size=k_size, padding=pad),
            ReLU()
        )

        self.up2 = UpsamplingNearest2d(scale_factor=2)
        self.block5 = Sequential(
            Conv2d(96, 32, kernel_size=k_size, padding=pad),
            ReLU(),
            Dropout2d(0.2),
            Conv2d(32, 32, kernel_size=k_size, padding=pad),
            ReLU()
        )

        self.conv2d = Conv2d(32, 1, kernel_size=pad)

    def forward(self, x):
        out1 = self.block1(x)
        out_pool1 = self.pool1(out1)

        out2 = self.block2(out_pool1)
        out_pool2 = self.pool1(out2)

        out3 = self.block3(out_pool2)

        out_up1 = self.up1(out3)
        # return out_up1
        out4 = torch.cat((out_up1, out2), dim=1)
        out4 = self.block4(out4)

        out_up2 = self.up2(out4)
        out5 = torch.cat((out_up2, out1), dim=1)
        out5 = self.block5(out5)

        out = self.conv2d(out5)

        return out


if rebuild_data:
    data = BuildData()
    data.make_training_data()

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

processed_images = np.load("/home/ryan/CrackProject/DataSets/Data/images.npy", allow_pickle=True)
processed_masks = np.load("/home/ryan/CrackProject/DataSets/Data/masks.npy", allow_pickle=True)


model = UNetMini().to(device)


# This builds the data set
class FormsDataset(Dataset):
    def __init__(self, images, masks, transforms):
        self.images = images
        self.masks = masks
        self.transforms = transforms

    def __getitem__(self, idx):
        image = self.images[idx]
        image = image / 255
        image = np.reshape(image, (1, image_height, image_width)) 
        if self.transforms:
            image = self.transforms(image)

        mask = self.masks[idx]
        mask[mask &gt; .7] = 1
        mask[mask &lt;= .7] = 0
        if self.transforms:
            mask = self.transforms(mask)
        return image, mask

    def __len__(self):
        return len(self.images)


train_dataset = FormsDataset(processed_images, processed_masks, trans)
train_data_loader = DataLoader(train_dataset, batch_size=b_size, shuffle=False)    

# Training loop
total_steps = len(train_data_loader)
print(f'Train dataset has {len(train_data_loader)} batches of size {b_size}')

# Loss and Optimization
loss_function = nn.BCEWithLogitsLoss(reduction='none')
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
initial = torch.rand([b_size, image_height, image_width]).to(device)


def fwd_pass(X, y):
    if train:
        model.zero_grad()       # Clears the gradients from the last step, otherwise they would accumulate
    output = model(X)
    out = output.squeeze(1)
    loss = loss_function(out, y)
    acc = []
    if train:
        loss.backward(gradient=initial)  # comp derivative of the loss w.r.t.the parameters(or other req grad) backprop.
        optimizer.step()
    return acc, loss, output


def train():
    total_steps = len(train_data_loader)
    print(f"{epochs} epochs, {total_steps} total_steps per epoch")

    for epoch in tqdm(range(epochs), desc="Epochs"):
        for i, (images, masks) in enumerate(train_data_loader):
            images = images.type(torch.FloatTensor)
            images = images.to(device)

            masks = masks.type(torch.FloatTensor)
            masks = masks.to(device)

            acc, loss, output = fwd_pass(images, masks)

train()
</code></pre>

username: KFrank
cooked: <p>Hello Ryan!</p>
<aside class="quote no-group" data-username="RyanPatterson" data-post="3" data-topic="81385" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/58f4c7/48.png" class="avatar"> RyanPatterson:</div>
<blockquote>
<p>I’m having trouble using the input image with shape of (batchsize, image_height, image_width). I’m getting “RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1 3 3, but got 3-dimensional input of size [1, 256, 256] instead”.</p>
</blockquote>
</aside>
<p>Since you’re using a version of “UNet” and the first layer of your<br>
model is:</p>
<pre data-code-wrap="python"><code class="lang-python">Conv2d(1, 32, kernel_size=k_size, padding=pad)
</code></pre>
<p>it looks like your model is expecting inputs of shape<br>
<code>[nBatch, nChannel, height, width]</code>, where <code>nChannel = 1</code>.<br>
This makes sense for a grayscale image.  (If your model were set up<br>
for color images, you would probably have <code>nChannel = 3</code> for the<br>
three rgb channels.)</p>
<aside class="quote no-group">
<blockquote>
<p>I deleted these two lines</p>
<p>image = np.reshape(image, (1, image_height, image_width))<br>
out = output.squeeze(1)</p>
</blockquote>
</aside>
<p>What was the shape of a <em>batch</em> of images to input to your model<br>
before you did any reshaping, etc.?  What was the shape of the<br>
output of your model before reshaping?  What is the shape of a<br>
batch of masks that you pass to your loss function?</p>
<p>The point is that “UNet” typically carries along a “channel”<br>
dimension, which, in your case, appears to be of size 1 for both<br>
the input and output of your model.</p>
<p><code>BCEWithLogitsLoss</code> requires that the shape of its <code>input</code> (the<br>
output of your model) and its <code>target</code> (your mask) be the same.<br>
They can both have this “singleton” <code>nChannel = 1</code> dimension,<br>
but, if so, they both have to have it.</p>
<aside class="quote no-group">
<blockquote>
<p>loss_function = nn.BCEWithLogitsLoss(reduction=‘none’)</p>
</blockquote>
</aside>
<p>This looks wrong.  <code>reduction='none'</code> means don’t sum (or<br>
average) the loss over the elements of <code>output</code> (and <code>target</code>).<br>
But, for backpropagation, you want a single scalar loss, so you<br>
should use the default <code>reduction = 'mean'</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: RyanPatterson
cooked: <p>Thanks Frank. I finally got it working. My data set is still very small, around 100, so I’m guessing it is over fitting the data. Does anything stand out to you in the graphs.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/4/0444f32cb910cd7bcc06fa6f4392fdf72f1d1839.png" alt="Figure_1" data-base62-sha1="BLDkuDUoXjdfNVLF4KAuCJU55v" width="640" height="480"></p>