Error in python’s multiprocessing library

Neda: <p>This is the Unet architecture for the customised dataset for semantic segmentation based on <a href="https://arxiv.org/pdf/1505.04597.pdf" rel="nofollow noopener">this paper</a>. I am getting this error when the num_worker = 2</p>
<pre><code class="lang-auto">`
  File "C:\Users\Neda\Anaconda3\lib\multiprocessing\spawn.py", line 172, in get_preparation_data
    main_mod_name = getattr(main_module.__spec__, "name", None)

AttributeError: module '__main__' has no attribute '__spec__'`

</code></pre>
<p>while with the num_worker = 0, I will get this error</p>
<pre><code class="lang-auto">raise TypeError((error_msg.format(type(batch[0]))))

TypeError: batch must contain tensors, numbers, dicts or lists; found &lt;class 'PIL.TiffImagePlugin.TiffImageFile'&gt;
</code></pre>
<p>The output of print(device)  is cuda:0  I don’t have any clue where I am doing wrong. Thank you in advance.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data.dataset import Dataset  # For custom data-sets
import torchvision.transforms as transforms
from PIL import Image
import glob


print(torch.version)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device)


# get all the image and mask path and number of images
folder_data = glob.glob("D:\\Neda\\Pytorch\\U-net\\BMMCdata\\data\\*.tif")
folder_mask = glob.glob("D:\\Neda\\Pytorch\\U-net\\BMMCmasks\\masks\\*.tif")

# split these path using a certain percentage
len_data = len(folder_data)
print(len_data)
train_size = 0.6

train_image_paths = folder_data[:int(len_data*train_size)]
# print(train_image_paths) # output is 25 image for train
test_image_paths = folder_data[int(len_data*train_size):]
#print(test_image_paths) # output is 18 image for test

train_mask_paths = folder_mask[:int(len_data*train_size)]
test_mask_paths = folder_mask[int(len_data*train_size):]


class CustomDataset(Dataset):
    def __init__(self, image_paths, target_paths):   # initial logic happens like transform

        self.image_paths = image_paths
        self.target_paths = target_paths
        self.transforms = transforms.ToTensor()

    def __getitem__(self, index):

        image = Image.open(self.image_paths[index])
        mask = Image.open(self.target_paths[index])
        t_image = self.transforms(image)
        return t_image, mask

    def __len__(self):  # return count of sample we have

        return len(self.image_paths)


train_dataset = CustomDataset(train_image_paths, train_mask_paths)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=4, shuffle=True, num_workers=2)

test_dataset = CustomDataset(test_image_paths, test_mask_paths)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=4, shuffle=False, num_workers=2)


class ConvRelu(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, padding, stride):
        super(ConvRelu, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, kernel_size=kernel_size,
                             padding=padding, stride=stride)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.conv(x)
        x = self.relu(x)
        return x


class UNet(nn.Module):
    def __init__(self):
        super(UNet, self).__init__()
        #channels, height, width = in_shape

        self.down1 = nn.Sequential(
            ConvRelu(1, 64, kernel_size=(3, 3), stride=1, padding=0),
            ConvRelu(64, 64, kernel_size=(3, 3), stride=1, padding=0)
            )
        self.maxPool1 = nn.MaxPool2d(kernel_size=(2, 2), stride=2)

        self.down2 = nn.Sequential(
            ConvRelu(64, 128, kernel_size=(3, 3), stride=1, padding=0),
            ConvRelu(128, 128, kernel_size=(3, 3), stride=1, padding=0)
            )
        self.maxPool2 = nn.MaxPool2d(kernel_size=(2, 2), stride=2)

        self.down3 = nn.Sequential(
            ConvRelu(128, 256, kernel_size=(3, 3), stride=1, padding=0),
            ConvRelu(256, 256, kernel_size=(3, 3), stride=1, padding=0)
        )
        self.maxPool3 = nn.MaxPool2d(kernel_size=(2, 2), stride=2)

        self.down4 = nn.Sequential(
            ConvRelu(256, 512, kernel_size=(3, 3), stride=1, padding=0),
            ConvRelu(512, 512, kernel_size=(3, 3), stride=1, padding=0)
        )

        self.maxPool4 = nn.MaxPool2d(kernel_size=(2, 2), stride=2)

        self.center = nn.Sequential(
            ConvRelu(512, 1024, kernel_size=(3, 3), stride=1, padding=0),
            ConvRelu(1024, 1024, kernel_size=(3, 3), stride=1, padding=0)
        )
        self.upSample1 = nn.ConvTranspose2d(1024, 1024, 2, stride=2)

        self.up1 = nn.Sequential(
            ConvRelu(1024, 512, kernel_size=(2, 2), stride=1, padding=0),
            ConvRelu(512, 512, kernel_size=(2, 2), stride=1, padding=0)
        )

        self.upSample2 = nn.ConvTranspose2d(512, 512, 2, stride=2)

        self.up2 = nn.Sequential(
            ConvRelu(512, 256, kernel_size=(2, 2), stride=1, padding=0),
            ConvRelu(256, 256, kernel_size=(2, 2), stride=1, padding=0)
        )

        self.upSample3 = nn.ConvTranspose2d(256, 256, 2, stride=2)

        self.up3 = nn.Sequential(
            ConvRelu(256, 128, kernel_size=(2, 2), stride=1, padding=0),
            ConvRelu(128, 128, kernel_size=(2, 2), stride=1, padding=0)
        )

        self.upSample4 = nn.ConvTranspose2d(128, 128, 2, stride=2)

        self.up4 = nn.Sequential(
            ConvRelu(128, 64, kernel_size=(2, 2), stride=1, padding=0),
        )
        # 1x1 convolution at the last layer
        self.output_seg_map = nn.Conv2d(64, 2, kernel_size=(1, 1), padding=0, stride=1)

    def crop_concat(self, upsampled, bypass, crop=False):
        if crop:
            c = (bypass.size()[2] - upsampled.size()[2]) // 2
            # -c is amount of pad which will add on each side for all dimension
            bypass = F.pad(bypass, (-c, -c, -c, -c))  # (padLeft, padRight, padTop, padBottom)
        return torch.cat((upsampled, bypass), 1)

    def forward(self, x):
        x = self.down1(x)
        out_down1 = x

        x = self.down2(x)
        out_down2 = x
        x = self.maxPool2(x)

        x = self.down3(x)
        out_down3 = x
        x = self.maxPool3(x)

        x = self.down4(x)
        out_down4 = x
        x = self.maxPool4(x)

        x = self.center(x)

        x = self.upSample1(x)
        x = self.up1(x)
        self.crop_concat(x, out_down4)

        x = self.upSample2(x)
        x = self.up2(x)
        self.crop_concat(x, out_down3)

        x = self.upSample3(x)
        x = self.up3(x)
        self.crop_concat(x, out_down2)

        x = self.upSample4(x)
        x = self.up4(x)
        self.crop_concat(x, out_down1)

        out = self.output_seg_map(x)

        return F.log_softmax(self.output_seg_map(out))  # applies log-softmax on last layer


net = UNet()
net = net.to(device)

criterion = nn.NLLLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.99)

def main():
    for epoch in range(2):  # loop over the dataset multiple times

        running_loss = 0.0
        for i, data in enumerate(train_loader, 0):
            # get the inputs
            t_image, mask = data
            t_image, mask = t_image.to(device), mask.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

           # forward + backward + optimize
            outputs = net(t_image)
            loss = criterion(outputs, mask)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.item()
            if i % 2000 == 1999:    # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0

print('Finished Training')

if __name__=='__main__':
    main()

</code></pre>

ptrblck: <p>The error is most likely thrown because your mask is still a <code>PIL.Image</code> instead of a tensor.<br>
I guess using <code>transforms.ToTensor()</code> wouldn’t be the best idea in this case, as you are dealing with a semantic segmentation task, i.e. your mask should contain class indices for all pixels.</p>
<p>Does you current mask hold a specific color code so that we could use it to create a mask tensor with class indices?</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> when I close the Spyder and run the code again I got this error <code>raise TypeError((error_msg.format(type(batch[0])))) TypeError: batch must contain tensors, numbers, dicts or lists; found &lt;class 'PIL.TiffImagePlugin.TiffImageFile'&gt;</code><br>
I guess should be about a mask that didn’t transform to tensor.</p>
<p>I did check the masks. They include mainly 85 and 170 value, and also some of them have 85,170, and 255 value. <a href="https://universityoflincoln-my.sharepoint.com/:i:/r/personal/nazarmehr_lincoln_ac_uk/Documents/BMMC_1.tif_segmentation.tifnomalized.tif?csf=1&amp;e=8ZBXsQ" rel="nofollow noopener">The mask looks like this</a>. How can I create a mask tensor with class indices?</p>

ptrblck: <p>Yeah, I think the mask is the problem as described earlier.<br>
Could you upload one sample somewhere else? I need some account to download the image from your university.</p>
<p>I guess you are currently loading a “normal” image using a specific color code for certain classes, so that we would need a mapping to decode them to class indices.</p>
<p>Which dataset are you using? Did you create it yourself?</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> yes, at the moment I am loading a normal image. Currently, I am using the TEM dataset in <a href="https://lmb.informatik.uni-freiburg.de/resources/datasets/tem.en.html" rel="nofollow noopener">this link</a>. I downloaded the BMMC dataset and BMMC segmentation as a mask. For now, I am using this dataset to understand how should I solve segmentation task. Possibly my data will have a class of 0 and 1. 0 is background and 1 is left ventricle.</p>
<p>My dataset will be different. It’s cardiac ultrasound images and mask of the left ventricle that needs to be annotated by a cardiologist.</p>

ptrblck: <p>Thanks for the link!<br>
I’ve downloaded the BMMC dataset with the masks and it seems three different color codes are used for the masks as you already explained.<br>
This code should load masks and convert these values to class indices:</p>
<pre><code class="lang-python">class CustomDataset(Dataset):
    def __init__(self, image_paths, target_paths):   # initial logic happens like transform
        self.image_paths = image_paths
        self.target_paths = target_paths
        self.transforms = transforms.ToTensor()
        self.mapping = {
            85: 0,
            170: 1,
            255: 2
        }
    
    def mask_to_class(self, mask):
        for k in self.mapping:
            mask[mask==k] = self.mapping[k]
        return mask

    def __getitem__(self, index):
        image = Image.open(self.image_paths[index])
        mask = Image.open(self.target_paths[index])
        t_image = self.transforms(image)
        mask = torch.from_numpy(np.array(mask))
        mask = self.mask_to_class(mask)
        return t_image, mask

    def __len__(self):  # return count of sample we have
        return len(self.image_paths)


train_dataset = CustomDataset(train_image_paths, train_mask_paths)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=4, shuffle=True, num_workers=2)

for data, target in train_loader:
    print(torch.unique(target))
</code></pre>
<p>You can of course change the mapping, e.g. if the pixel value 85 should be mapped to another class index.<br>
Just make sure your classes start with 0 and end with <code>num_classes-1</code> in case you would like to use a classification criterion like <code>nn.CrossEntropyLoss</code>.</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Thanks a lot for the effort you put in this forum. I really appreciate it. At the moment I am getting CUDA error: out of memory. I will sort out memory issue and will update you.</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> my dedicated video memory is 8192 MB. Shouldn’t be enough to train this UNet model? Although image size is really big 1024*1024</p>

vmirly1: <p>What’s the batch_size? You may need to use a smaller batch_size.</p>

Neda: <p><a class="mention" href="/u/vmirly1">@vmirly1</a> bach_size is 4. I tried with smaller and got out of memory error.</p>

vmirly1: <p>O I see!</p>
<p>I have never worked with UNet. Do you really need to work with such big image sizes? At least you can downsample the input images for debugging purposes.</p>

ptrblck: <p>1024 is pretty large for this kind of model.<br>
In the original paper the authors also used a sliding window approach to get their model working.</p>
<p>I tried to debug the model as <a class="mention" href="/u/vmirly1">@vmirly1</a> suggested, however, there seem to be a few issues in your code.<br>
If I resize the images to <code>512</code> or <code>256</code>, I get a size mismatch error even if I use <code>crop=True</code> in the <code>crop_concat</code> layers. I fixed it with an ugly hack to get it working.<br>
However, I also realized that you are actually never using the concatenated output of the <code>crop_conat</code> layers.<br>
If you assign the result back to <code>x</code>, you’ll get another error, since the number on input channels won’t match for your conv layers.</p>
<p>If I use <a href="https://github.com/ptrblck/pytorch_misc/blob/master/unet_demo.py">my implementation</a> as a drop-in replacement for your model, I can get the model working with a batch size of 1 for the original image shapes using approx. 7GB of GPU memory.<br>
Note that the implementations differ a bit, e.g. your model seem to have an additional layer in the bottleneck, so that will most likely explain the difference in memory usage.</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for spending time and explain to me about issues in my code. Does your unet model works for images with 1 channel? I think your dataset was RGB when you have written this model. what is in_channels_skip in your model?</p>

ptrblck: <p>Yes, you can pass the number of input channels using <code>in_channels</code>.<br>
This should work for your dataset:</p>
<pre><code class="lang-python">model = UNet(in_channels=1,
             out_channels=64,
             n_class=3,
             kernel_size=3,
             padding=1,
             stride=1)
</code></pre>
<p><code>in_channels_skip</code> is the number of channels from the block in the downward path which should be concatenated with the other input to the current up block.</p>
<p>So lets take <code>up3</code> as an example.<br>
This block takes an activation from <code>down3</code> as its input (<code>in_channels = 8 * out_channels</code> ) and additionally concatenates it with the output of <code>down2</code> (<code>in_channels_skip = 4 * out_channels</code> ).<br>
So you’ll have <code>12 * out_channels</code> channels for your conv layer in <code>up3</code> now, which is why it’s defined in this way:</p>
<pre><code class="lang-python">self.conv_block = BaseConv(
    in_channels=in_channels + in_channels_skip,
...
</code></pre>
<p>Does this example make it clear in any way or do you need more information?</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks a lot for example. Before I make sure I understand this, I have two more questions. One is what<br>
is the second in_channels in self.conv_trans1?</p>
<pre><code class="lang-auto">class UpConv(nn.Module):
    def __init__ (self, in_channels, in_channels_skip, out_channels, kernel_size, padding, stride):
        super(UpConv, self).__init__()
        self.conv_trans1 = nn.ConvTranspose2d(in_channels, in_channels, kernel_size=2, padding=0, stride=2)
</code></pre>
<p>and also why you did <code>2 * out_channels in self.down1 layer in UNet class</code>?<br>
sorry, I am asking a lot of question. I am follwoing your model before I go to debug mine.</p>

ptrblck: <p>The transposed conv layer just doubles the spatial size without changing the number of channels, so that is why I’m passing <code>in_channels</code> twice. I realize that the naming of my variables is really bad, sorry about that.</p>
<p>I’m basically multiply the number of output channels by a power of 2 for each down conv, i.e. <code>2*out_channels, 4*out_channels, 8*out_channels</code>.<br>
You might want to change the multiplicative factor, if you want to increase the number of channels. <code>out_channels</code> is passed to the <code>__init__</code> of the model.</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks a lot for clarifying on this. Do you have any figure for your model like fig 1 in <a href="https://arxiv.org/pdf/1505.04597.pdf" rel="nofollow noopener">this paper</a>?  Just wanted to know did you written this model based on a research paper?</p>

ptrblck: <p>No, unfortunately I didn’t create such a nice figure for this implementation.<br>
Yes, I used UNet as the base model and have rewritten it a bit to work with differently shaped inputs and also to output the same spatial size (instead of a cropped version like in the original implementation).</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> That’s great. Does it mean in your model the size of output will be same as input size? I can see you used Adam as an optimizer and in the paper which I sent earlier, they used SGD. Any specific reason that you chose this as an optimizer?</p>

ptrblck: <p>Yes, the output size should match the input size, which makes it a bit easier for segmentation tasks, especially if you have valid classes near the borders.</p>
<p>Yeah, just my default optimizer based on my biased decisions for a good baseline. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>