Segmentation Fault/Timeout using Dataloaders

pytorch_dummy: <p>Hi all,</p>
<p>I’m currently working on a Pix2Pix Gan and I’m running into unexpected problems on my Ubuntu Linux machine (24GB GPU + 16 core CPU).<br>
My dataset class does nothing else than loading images and masks of the disk. If a mask is not found, an empty one with a certain size is generated. Other parameters for the class are various transformation objects for augmenting the data.<br>
However, my problem is that after the Validation Dataloader has been run once, the Training Dataloader gets stuck and sometimes throws a segmentation error. I am using Pytorch 1.12.1 and the number of workers is 4 for the Training Dataloader and 0 for the Validation Dataloader.<br>
If I also set the number to 0 on the Training Dataloader everything works as it should.</p>
<pre><code class="lang-auto">init_transform_train, init_transform_test, transform_only_input, transform_only_target = get_transforms()

    # split between training and validation
    total_dataset = Pix2PixDataset(root_dir=args.data_dir, init_transform_train=init_transform_train,
                                       init_transform_test=init_transform_test,
                                       transform_only_input=transform_only_input,
                                       transform_only_target=transform_only_target,
                                       target_image_size=256)

    rs = ShuffleSplit(n_splits=1, test_size=.1, random_state=123)

    train_indices, val_indices = next(rs.split(total_dataset))
    train_dataset = torch.utils.data.Subset(total_dataset, train_indices)
    val_dataset = torch.utils.data.Subset(total_dataset, val_indices)
    train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True,
                              num_workers=4)
    val_loader = DataLoader(val_dataset, batch_size=1, shuffle=False)

    loop = tqdm(train_loader, leave=True)
    for idx, (x, y) in enumerate(loop):
        x, y = x.to(device)
    loop = tqdm(train_loader, leave=True)
    for idx, (x, y) in enumerate(loop):
        x, y = x.to(device)
    loop = tqdm(val_loader, leave=True)
    for idx, (x, y) in enumerate(loop):
        x, y = x.to(device)
    loop = tqdm(train_loader, leave=True) 
    for idx, (x, y) in enumerate(loop): # &lt;- seems to stuck here
        x, y = x.to(device)
</code></pre>
<p>As you can see in the image below, the 4th call is not executed. I am not using cv2 (which is sometimes mentioned as root cause for such problems) and for image loading I am using PIL.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c0d45eb305164fee2b370d1c7efa8693541b94a.png" data-download-href="https://discuss.pytorch.org/uploads/default/6c0d45eb305164fee2b370d1c7efa8693541b94a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c0d45eb305164fee2b370d1c7efa8693541b94a.png" alt="image" data-base62-sha1="fpS0gFAqL2GZaHG19dm86laboAa" width="690" height="36" data-dominant-color="858A85"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1741×91 4.23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>On my Windows machine I am not able to replicate the problem. Do you have an idea what might be a problem? When I am switching roles of the train and val loaders I get a similar problem (I am not allowed to post another media item but I hope that the following text might give the impression)<br>
Val-Loader successful<br>
Val-Loader successful<br>
Train-Loader stucks</p>
<p>Edit:<br>
The problem also occurs when training and validation indices are equivalent!</p>
<p>Edit2:<br>
After a some hours of debugging I might have identified the problem.<br>
One of my image transformations stops working:</p>
<pre><code class="lang-auto">import albumentations as A

init_transform_train = A.Compose(
        [
            A.Resize(width=default_config['TARGET_IMAGE_SIZE'], height=default_config['TARGET_IMAGE_SIZE']),
            A.PadIfNeeded(min_height=default_config['ZERO_PADDED_IMAGE_SIZE'],
                          min_width=default_config['ZERO_PADDED_IMAGE_SIZE'],
                          value=0,
                          p=1.0,
                          border_mode=cv2.BORDER_CONSTANT),
            A.HorizontalFlip(p=0.5),
            A.VerticalFlip(p=0.5),
            ToTensorV2()
        ], additional_targets={'image0': 'image'}
    )
</code></pre>
<p>I have no clue why but this transformation does not work when it is used in the second epoch for the first time. Might be a bug.</p>

pytorch_dummy: <p><a href="https://github.com/albumentations-team/albumentations/issues/1246" class="inline-onebox" rel="noopener nofollow ugc">Albumentations blocks CPU by spawning multiple threads · Issue #1246 · albumentations-team/albumentations · GitHub</a> has “solved” the problem</p>