Can't iterate splited Dataloader

username: Ramzes_III
cooked: <p>Hello, I’m complitely new in Pytorch and here is my trouble.</p>
<ol>
<li>I create a Dataset with <code>ImageFolder</code>
</li>
<li>Split indices with sklearn <code>train_test_split</code>
</li>
<li>Make 2 Subsets.</li>
<li>Make new MyDataset class like <a href="https://discuss.pytorch.org/t/torch-utils-data-dataset-random-split/32209/4">here</a> because I want make different transforms.</li>
<li>Create 2 DataLoaders</li>
<li>When I’m trying to take 1 sample from DataLoader(with <code>next(iter(DataLoader)</code>) nothing happens. Code runs endlessly. No errors, no warnings.</li>
</ol>
<p>Here is the code:</p>
<pre><code class="lang-auto"># make Dataset
train_data = datasets.ImageFolder(root=train_path)

# all classes list
all_classes = list(
    (x.parent.name for x in sorted(
        Path('train/simpsons_dataset/').rglob('*.jpg'))))

# split indices
train_indices, val_indices = train_test_split(
    range(len(train_data)),
    train_size = 0.8,
    stratify=all_classes,
    random_state=42)

# make Subsets
train_subset = Subset(train_data, train_indices)
val_subset = Subset(train_data, val_indices)

# Dataset class for Subsets
class MyDataset(Dataset):
    def __init__(self, subset, transform=None):

        self.subset = subset
        self.transform = transform
        
    def __getitem__(self, index):
        img, label = self.subset[index]
        if self.transform:
            img = self.transform(img)
        return img, label
        
    def __len__(self):
        return len(self.subset)

# make new Datasets
train_dataset = MyDataset(train_subset, data_transforms['train'])
val_dataset = MyDataset(val_subset, data_transforms['val'])

# make Dataloaders
train_dataloader = DataLoader(train_dataset, batch_size=64, num_workers=2, shuffle=True)
val_dataloader = DataLoader(val_dataset, batch_size=64, num_workers=2, shuffle=False)
train_dataloader.dataset.transform

#finally I'm trying to iterate my Dataloader
img, label = next(iter(train_dataloader))
print(f"Image shape: {img.shape} ")
print(f"Label shape: {label.shape}")
</code></pre>
<p>If I iterate <code>train_data = datasets.ImageFolder(root=train_path)</code> - everything is ok!<br>
Please help!</p>

username: ptrblck
cooked: <p>Based on your description it seems you might be running into issues with the <code>DataLoader</code> so I would recommend trying to iterate the <code>DataLoader</code> object directly to see if this would work (I would assume it would also hang) and then trying to isolate the issue further by e.g. setting the <code>num_workers</code> to <code>0</code>.</p>

username: Ramzes_III
cooked: <p><span class="mention">@ptrblck_de</span>,<br>
Thank you for helping. I’m not sure, that I understand You correct. What do yo mean here?</p>
<blockquote>
<p>trying to iterate the <code>DataLoader</code> object directly</p>
</blockquote>

username: ptrblck
cooked: <p>You’ve mentioned that calling:<br>
<code>next(iter(loader)</code> seems to hang so I would also be interested to double check if:</p>
<pre><code class="lang-python">for data in loader:
    ...
</code></pre>
<p>would also hang as I would assume that’s the case.</p>

username: Ramzes_III
cooked: <p>The same with:</p>
<pre><code class="lang-auto">for data in loader:
    ...
</code></pre>
<p>I have set <code>num_workers=0</code> and everything is working! Thank you so much!<br>
Can you please explain why so? Because before this I always set <code>num_workers=2</code> and haven’t problens with it,</p>