Weird sampling issue - SubsetRandomSampler is not shuffling my dataset

nabsabs: <p>Hi everyone! I’m working on a classification problem where I have a folder with images and the label is the folder name. I am using the following script to generate loaders but when I iterate through any of the two loaders, I do not get random samples. Rather, its all samples from the first class (observed by printing the <code>labels</code>).  The pytorch version is 0.4.1.post2 (Ubuntu).</p>
<pre><code class="lang-auto">def get_dataloaders(data_path, val_split, batch_size, shuffle=True):
    t = transforms.Compose([transforms.Resize((150,150)), 
                            transforms.ToTensor()])
    dataset = torchvision.datasets.ImageFolder(root=data_path,transform=t)

    dataset_size = len(dataset)
    indices = list(range(dataset_size))
    split = int(np.floor(val_split * dataset_size))
    
    if shuffle:
        np.random.seed(42)
        np.random.shuffle(indices)
      
    train_indices, val_indices = indices[split:], indices[:split]
    train_sampler = SubsetRandomSampler(train_indices)
    val_sampler = SubsetRandomSampler(val_indices)
        
    train_loader = DataLoader(dataset, batch_size=batch_size, sampler=train_sampler)
    val_loader = DataLoader(dataset, batch_size=batch_size, sampler=val_sampler)
    return train_loader, val_loader

train_loader, val_loader = get_dataloaders(data_path, val_split, batch_size,shuffle=True)
images, labels = next(iter(train_loader))
</code></pre>
<p>Am I doing this incorrectly? Strangely, on my Windows machine with pytorch version <code>0.4.1</code> all is fine and I see random samples when I print the labels.</p>
<p>Btw, the folder structure looks like this:</p>
<pre><code class="lang-auto">root
|_ class1
       |_ patient1
                 |_ img1.jpg  etc
|_ class2
       |_ patient1
                  |_ img1.jpg                  
</code></pre>

rasbt: <p>Just to make sure that this is not due to an old bug, can you upgrade to PyTorch 1.0.1 to see if this problem persists?</p>

nabsabs: <p>I just updated to torch <code>1.0.1.post2</code> and this is still occurring. Could you maybe try to recreate this error on your end?</p>

rasbt: <p>It’s kind of impossible to recreate this because the labels you return depend on the DataLoader – not sure how you implemented this.</p>

nabsabs: <p>The dataloader I use is standard from pytorch. You can import it with <code>torch.utils.data. DataLoader</code>.<br>
It’s possible to give this a try with some dummy data organised with the folder structure that I mentioned. The output I see is that it is the same class in each batch of images as I iterate with <code>next(iter(train_loader))</code>; if you see the same class label with my dataloader script on some other dataset, then it is the same issue.</p>

rasbt: <p>I see. I thought you were using a custom one.</p>

rasbt: <p>Seems to work for me though:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/21f6e006155d078495ae0eb76f1445a32f160539.png" alt="53%20PM" data-base62-sha1="4QsGpbDiCBn6WNGQdaxgCzqzM3n" width="413" height="213"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/1fe43288704b3720b501ab5c4284c0714447861c.png" data-download-href="https://discuss.pytorch.org/uploads/default/1fe43288704b3720b501ab5c4284c0714447861c" title="25%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1fe43288704b3720b501ab5c4284c0714447861c_2_690x439.png" alt="25%20PM" data-base62-sha1="4y7I9SizSO2Efe9DMQ3QYLbps2E" width="690" height="439" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/1fe43288704b3720b501ab5c4284c0714447861c_2_690x439.png, https://discuss.pytorch.org/uploads/default/original/2X/1/1fe43288704b3720b501ab5c4284c0714447861c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/1fe43288704b3720b501ab5c4284c0714447861c.png 2x" data-dominant-color="ECEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">25%20PM</span><span class="informations">1034×659 95.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

nabsabs: <p>Thanks so much for your efforts!<br>
This is really strange - my setup is Ubuntu 16.04, Python3.5, torch 1.0.1.post2. Could you share your setup details? I’ve tried this on 3 other different machines with the same results x_x</p>

rasbt: <p>Also using Ubuntu, but 18.04 and Python 3.7. I am also using torch 1.0.1.post2</p>

nabsabs: <p>Okay - I’m in the processing of creating some new environments so will update you in a few. Could you  lastly share the full call to the <code>get_dataloaders</code> function? It seems you are not using the <code>val_split</code> command which is the whole point of me using <code>SubsetRandomSampler</code>.</p>

rasbt: <p>I tried it on both my laptop (screenshot from above) and on my Ubuntu machine (which has the cuda version of PyTorch). It seems to work in both cases. Regarding the val_loader, just enabled it:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8f231939739a3fcad46e10d5285642daa664c1a0.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f231939739a3fcad46e10d5285642daa664c1a0" title="36%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8f231939739a3fcad46e10d5285642daa664c1a0_2_515x500.png" alt="36%20PM" data-base62-sha1="kqftxNqkmIu37U1bl7iUSbH697G" width="515" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8f231939739a3fcad46e10d5285642daa664c1a0_2_515x500.png, https://discuss.pytorch.org/uploads/default/original/2X/8/8f231939739a3fcad46e10d5285642daa664c1a0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/8/8f231939739a3fcad46e10d5285642daa664c1a0.png 2x" data-dominant-color="EDECED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">36%20PM</span><span class="informations">708×687 73.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

rasbt: <p>Oh, maybe the issue is with your dataset naming, can you try to use class_1 instead of class1 etc?</p>

nabsabs: <p>my class names are actually <code>AD</code>, <code>MCI</code> and <code>Normal</code></p>

rasbt: <p>yeah, even if I change my class names to A, B, … it doesn’t seem to make a difference. Weird. If you cannot resolve this issue, how about going about this in a more classic way by defining the class labels you want in a CSV file associated with the file names? E.g., something like this: <a href="https://github.com/rasbt/stat479-deep-learning-ss19/blob/master/L09_mlp/code/custom-dataloader/custom-dataloader-example.ipynb" rel="nofollow noopener">https://github.com/rasbt/stat479-deep-learning-ss19/blob/master/L09_mlp/code/custom-dataloader/custom-dataloader-example.ipynb</a></p>

nabsabs: <p>hmm, this is super annoying! I think I will have to resort to making my own <code>Dataset</code> class like your example but since <code>ImageFolder</code> was built for this, I was hoping to get away with inbuilt functionality.</p>

nabsabs: <p>I HAVE SOLVED MY STUPID ISSUE!<br>
my folder structure had an extra folder before the classes and so the entire dataset had only one label.<br>
I’ll be killing myself now - thanks a bunch <a class="mention" href="/u/rasbt">@rasbt</a></p>

rasbt: <p>Haha, glad to hear that this was solved :). Means that it’s an easy fix and no bug, that’s great <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p><code>np.random.shuffle</code> is done before the data splitting to create datasets with randomly shuffled input samples.<br>
The <code>DataLoader</code> itself makes sure that the sample of each batch are shuffled.</p>

theairbend3r: <p>Oh. Got it, thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>