Dataloader: Batch then shuffle

username: RahimEntezari
cooked: <p>I want to change the order of shuffle and batch. Normally, when using the dataloader, the data is shuffles and then we batch the shuffled data:</p>
<pre><code class="lang-auto">import torch, torch.nn as nn
from torch.utils.data import DataLoader

x = DataLoader(torch.arange(10), batch_size=2, shuffle=True)


print(list(x))

batch [tensor(7), tensor(9)]
batch [tensor(4), tensor(2)]
batch [tensor(5), tensor(3)]
batch [tensor(0), tensor(8)]
batch [tensor(6), tensor(1)]

</code></pre>
<p>what I want is first batch then shuffle. One example output is as following:<br>
batch [tensor(6), tensor(7)]<br>
batch [tensor(0), tensor(1)]<br>
batch [tensor(2), tensor(3)]<br>
batch [tensor(8), tensor(9)]<br>
batch [tensor(4), tensor(5)]</p>

username: ptrblck
cooked: <p>Based on your description it seems you would like to return shuffled pairs of data.<br>
If that’s the case, I think the easiest way would be to return the pairs in <code>Dataset.__getitem__</code> and reduce the length of the <code>Dataset</code> by <code>2x</code>.<br>
Let me know, if this would work for you.</p>

username: RahimEntezari
cooked: <p>Thanks for your suggestion. I guess what I did is similar to what you suggested.<br>
Let me explain what I want to do in more detail.<br>
I have CIFAR10 and ordered all samples based on a hardness measure (Curriculum). Then I want to load every 100 from the dataset, in order.<br>
This is what I did:</p>
<ol>
<li>change the dataset of (60000, 32, 32, 2) to (600, 100, 32, 32, 3)</li>
<li>write my custom dataloader</li>
<li>load every 100 one time and change the batch size for the loader to 1.</li>
</ol>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, data, targets, transform=None):
        self.data = data
        self.targets = torch.LongTensor(targets)
        self.transform = transform

    def __getitem__(self, index):
        x = self.data[index]
        y = self.targets[index]

        if self.transform:
            x = np.zeros((100, 3, 32, 32))
            for k in range(self.data[index].shape[0]):
                x[k] = self.transform((255.0 * self.data[index][k]).astype(np.uint8))
        return x, y

    def __len__(self):
        return len(self.data)

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[255.0* 0.229, 255.0* 0.224, 255.0*0.225])

transform = transforms.Compose([transforms.ToPILImage(),
                                transforms.RandomHorizontalFlip(),
                                transforms.RandomCrop(32, 4),
                                transforms.ToTensor(),
                                normalize])

features_list = []
labels_list = []
for i in range(500):
    a = [features[i * 100:((i + 1) * 100)]]
    features_list.append(a[0])
    b = [labels[i * 100:((i + 1) * 100)]]
    labels_list.append(b[0])

dataset = MyDataset(np.asarray(features_list), np.asarray(labels_list), transform=transform)
train_loader = DataLoader(dataset, batch_size=1, shuffle=True)



features_list = []
labels_list = []
for i in range(100):
    a = [features[50000 + (i * 100):50000 + ((i + 1) * 100)]]
    features_list.append(a[0])
    b = [labels[i * 100:((i + 1) * 100)]]
    labels_list.append(b[0])

dataset = MyDataset(np.asarray(features_list), np.asarray(labels_list), transform=transform)
val_loader = DataLoader(dataset, batch_size=1, shuffle=True)
</code></pre>
<p>What do you think of this?</p>

username: ptrblck
cooked: <p>The approach looks fine to me.<br>
A minor suggestion: you probably don’t need the <code>list</code> creation and <code>[0]</code> indexing here:</p>
<pre><code class="lang-python">a = [features[50000 + (i * 100):50000 + ((i + 1) * 100)]]
features_list.append(a[0])
</code></pre>