Dataloader shuffle

username: emma_ng
cooked: <p>Hi, sorry for the naive questions, I’m just started learning.</p>
<p>I have a custom dataset to load image and its label, which is used with DataLoader with shuffle=True. My questions are:</p>
<ol>
<li>
<p>When DataLoader shuffle the batches, do they shuffle both images and labels or just images? Because the answer here <a href="https://stackoverflow.com/questions/65402802/pytorch-shuffle-dataloader?rq=1" rel="noopener nofollow ugc">https://stackoverflow.com/questions/65402802/pytorch-shuffle-dataloader?rq=1</a> is saying that only the images are shuffled, not the label.</p>
</li>
<li>
<p>I need to accumulate the prediction output across the whole epoch to compute some specific scores of that epoch. My code is as below, I use accuracy here for simplicity. Is this correct to simply accumulate prediction output across all batches of train_loader and compute the scores at the end of the epoch?</p>
</li>
</ol>
<p>Thank you</p>
<pre><code class="lang-auto">## Custom dataset class
class MyDataset(Dataset):
    def __init__(self, label_csv):
        self.label_df = pd.read_csv(label_csv)  # &lt;img_id&gt;, &lt;label&gt;

    def __len__(self):
        return len(self.label_df)

    def __getitem__(self, idx):
        img_id, label = self.label_df.iloc[idx]

        img = read_and_preprocess_image(img_id)

        return img, label

## Create datasets and dataloaders
training_data = MyDataset(train_label_csv)
train_dataloader = DataLoader(training_data, batch_size=batch_size, shuffle=True, num_workers=8)
validation_data = MyDataset(val_label_csv)
val_dataloader = DataLoader(validation_data, batch_size=batch_size, shuffle=True, num_workers=8)

## Loop through all epochs
for epoch in range(num_epoch):
    running_loss = 0.0
    pred_epoch = []
    label_epoch = []

    for inputs, labels in train_loader:
        inputs = inputs.to(device)
        labels = labels.to(device)

        # Zero the parameter gradients
        optimizer.zero_grad()

        # Make prediction
        outputs = model(inputs)

        _, preds = torch.max(outputs, 1)
        
        # Compute loss
        loss = loss_fn(outputs, labels)
        loss.backward()

        optimizer.step()

        # Accumulate labels
        running_loss += loss.detach() * inputs.size(0)
        pred_epoch.extend(preds.tolist())
        label_epoch.extend(labels.tolist())

    epoch_loss = running_loss / train_size
    epoch_correct = sum(pred_epoch[i] == label_epoch[i] for i in range(len(pred_epoch)))
    epoch_acc = epoch_correct / train_size
</code></pre>

username: ptrblck
cooked: <aside class="quote no-group quote-modified" data-username="emma_ng" data-post="1" data-topic="164996">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/b4bc9f/48.png" class="avatar"> emma_ng:</div>
<blockquote>
<p>Because the answer here <a href="https://stackoverflow.com/questions/65402802/pytorch-shuffle-dataloader?rq=1" class="inline-onebox">PyTorch: Shuffle DataLoader - Stack Overflow</a> is saying that only the images are shuffled, not the label.</p>
</blockquote>
</aside>
<p>That’s not completely right, since the answer correctly pointed out that the shuffled predictions (created via shuffling the inputs via the <code>DataLoader</code>) are compared to the unshuffled targets (which were not created by the <code>DataLoader</code>):</p>
<pre data-code-wrap="python"><code class="lang-python">train_accuracy = sum(train_preds.argmax(axis=1) == y_train)/len(y_train)
</code></pre>
<p><code>y_train</code> is a global tensor, which was not shuffled, while <code>train_preds</code> was shuffled, so the answer is correct.</p>
<p>Now to your question: the <code>DataLoader</code> will shuffle both assuming the data <em>and</em> target are created in the <code>Dataset.__getitem__</code> via the passed <code>index</code> (which is the default use case).</p>
<ol start="2">
<li>Your code looks correct assuming <code>train_size</code> is the length of the <code>Dataset</code>.</li>
</ol>

username: emma_ng
cooked: <p>Oh, I see. Thank you for the quick response.<br>
I missed the part about <code>y_train</code> that you mentioned.<br>
Thank you so much.</p>