Dataloader just shuffles the order of batches or does it also shuffle the images in each batch?

username: zimmer550
cooked: <p>Sorry for asking this basic question but I think I was always under the impression that Dataloader shuffle just reorders the batches without doing changing the order of the images. So for example, my batch size is 2 and my images are: 0, 1, 2, 3, 4, 5, 6, 7</p>
<p>If I call the dataloader with shuffle set to true, I get the following batches: [0, 1], [2, 3], [4, 5], [6, 7] and then the order of these batches is changed, so, in the end, I could get something like: [2, 3], [0, 1], [6, 7], [4, 5]</p>
<p>Is this how shuffle works in dataloader or is the order of the images changed entirely (meaning the order of the images changes to 3, 4, 7, 0, 1, 5, 2, 6) and then they are converted to batches?</p>

username: albanD
cooked: <p>HI,</p>
<p>It should shuffle all the images and then give you batches of shuffled images. So it should shuffle images in each batch.<br>
Can you give me a small code sample that reproduce what you observer please?</p>

username: zimmer550
cooked: <p>I guess I had the wrong assumption then. I thought that images were left untouched and only the batches were shuffled around. Also, what if I use a random sampler and then call the dataloader with the random sampler and shuffle set to false. Will I get the behavior where the images are not shuffled but the batches are?</p>

username: albanD
cooked: <p>Hi,</p>
<p>I don’t think we have a fixed-batched random sampler.<br>
It should be “easy” to do: you can have you dataset of size <code>real_size / batch_size</code>. Have your dataset return a whole batch when asked for a single index .Use a regular random sampler and a batch-size of 1 for the dataloader.</p>

username: zimmer550
cooked: <p>Then why do some people use the pytorch random sampler and use that with the dataloader instead of just setting the shuffle argument to true in the dataloader?</p>

username: albanD
cooked: <p>I don’t think there is any difference between the two <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
If you have some code reference, I can double check.</p>

username: zimmer550
cooked: <p>Thank you for your reply and for clearing the confusion. Actually, this is where I noticed this:<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/TengdaHan/DPC/blob/7bb73f4a934bbf34eda10a6102629bacb9b4e348/dpc/main.py#L303" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/TengdaHan/DPC/blob/7bb73f4a934bbf34eda10a6102629bacb9b4e348/dpc/main.py#L303" target="_blank" rel="nofollow noopener">TengdaHan/DPC/blob/7bb73f4a934bbf34eda10a6102629bacb9b4e348/dpc/main.py#L303</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="293" style="counter-reset: li-counter 292 ;">
<li>                          big=use_big_K400)</li>
<li>elif args.dataset == 'ucf101':</li>
<li>    dataset = UCF101_3d(mode=mode,</li>
<li>                     transform=transform,</li>
<li>                     seq_len=args.seq_len,</li>
<li>                     num_seq=args.num_seq,</li>
<li>                     downsample=args.ds)</li>
<li>else:</li>
<li>    raise ValueError('dataset not supported')</li>
<li>
</li>
<li class="selected">sampler = data.RandomSampler(dataset)</li>
<li>
</li>
<li>if mode == 'train':</li>
<li>    data_loader = data.DataLoader(dataset,</li>
<li>                                  batch_size=args.batch_size,</li>
<li>                                  sampler=sampler,</li>
<li>                                  shuffle=False,</li>
<li>                                  num_workers=32,</li>
<li>                                  pin_memory=True,</li>
<li>                                  drop_last=True)</li>
<li>elif mode == 'val':</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: albanD
cooked: <p>I have to admit, I’m not sure why it is done this way in this code. Maybe for historical reasons, he was trying other samplers before.<br>
You can double check the doc <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="nofollow noopener">here</a> to make sure what the behavior will be with shuffle or if you set the sampler.</p>

username: franciscocms
cooked: <p>Hello! I’m quite new here, but I think my question is related to this topic.<br>
When shuffle is True, it just kind of build batches with random indexes? Or it actually shuffles my dataset?<br>
If it shuffles the entire dataset, my labels tensor is shuffled the same way?<br>
I know it can be a stupid question, but as I am building my labels tensor inside my custom dataset class, I don’t know if shuffle can change the match I’m creating.<br>
Thank you!</p>

username: ptrblck
cooked: <p>If <code>shuffle=True</code> is set in the <code>DataLoader</code>, a <code>RandomSampler</code> will be used as seen in <a href="https://github.com/pytorch/pytorch/blob/e841bea46578472b01a1bb7631a14aede7e8b3ba/torch/utils/data/dataloader.py#L212-L213">these lines of code</a>.<br>
This sampler will create random indices and pass them to the <code>Dataset.__getitem__</code> method as seen <a href="https://github.com/pytorch/pytorch/blob/e841bea46578472b01a1bb7631a14aede7e8b3ba/torch/utils/data/sampler.py#L103-L107">here</a>.</p>
<p>Your data and target correspondence should thus hold, since the same <code>index</code> should be used to load these tensors.</p>

username: RahimEntezari
cooked: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="60900">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>real_size / batch_size</code></p>
</blockquote>
</aside>
<p>I was interested in your suggestion, but I do not know how to create such a dataset of of size  <code>real_size / batch_size</code></p>

username: albanD
cooked: <p>The idea is to have an extra dimension.<br>
In particular, if you use a TensorDataset, you want to change your Tensor from <code>real_size, ...</code> to <code>real_size / batch_size, batch_size, ...</code> and as for batch 1 from the Dataloader. That way you will get one batch of size <code>batch_size</code> every time. Note that you get an input of size <code>1, batch_size, ...</code> that you might want to reshape to remove the leading 1.</p>

username: B_Paudel
cooked: <p>Is there a way to access/return the random indices passed to Dataset.<strong>getitem</strong> method?<br>
Thank you</p>

username: ptrblck
cooked: <p>Yes, you can directly return them with the data:</p>
<pre><code class="lang-python">def __getitem__(self, index):
    x = self.data[index]
    y = self.target[index]
    return x, y, index
</code></pre>

username: Yuv
cooked: <p>I know it is an old post, but I came across the exact same problem.<br>
Can you give an example how to do what you suggested?<br>
I’m working on CIFAR for example - and I want to have certain order of images and that the dataloder will only shuffle between batches and not inside a batch.<br>
Thanks</p>