Re-permute SubsetRandomSampler at every epoch

omrjml: <p>Is there a way to re-permute the indices for SubsetRandomSampler at each epoch without re-initialising the DataLoader at each epoch? I am loading a large dataset so do not want to reload it at every epoch.</p>
<p>my dataloader function is:</p>
<pre><code class="lang-auto">def train_dataloader(args):
    train_dataset_file = "{0}/train_data_{1}.hdf5".format(args.locations["train_test_datadir"],args.region)
    
train_dataset = data_io.ConcatDataset("train",args.nlevs, train_dataset_file, args.locations['normaliser_loc'], args.batch_size, xvars=args.xvars,
             yvars=args.yvars, yvars2=args.yvars2, samples_frac=args.samples_fraction, data_frac=args.data_fraction, no_norm=args.no_norm)
    indices = list(range(train_dataset.__len__()))
    train_sampler = torch.utils.data.SubsetRandomSampler(indices)
    train_loader = torch.utils.data.DataLoader(train_dataset, batch_sampler=None, batch_size=None, sampler=train_sampler, shuffle=False)
    return train_loader
</code></pre>
<p>And I call it as follows:</p>
<pre><code class="lang-auto">def train_loop(model, loss_function, optimizer, scheduler, args):
  
    training_loss = []
    train_ldr = train_dataloader(args)
    validation_loss = []
    test_ldr = test_dataloader(args)
    
    for epoch in range(1, args.epochs + 1):
        ## Training
        train_loss = 0
        for batch_idx, batch in enumerate(train_ldr):
            # Sets the model into training mode
            # print(batch_idx)
            model.train()
</code></pre>

omrjml: <p>Looking further at this it seems the change here</p>
<aside class="onebox githubcommit">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/commit/0b90b9cdd36de3131e7b3868c71acc48170f3f1a" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Commit">
    <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M10.86 7c-.45-1.72-2-3-3.86-3-1.86 0-3.41 1.28-3.86 3H0v2h3.14c.45 1.72 2 3 3.86 3 1.86 0 3.41-1.28 3.86-3H14V7h-3.14zM7 10.2c-1.22 0-2.2-.98-2.2-2.2 0-1.22.98-2.2 2.2-2.2 1.22 0 2.2.98 2.2 2.2 0 1.22-.98 2.2-2.2 2.2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/commit/0b90b9cdd36de3131e7b3868c71acc48170f3f1a" target="_blank" rel="nofollow noopener">Allow shuffle when auto-batching disabled in DataLoader (#39865)</a>
    </h4>

    <div class="github-info">
      <div class="date">
        committed <span class="discourse-local-date" data-format="ll" data-date="2020-06-11" data-time="22:13:57" data-timezone="UTC">10:13PM - 11 Jun 20 UTC</span>
      </div>

      <div class="user">
        <a href="https://github.com/ydaiming" target="_blank" rel="nofollow noopener">
          <img alt="ydaiming" src="https://avatars1.githubusercontent.com/u/66369380?v=4" class="onebox-avatar-inline" width="20" height="20">
          ydaiming
        </a>
        
      </div>

      <div class="lines" title="changed 2 files with 24 additions and 11 deletions">
        <a href="https://github.com/pytorch/pytorch/commit/0b90b9cdd36de3131e7b3868c71acc48170f3f1a" target="_blank" rel="nofollow noopener">
          <span class="added">+24</span>
          <span class="removed">-11</span>
        </a>
      </div>
    </div>

  </div>
</div>


  <div class="github-row">
    <pre class="github-content" style="white-space: normal;">Summary:
Fix https://github.com/pytorch/pytorch/issues/35761
cc SsnL
Note: closed the other PR for this new branch.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/39865
Differential Revision: D22003612
Pulled By: ezyang
fbshipit-source-id: 26aecd1b298fe99d3924f4c8157cd6cae2561c7c</pre>
  </div>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>will allow me to use shuffle=True with batch_size=None and I will not need to use SubsetRandomSampler. I am just wondering if anyone can think of a way I can do this in the mean time? Thanks.</p>

ptrblck: <p>Do you want to shuffle the passed indices in the <code>SubsetRandomSampler</code> or would you like to shuffle the global indices again and create new subsets?<br>
In the former case the indices would already be reshuffled in each epoch as seen here:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        pass
    
    def __getitem__(self, index):
        print(index)
        return torch.randn(1)
    
    def __len__(self):
        return 10


indices = torch.arange(10)
sampler = torch.utils.data.sampler.SubsetRandomSampler(indices)
dataset = MyDataset()
loader = DataLoader(dataset, sampler=sampler)

for epoch in range(2):
    print(epoch)
    for _ in loader:
        pass
</code></pre>

omrjml: <p>Thank you! I misunderstood how SubsetRandomSampler works. I thought it was keeping the same order of indices as it was initialised with and shuffle was required in the DataLoader.</p>