Does a dataloader change random state even when shuffle argument is False?

jmaronas: <p>Hello.</p>
<p>I am facing an unexpected behaviour in my dataloading scheme. I have two dataloaders, a <code>train_dl</code> and a <code>test_dl</code>. The <code>train_dl</code> provides batches of data with the argument <code>shuffle=True</code> and the <code>test_dl</code> provide batches with the argument <code>shuffle=False</code>.</p>
<p>I evaluate my test metrics each N epochs, i.e each N epochs I loop over <code>test_dl</code> dataset. I have realized that if the value of N changes, then the shuffled batches provided by the <code>train_dl</code> after evaluating the test metrics differ. In other words, suppose my training points are labelled: a,b,c,d,e,f and my batch size is 2. Then if I seed everything correctly I observe the following pattern always:</p>
<p>Epoch 1:<br>
Batch 1: [a,b]<br>
Batch 2: [e,d]<br>
Batch 3: [c,f]</p>
<p>Epoch 2:<br>
Batch 1: [a,f]<br>
Batch 2: [b,d]<br>
Batch 3: [e,c]</p>
<p>However, if in between epoch 1 and epoch 2 I loop over my <code>test_dl</code> then the provided batches at epoch 2 differ or in other words it seems that even <code>shuffle = False</code> for the <code>test_dl</code>, the random state seems to change.</p>
<p>Is this true?</p>
<p>Thanks</p>

ptrblck: <p>Based on the description it seems that the test <code>DataLoader</code> is calling into the PRNG at one point. Since <code>shuffle</code> is set to <code>False</code>, the <code>Dataset</code> might still use it at one point. Could you post the <code>Dataset</code> implementation (for the testset in particular) as well as the test loop?</p>

jmaronas: <p>This is the dataset class:</p>
<pre><code class="lang-auto">
class dataset_class(torch.utils.data.Dataset):
    ''' General dataset class
        Args:
                X: input -&gt; torch.tensor
                Y: targets -&gt; torch.tensor
    '''

    def __init__(self,X: torch.tensor, Y: torch.tensor):
        super(dataset_class, self).__init__()
        assert X is not None , "Invalid None type"
        assert Y is not None, "Invalid None type"
        self.X = X
        self.Y = Y

    def __len__(self):
        return self.X.size(0)

    def __getitem__(self,idx):
      
        return self.X[idx],self.Y[idx]
</code></pre>
<p>where  <code>self.X</code> and <code>self.Y</code> are constant for all the runs, i.e its order is not shuffled unless the dataloader does it. This is the main loop:</p>
<pre><code class="lang-auto">for b,(x,y) in enumerate(train_loader):

     x,y = x.to(cg.device),y.to(cg.device)
                   
      ## loss
      loss = model.loss(x,y)
      optimizer.zero_grad() # just in case
      loss.backward()
      optimizer.step()
              
      # Performance metrics                      
      if ((ep+1)%self.validate_each) == 0:
           model.performance_metrics(x,y,dataset='train')
 
      with torch.no_grad():

          if ((ep+1)%validate_each) == 0:
                    for b,(x,y) in enumerate(test_loader):
                        x,y = x.to(cg.device),y.to(cg.device)
                        model.performance_metrics(x,y,dataset='test')
</code></pre>

ptrblck: <p>Thanks for the code.<br>
I can reproduce this behavior using two simple <code>DataLoaders</code>:</p>
<pre><code class="lang-python">X_train = torch.arange(10).float().view(-1, 1)
y_train = torch.arange(10).float().view(-1, 1) + 0.1
train_dataset = dataset_class(X_train, y_train)
train_loader = DataLoader(train_dataset, batch_size=2, shuffle=True)

X_val = torch.arange(10, 20).float().view(-1, 1)
y_val = torch.arange(10, 20).float().view(-1, 1) + 0.1
val_dataset = dataset_class(X_val, y_val)
val_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)


seed = 2809
print('Seeding with {}'.format(seed))
torch.manual_seed(seed)
print('Training loop only')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    print('='*10)

seed = 2809
print('Seeding with {}'.format(seed))
torch.manual_seed(seed)
print('Training loop only')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    print('='*10)

seed = 2809
print('Seeding with {}'.format(seed))
torch.manual_seed(seed)
print('Adding validation loop')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    for j, (x_v, y_v) in enumerate(val_loader):
        print('ValIter{}, X_val: {}'.format(j, x_v))
    print('='*10)
</code></pre>
<p>If you execute the code (with your <code>dataset_class</code> definition), you’ll see that the <code>train_loader</code> batches are not the same for the second epoch, if the <code>val_loader</code> was executed without shuffling.</p>
<p>My best guess is, that the <code>_BaseDataLoaderIter</code> calls into the PRNG in <a href="https://github.com/pytorch/pytorch/blob/bcab2d68486fbacf9c68674e6b953b407ac09b98/torch/utils/data/dataloader.py#L387">this line of code</a>, which would be needed to seed each worker <a href="https://github.com/pytorch/pytorch/blob/bcab2d68486fbacf9c68674e6b953b407ac09b98/torch/utils/data/dataloader.py#L764-L769">here</a>.</p>
<p>A workaround would be to create a <code>torch.Generator</code> manually and pass it to your <code>train_loader</code>, so that PyTorch uses it for the <code>_base_seed</code> creation:</p>
<pre><code class="lang-python">gen = torch.Generator()

X_train = torch.arange(10).float().view(-1, 1)
y_train = torch.arange(10).float().view(-1, 1) + 0.1
train_dataset = dataset_class(X_train, y_train)
train_loader = DataLoader(train_dataset, batch_size=2, shuffle=True, generator=gen)

X_val = torch.arange(10, 20).float().view(-1, 1)
y_val = torch.arange(10, 20).float().view(-1, 1) + 0.1
val_dataset = dataset_class(X_val, y_val)
val_loader = DataLoader(val_dataset, batch_size=2, shuffle=False)


seed = 2809
print('Seeding with {}'.format(seed))
#torch.manual_seed(seed)
gen.manual_seed(seed)
print('Training loop only')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    print('='*10)

seed = 2809
print('Seeding with {}'.format(seed))
#torch.manual_seed(seed)
gen.manual_seed(seed)
print('Training loop only')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    print('='*10)

seed = 2809
print('Seeding with {}'.format(seed))
#torch.manual_seed(seed)
gen.manual_seed(seed)
print('Adding validation loop')
for epoch in range(2):
    for i, (x, y) in enumerate(train_loader):
        print('Iter{}, X_train: {}'.format(i, x))
    for j, (x_v, y_v) in enumerate(val_loader):
        print('ValIter{}, X_val: {}'.format(j, x_v))
    print('='*10)
</code></pre>

jmaronas: <p>Okei thank you. Normally I dont see a difference in performance due to how batches are shuffled. But for my particular problem I am experimenting  very different results just by setting validate_each to different numbers (different numbers implies different number loops over the test dataset hence different order).</p>
<p>Thanks for your solution</p>