Why iterating over a deterministic dataloader changes the seed state?

username: LinglanZhao
cooked: <p>By switching <strong>eval_testset = False</strong> to <strong>eval_testset = True</strong> in the above codes (i.e., whether iterating over a deterministic dataloader with shuffle=False and fixed transforms), the program with a fixed seed gets different results:</p>
<pre><code class="lang-auto">import torch
import numpy as np
import random
import torchvision
import torchvision.transforms as transforms

def set_seed(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    np.random.seed(seed)
    random.seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=32,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=32,
                                         shuffle=False, num_workers=2)

set_seed(42)
eval_testset = True

for epoch in range(10):
    if epoch &gt;= 2:
        break # for the purpose of saving time

    for i, data in enumerate(trainloader):
        inputs, labels = data
        # train the model ...
        if i == 0:
            print("eval_testset=%s, epoch=%d, iter=%d, loaded labels=%s"%(eval_testset, epoch, i, labels.tolist()))

    if eval_testset:
        with torch.no_grad():
            for data in testloader:
                images, labels = data
                # test the model ...
</code></pre>
<p><strong>eval_testset = False:</strong></p>
<pre><code class="lang-auto">eval_testset=False, epoch=0, iter=0, loaded labels=[6, 0, 4, 1, 2, 7, 9, 4, 7, 8, 4, 5, 6, 0, 4, 2, 0, 1, 6, 1, 4, 3, 2, 3, 2, 4, 0, 7, 5, 1, 8, 6]
eval_testset=False, epoch=1, iter=0, loaded labels=[8, 7, 1, 6, 2, 3, 0, 7, 7, 2, 9, 8, 8, 7, 5, 9, 0, 6, 9, 6, 7, 5, 9, 3, 7, 8, 9, 4, 4, 4, 0, 7]
</code></pre>
<p><strong>eval_testset = True:</strong></p>
<pre><code class="lang-auto">eval_testset=True, epoch=0, iter=0, loaded labels=[6, 0, 4, 1, 2, 7, 9, 4, 7, 8, 4, 5, 6, 0, 4, 2, 0, 1, 6, 1, 4, 3, 2, 3, 2, 4, 0, 7, 5, 1, 8, 6]
eval_testset=True, epoch=1, iter=0, loaded labels=[9, 5, 7, 2, 2, 3, 3, 6, 2, 4, 2, 4, 1, 4, 1, 1, 6, 9, 7, 6, 9, 4, 8, 1, 3, 4, 7, 9, 3, 9, 3, 9]
</code></pre>
<p>I am wondering how can a deterministic dataloader change the seed state of the codes.<br>
(environment: tested on pytorch 1.13.1 and 2.0.1)</p>

username: KFrank
cooked: <p>Hi Linglan!</p>
<aside class="quote no-group" data-username="LinglanZhao" data-post="1" data-topic="199610" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/linglanzhao/48/68750_2.png" class="avatar"> LinglanZhao:</div>
<blockquote>
<p>I am wondering how can a deterministic dataloader change the seed state of the codes.</p>
</blockquote>
</aside>
<p>I can reproduce your issue (running pytorch version 2.2.1 on both the<br>
cpu and gpu.)</p>
<p>I’ve narrowed it down to the operation that create an iterator (in my case a<br>
<code>_SingleProcessDataLoaderIter</code>) from a <code>DataLoader</code>.  Here is a script<br>
that shows that merely creating the iterator advances the state of the<br>
pseudo-random-number generator (while iterating over the iterator does<br>
not seem to further advance the rng):</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

print ('reset seed ...')
_ = torch.manual_seed (2024)
print ('first eight rands ...')
print ('torch.rand (8):', torch.rand (8))

print ('reset seed ...')
_ = torch.manual_seed (2024)
print ('create tensors, Dataset, and Dataloader ...')

inp = torch.arange (4.0)
trg = torch.arange (4)
ds = torch.utils.data.TensorDataset (inp, trg)
dl = torch.utils.data.DataLoader (ds, batch_size = 2)

print ('no rands consumed, still the same first eight rands ...')
print ('torch.rand (8):', torch.rand (8))

print ('reset seed ...')
_ = torch.manual_seed (2024)
print ('create iterator from DataLoader ...')

itdl = iter (dl)

print ('two rands consumed, rands start with third of the first eight ...')
print ('torch.rand (8):', torch.rand (8))

print ('reset seed ...')
_ = torch.manual_seed (2024)
print ('extract two batches from DataLoader iterator ...')

print ('next (itdl) =', next (itdl))
print ('next (itdl) =', next (itdl))

print ('no rands consumed, get first eight rands again ...')
print ('torch.rand (8):', torch.rand (8))
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">2.2.1
reset seed ...
first eight rands ...
torch.rand (8): tensor([0.5317, 0.8313, 0.9718, 0.1193, 0.1669, 0.3495, 0.2150, 0.6201])
reset seed ...
create tensors, Dataset, and Dataloader ...
no rands consumed, still the same first eight rands ...
torch.rand (8): tensor([0.5317, 0.8313, 0.9718, 0.1193, 0.1669, 0.3495, 0.2150, 0.6201])
reset seed ...
create iterator from DataLoader ...
two rands consumed, rands start with third of the first eight ...
torch.rand (8): tensor([0.9718, 0.1193, 0.1669, 0.3495, 0.2150, 0.6201, 0.4849, 0.7492])
reset seed ...
extract two batches from DataLoader iterator ...
next (itdl) = [tensor([0., 1.]), tensor([0, 1])]
next (itdl) = [tensor([2., 3.]), tensor([2, 3])]
no rands consumed, get first eight rands again ...
torch.rand (8): tensor([0.5317, 0.8313, 0.9718, 0.1193, 0.1669, 0.3495, 0.2150, 0.6201])
</code></pre>
<p>I can’t say that this behavior actually violates anything in the pytorch<br>
documentation, but, even so, I would say that it’s sufficiently odd and<br>
unexpected to count as a bug (and is seemingly completely unnecessary).</p>
<p>It might make sense for you to file a github issue about this.</p>
<p>Best.</p>
<p>K. Frank</p>

username: ptrblck
cooked: <p>Creating the <code>base_seed</code> should be responsible for this behavior as seen <a href="https://github.com/pytorch/pytorch/blob/a4b02a15d33cdd946d95af51f48bb0640a6abcb5/torch/utils/data/dataloader.py#L605">here</a>.</p>

username: LinglanZhao
cooked: <p>Thank you for your reply!</p>

username: LinglanZhao
cooked: <p>Thanks for your reply!</p>

username: KFrank
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="199610" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Creating the <code>base_seed</code> should be responsible for this behavior</p>
</blockquote>
</aside>
<p>This indeed looks like the cause.</p>
<p>It’s hardly a big deal, but still, why consume some random numbers<br>
unnecessarily?</p>
<p>It appears that this is strictly for multiprocessing.  <code>_BaseDataLoaderIter</code><br>
knows whether or not it will be used for multiprocessing (as, for example,<br>
through its <code>_num_workers</code> property).</p>
<p>Or, probably better in my mind, why not push the initialization of <code>_base_seed</code><br>
down into <code>_MultiProcessingDataLoaderIter</code>, which has the (only?) code<br>
that actually uses <code>_base_seed</code>?</p>
<p>I can’t say that this is  really a major issue, but it was unexpected enough to<br>
catch at least one pytorch user by surprise.</p>
<p>As an aside, in Linglan’s case:</p>
<aside class="quote no-group" data-username="LinglanZhao" data-post="1" data-topic="199610">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/linglanzhao/48/68750_2.png" class="avatar"> LinglanZhao:</div>
<blockquote>
<pre><code class="lang-auto">testloader = torch.utils.data.DataLoader(testset, batch_size=32,
                                         shuffle=False, num_workers=2)
</code></pre>
</blockquote>
</aside>
<p>he would still see his issue even if the problem were “fixed” because he is<br>
explicitly asking for multiprocessing with <code>num_workers = 2</code>.  Even though<br>
there is not actually any randomness in Linglan’s <code>testloader</code>, I don’t see<br>
any practical way for this to be known by <code>_BaseDataLoaderIter</code> (or by<br>
<code>_MultiProcessingDataLoaderIter</code>) if randomness were hiding in some<br>
<code>transform</code> or <code>collate_fn</code> code.</p>
<p>Best.</p>
<p>K. Frank</p>

username: ptrblck
cooked: <p>I agree with your point in trying to avoid seeding code when it’s not really needed as it can indeed create unexpected issues.<br>
I also agree with your suggestion to create an issue on GitHub discussing this.</p>

username: LinglanZhao
cooked: <p>Thank you again for both of your valuable discussions. I have (re)opened an <a href="https://github.com/pytorch/pytorch/issues/122697" rel="noopener nofollow ugc">issue</a> on GitHub to discuss it.</p>