DataLoader is not deterministic

username: MosheM
cooked: <p>Hi,</p>
<p>I am using DataLoader and even with single worker getting different generation of image order.  How can I make it generate the same image order every time ?  (and does it depend on <span class="hashtag">#workers</span>?)</p>
<p>Code is initialized with:</p>
<pre><code>random.seed(1)
torch.manual_seed(1)
torch.backends.cudnn.deterministic = True
</code></pre>
<p>In addition, I would like to be able to serialize DataLoader’s internal state to file so that if I stop the run at the middle of an epoch and resume it later then I can continue from the same place it stopped.</p>
<p>Thank you,<br>
Moshe</p>

username: ptrblck
cooked: <p>The <code>DataLoader</code> should provide the same random ordering when seeded and if there are no race conditions.<br>
This example gives the same ordering for <code>num_workers=0</code> or <code>=1</code>.</p>
<pre><code class="lang-auto">torch.manual_seed(2809)
torch.backends.cudnn.deterministic = True # Not necessary in this example

class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(25, 1)
        
    def __getitem__(self, index):
        print('Index: ', index)
        return self.data[index]

    def __len__(self):
        return len(self.data)


dataset = MyDataset()
loader = DataLoader(dataset,
                    batch_size=5,
                    shuffle=True,
                    num_workers=0,
                    pin_memory=True)

for batch_idx, data in enumerate(loader):
    data = data.to('cuda')
</code></pre>
<p>If you use a higher number of workers the order or the samples might differ.<br>
At least I observe this effect on my machine. I assume this is due to the race conditions in different processes.</p>

username: MosheM
cooked: <p>Hi,</p>
<p>Thanks for the code. Your example indeed reproduce numbers in same orders over different trials.</p>
<p>In my code, however, which is very similar in principle to this example, I get a different behavior even for #workers=1.  I don’t have a clue where to start looking for the source of this problem.</p>
<p>I would appreciate any advice.</p>
<p>Please note that I also asked about saving the state of DataLoader so that I can stop and resume run during epoch… If you can then please address that part as well.</p>
<p>Thank you,<br>
Moshe</p>

username: ptrblck
cooked: <p>Do you call any other random functions from another lib like numpy?<br>
If it’s possible, I would remove unnecessary parts of the code and see if a minimal example produces the deterministic behavior.</p>
<p>Regarding your second question:<br>
<a class="mention" href="/u/alband">@albanD</a> posted a suggestion <a href="https://discuss.pytorch.org/t/start-dataloader-at-specific-batch-idx/15883/2?u=ptrblck">here</a></p>

username: MosheM
cooked: <p>Hi,</p>
<p>Thanks… I traced the problem to (unordered) dictionary creation and iteration which led to different results every run.  After switching to OrderedDict things work as expected.  I wasn’t aware that there is any randomness involved in standard dictionary (thought the keys are unordered but not in a random sense).</p>
<p>Regarding my second question, I went over the linked code but I still would like to use RandomSampler.  If I understand correctly, it is sufficient to record 2 variable: the RNG state when creating the iterator over DataLoader (since this is when the random permutation is taken) and also how many “next” requests it received so far.  Is it correct that these 2 variable capture the state of the DataLoader ?</p>
<p>Thanks,<br>
-Moshe</p>

username: ptrblck
cooked: <p>Nice you’ve found the problematic part!</p>
<p>I think you are basically right. The use case would be a bit more complicated if you use more workers.<br>
Also how are you going to stop the <code>DataLoader</code>? Do you want to stop it with CTRL+C?<br>
If so, you have to take care of stopping all workers, since I’ve seen quite often zombie python processes from stopped <code>DataLoader</code>s.</p>
<p>I’m not sure, if this will yield to a clean solution.</p>

username: John1231983
cooked: <p>Why <code>torch.backends.cudnn.deterministic = True</code> does make speed of my code too slow? I am using TitanXp. Without the flag, the code run fast. I am also using cudnn 7 and cuda 9.1 pytotch 0.4</p>

username: ptrblck
cooked: <p>Yes, <code>cudnn.deterministic=True</code> trades speed for determinism.<br>
If you really need the deterministic behavior, you won’t have another option.<br>
Usually, it’s fine to leave it disabled and also set <code>torch.backends.cudnn.benchmark = True</code> to gain a bit more speed.</p>

username: John1231983
cooked: <p>Thanks. I am writing a network that train from scratch with own dataset. My aim to reproduce the result after each running. However, i cannot reproduce it ( because the losses are different when i rerun the code). This is my setting</p>
<pre><code class="lang-auto">random.seed(1234) #  np.randint()
torch.manual_seed(1234)
cudnn.enabled = True
cudnn.benchmark=True
</code></pre>
<p>Do you have any suggession?</p>

username: ptrblck
cooked: <p>How large is the difference in the results?</p>

username: John1231983
cooked: <p>The first three epoch of the first running like</p>
<pre><code class="lang-auto">0 | 1.78229 
1 | 1.31438 
2 | 1.06264 
3 | 0.95527 
4 | 1.22681 
5 | 0.87360 
6 | 0.84937 
7 | 0.82182 
8 | 0.73692
</code></pre>
<p>The second running is</p>
<pre><code class="lang-auto">0 | 1.65189 
1 | 1.30213 
2 | 1.23979 
3 | 1.01453 
4 | 0.94315 
5 | 0.78658 
6 | 0.88853 
7 | 0.71684 
8 | 0.64936
</code></pre>

username: ptrblck
cooked: <p>With <code>cudnn.benckmark=False</code> and <code>cudnn.deterministic=True</code> the values are identical?<br>
The difference seems to be quite large.</p>

username: John1231983
cooked: <p>Sorry. I have updated the new result. It looks also too larger. I will test your case</p>

username: John1231983
cooked: <p>This is the result based on your settinh</p>
<pre><code class="lang-auto">  0 | 1.63084 
  1 | 1.26171 
  2 | 1.00780 
  3 | 0.87280
</code></pre>
<p>And</p>
<pre><code class="lang-auto">    0 | 1.62280 
    1 | 1.32109 
    2 | 1.16098

</code></pre>
<p>But the problem is that setting deterministic=True  spends a lot of time for 1 epoch (about 100x slower)</p>

username: ptrblck
cooked: <p>I just wanted to see, if the results are equal in such a case, which doesn’t seem to be the case.<br>
Maybe the difference is based on some other effect like the <code>dict</code>/<code>OrderedDict</code> issue?</p>

username: John1231983
cooked: <p>How to check if i used it? I remember i did not use them</p>

username: ptrblck
cooked: <p>Could you try to get deterministic results by setting the seeds, set cudnn to deterministic etc.?<br>
This would exclude any possible effects like unknown order in python <code>dicts</code> like in <a class="mention" href="/u/moshem">@MosheM</a>’s case.</p>

username: deepanwayx
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I wasn’t able to obtain the same random ordering from your example in pytorch&gt;=1.1.0.</p>
<p>Here’s the <a href="https://colab.research.google.com/drive/1pYUH09Bz4gjLY9gZZR2EMs_tJCJdir4y" rel="nofollow noopener">notebook</a>. Is there a way to make the dataloader deterministic in newer pytorch versions?</p>