Reproducibility over Different Machines

username: kko
cooked: <p>I read all the posts regarding reproducibility including <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/randomness.html</a>. Here are my experiences so far:</p>
<ol>
<li>
<p>calling the below code segments makes sure that I got consistent results within the same machine.<br>
seed = 1234<br>
random.seed(seed)<br>
os.environ[‘PYTHONHASHSEED’] = str(seed)<br>
np.random.seed(seed)<br>
torch.manual_seed(seed)<br>
torch.cuda.manual_seed(seed)<br>
torch.cuda.manual_seed_all(seed)<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.enabled = False</p>
</li>
<li>
<p>My goal is to get consistent results over different machines. For this, I made sure that I have the same python, numpy, pytorch, torchvision versions etc. for both machines. Still, results are very different for two machines.</p>
</li>
<li>
<p>To point out which part of the code results in inconsistency, I removed the parts of the code one by one<br>
i- One reason was dropout. When I change to in_replace dropout, this problem solved. I.e. dropout gives consistent results.<br>
ii- The only remeaning part that leads to inconsistent results is optimizer.step(). However, I could not find any solution to solve this inconsistency.</p>
</li>
</ol>
<p>I am not sure if loss.backward() or optimizer.step() functions use CUDA functions mentioned in <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/randomness.html</a>. If so, can anyone suggest a possible solution for inconsistencies due to optimizer.step()?</p>
<p>This is very important for me. Even if there is no absolute solution to this problem, I like to try different options.</p>

username: albanD
cooked: <p>Hi,</p>
<p>The problem is that across different machines, the hardware (for example different GPU cards) or different library version (cudnn adding/deleting algorithms for example) can give different results. So there is little we can do to ensure the same results <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: kko
cooked: <p>Thanks! I am aware of this possibility. But, when I remove optimizer.step() everything exactly same even though I have dataloader shuffle, dropout etc. So, maybe there is something I can try also for optimizer.step()?</p>

username: albanD
cooked: <p>You might be able to track it down for this model and this pair of machines.<br>
But then this most likely won’t work if you try on other machines/other environments.</p>
<p>Now if you really want to reproduce between these two machines, which optimizer are you using?</p>

username: kko
cooked: <p>Thanks for the response!</p>
<p>I am using Adam optimizer, but I also tried with SGD and it did not help.</p>

username: albanD
cooked: <p>I’m afraid you’ll have to check each op in the optimizer and see which one differs. You might want to use sgd as it is much simpler.</p>

username: kko
cooked: <p>Thanks. I will try your suggestion once I find time. It has been big pain to get the same results <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Maybe, I should first check gradient values? Possibly, inconsistency starts there.</p>

username: albanD
cooked: <p>Ho yes, very likely the gradients are different as well.<br>
“It has been big pain to get the same results” Yes unfortunately the floating point standard makes it very hard as it does not force a given value of a result, but just forces values to be close enough to them.</p>

username: kko
cooked: <p>Thanks, I was hopeful as I was getting exact same network outputs when commenting out optimizer.step().</p>
<p>If I get different grads for simple loss function like cross entropy, is there any way to solve that problem?</p>

username: albanD
cooked: <p>As I said above. I don’t think there is any way to do that.<br>
Even <code>a.sum()</code> depending on your GPU will give you different results :’(<br>
So it’s going to be very hard in any case.</p>

username: barakb
cooked: <p><a class="mention" href="/u/alband">@albanD</a> Hi, I’m having the same issue, 2 machines, 1 give an accuracy of x+1 the othe x-0.5, where x is (the baseline), I did 3 experiments on each machine.<br>
The machines run the same commands, all the stochastic elements disabled, and the exact same environment. <strong>except Titan V vs Titan XP.</strong><br>
The thing you said was sharp “different HW will give with different results” .<br>
It’s sounds like reproducible is  almost unreachable, isn’t there anything to do with that?</p>
<p>Further more even with the same machines I still get big gaps (although not large as using different machines.)</p>

username: albanD
cooked: <blockquote>
<p>It’s sounds like reproducible is almost unreachable, isn’t there anything to do with that?</p>
</blockquote>
<p>I’m afraid no. This is a hardware/floating point limitation. The floating point standard specifies only how close from the real value the result should be. But the hardware can return any value that is that close. So different hardware can give different result <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

username: sinemuysal
cooked: <p>Hi,</p>
<p>Just want to follow-up on this issue. Is there still no solution for this problem? I am experiencing the same issue. Same seed on different machines result in different gradients and different results. This is really problematic for model tuning and selection. Any suggestions would be really appreciated.</p>

username: albanD
cooked: <p>As mentioned above, this is not a problem we can fix. It is a hardware/floating point standard issue.<br>
If you need more precision, you will have to use double precision floats (which won’t be perfect either) at the cost of speed.</p>