The same code and the same parameters, but got different results

username: Derek
cooked: <p>Put the same code(model and training) and the same parameters on two GPUs(two same GPUs on one server).<br>
But the different results appear. Why?<br>
In the code, we set the random seed using the following code:</p>
<p>CUDA = torch.cuda.is_available()<br>
import random<br>
random.seed(123)<br>
np.random.seed(123)<br>
torch.manual_seed(123)<br>
if CUDA:<br>
torch.cuda.manual_seed(123)<br>
torch.cuda.manual_seed_all(123)<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False</p>
<p>How to reproduce the experiment?</p>
<p>Here, the two figures from tensorboard as the following show different results.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/1/914f7a7b4b880aa508ae276c5c117a4cb26a489c.png" data-download-href="https://discuss.pytorch.org/uploads/default/914f7a7b4b880aa508ae276c5c117a4cb26a489c" title="Screenshot%20from%202019-11-21%2009-53-39"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/914f7a7b4b880aa508ae276c5c117a4cb26a489c.png" alt="Screenshot%20from%202019-11-21%2009-53-39" data-base62-sha1="kJtvVDTpvFVWYb4ArEOQ13Krudm" width="527" height="500" data-dominant-color="EEF6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-11-21%2009-53-39</span><span class="informations">682×647 41.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/0074bdb9097177785b8c5348fe91374f07b59fb2.png" data-download-href="https://discuss.pytorch.org/uploads/default/0074bdb9097177785b8c5348fe91374f07b59fb2" title="Screenshot%20from%202019-11-21%2009-59-54"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/0074bdb9097177785b8c5348fe91374f07b59fb2_2_544x500.png" alt="Screenshot%20from%202019-11-21%2009-59-54" data-base62-sha1="427es7G5ao2PeDMSO1x5C3XErM" width="544" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/0074bdb9097177785b8c5348fe91374f07b59fb2_2_544x500.png, https://discuss.pytorch.org/uploads/default/original/3X/0/0/0074bdb9097177785b8c5348fe91374f07b59fb2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/0/0074bdb9097177785b8c5348fe91374f07b59fb2.png 2x" data-dominant-color="F3F7F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-11-21%2009-59-54</span><span class="informations">696×639 37.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Not all CUDA ops are currently deterministic in PyTorch as explained in the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>. Could you check, if you are using such operations in your model?</p>

username: Derek
cooked: <p>Thanks for quick response.<br>
There is a torch.Tensor.scatter_add_() operation in the model code.<br>
Does this operation take such great difference?</p>
<p>Does it mean if there is torch.Tensor.scatter_add_() in the model, the result will not be reproduced?<br>
The torch.Tensor.scatter_add_() function is needed in the model. How to solve this problem?</p>

username: ptrblck
cooked: <p>The difference is usually in the range of floating point precision, which can accumulate over time.<br>
If a bitwise accuracy is needed, you would have to work around <code>scatter_add_</code> using some indexing or push this operation to the CPU.</p>