GPU gives different results than CPU in training the model

MUSTAFA_CAGRI_CIVICI: <p>I have a quick question. What I noticed is that GPU leads the model overfitting while CPU gets good result. My data is very small so I built the model basic. But interestingly CPU gave good results then I wanted the model runs through GPU so that it moves fast. But i am seeing dramatic bad results in val_loss. But the results can be considered as good in CPU.</p>
<p>I searched from somewhere, then it is said that CPU accidentally regulates the model by moving slow in updating gradients. Is it correct? What are your thoughts?</p>

ptrblck: <aside class="quote no-group" data-username="MUSTAFA_CAGRI_CIVICI" data-post="1" data-topic="217422">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mustafa_cagri_civici/48/74964_2.png" class="avatar"> MUSTAFA_CAGRI_CIVICI:</div>
<blockquote>
<p>then it is said that CPU accidentally regulates the model by moving slow in updating gradients. Is it correct?</p>
</blockquote>
</aside>
<p>No, a slower compute won’t regularize the model in any way.</p>

Naming-isDifficult: <p>Though I cannot convince myself this is where the issue arises, have you tried to fix your random seed?</p>
<pre><code class="lang-auto">import torch
import random
import numpy as np

torch.manual_seed(42)
random.seed(42)
np.random.seed(42)
</code></pre>

ptrblck: <p>Setting the seed will not guarantee to reproduce exactly the same values on different devices.</p>

MUSTAFA_CAGRI_CIVICI: <p>I did as <a class="mention" href="/u/naming-isdifficult">@Naming-isDifficult</a> mentioned. But when i kept searching, i discovered that each results after every single training are different from each other. I was thinking that randomness would be the same in each run if i set seeds with a generic number. But some results are very dramatically different somehow. It is very annoying to catch the best model evaluation. Even without doing anything btw these 2 runs, i got these scores.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa08f113a41eec0d33055defc8dc873940f96931.png" data-download-href="https://discuss.pytorch.org/uploads/default/fa08f113a41eec0d33055defc8dc873940f96931" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa08f113a41eec0d33055defc8dc873940f96931_2_690x335.png" alt="image" data-base62-sha1="zFUwyAV9Zm4zpXMDDUgQNw12CXf" width="690" height="335" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa08f113a41eec0d33055defc8dc873940f96931_2_690x335.png, https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa08f113a41eec0d33055defc8dc873940f96931.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa08f113a41eec0d33055defc8dc873940f96931.png 2x" data-dominant-color="F6F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">760×370 59.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bc3d14757d7be0a09f64a4ac0b15b53c77a348c.png" data-download-href="https://discuss.pytorch.org/uploads/default/7bc3d14757d7be0a09f64a4ac0b15b53c77a348c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bc3d14757d7be0a09f64a4ac0b15b53c77a348c_2_690x318.png" alt="image" data-base62-sha1="hESguax78zh8I6ilj0ccOPkKayU" width="690" height="318" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bc3d14757d7be0a09f64a4ac0b15b53c77a348c_2_690x318.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bc3d14757d7be0a09f64a4ac0b15b53c77a348c_2_1035x477.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bc3d14757d7be0a09f64a4ac0b15b53c77a348c.png 2x" data-dominant-color="F4F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1143×528 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Note: testing accuracy was meant to say ‘validation’. it is written wrong.</p>
<p>My data would be exposed by image augmentation (cv2.compose) with random.choice. the data is splitted into train and val in a random way (seed = 42), torch tensor would gather the data in a shuffle way.</p>
<p>Is there any proper solution to get the model to give the same results in each run? Did i miss anyhing in this concept?</p>

roach: <p>Could you try to initialize a model (cpu or gpu), then save the initialized state of the weights and run it without shuffles and augmentations, then port the initial model to the 2nd device and repeat.<br>
It should come back identical (i think).</p>

lostalot: <p>You might want to see this paper about randomness and results:    Hadges, A., &amp; Bellur, S. (2025). Statistical Validity of Neural-Net Benchmarks. <em>IEEE Open Journal of the Computer Society</em>, <em>6</em>(01), 211-222.  <a href="http://doi.ieeecomputersociety.org/10.1109/OJCS.2024.3523183" class="inline-onebox" rel="noopener nofollow ugc">CSDL | IEEE Computer Society</a></p>