Randn output differs locally vs Google Colab instance with same seed

peterdn: <p>I’m very new to PyTorch, coming at it from playing with Stable Diffusion. A couple of days ago I noticed that I was no longer able to reproduce SD output on my laptop that I was getting in Google Colab, despite using the same seed and other parameters. Furthermore and confusingly, this appears to have <em>changed</em> on my laptop somehow: an image I was previously able to reproduce now looks different when I generate it locally.</p>
<p>I’ve tracked down the difference to the output of <code>torch.randn(N, device=cuda)</code>. Locally vs on Colab, the sequence begins to diverge after N=10240. That is, every value from index 0…10239 is identical. Value at index 10240 is different, and from there on the sequence diverges.</p>
<p>Is this expected or unusual? Are there any obvious reasons why this would suddenly change without any known change to my hardware or software? I have since upgraded my drivers to the latest and am still getting the (same) diverging output.</p>
<p>To verify I used the following small script:</p>
<pre><code class="lang-auto">import torch
cuda = torch.device("cuda")
torch.manual_seed(12345)
torch.randn(10241, device=cuda)
</code></pre>
<p>Locally this outputs:</p>
<pre><code class="lang-auto">tensor([ 0.5786, -0.5248, -0.2919,  ..., -0.2040, -1.8688,  0.2480],
       device='cuda:0')
</code></pre>
<p>On Colab (note the last element is different):</p>
<pre><code class="lang-auto">tensor([ 0.5786, -0.5248, -0.2919,  ..., -0.2040, -1.8688, -0.1346],
       device='cuda:0')
</code></pre>
<p>I’m running Windows 10 x64, GeForce GTX 950M, Core i7-6700HQ.</p>

gug: <p>Hey there,</p>
<p>I am not expert in this but, One possible reasons might be the different GPUs/cuda version on colab.</p>
<p>you can check the GPU type of your colab environment by <code>!nvidia-smi</code> command.</p>
<p>Edit:- found a similar <a href="https://discuss.pytorch.org/t/getting-different-results-running-the-code-in-colab-and-jupyternotebook/88655">post</a></p>

KFrank: <p>Hi Peter!</p>
<aside class="quote no-group" data-username="peterdn" data-post="1" data-topic="161398" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peterdn/48/53221_2.png" class="avatar"> peterdn:</div>
<blockquote>
<p>Locally this outputs:</p>
<pre><code class="lang-auto">tensor([ 0.5786, -0.5248, -0.2919,  ..., -0.2040, -1.8688,  0.2480],
       device='cuda:0')
</code></pre>
<p>On Colab (note the last element is different):</p>
<pre><code class="lang-auto">tensor([ 0.5786, -0.5248, -0.2919,  ..., -0.2040, -1.8688, -0.1346],
       device='cuda:0')
</code></pre>
</blockquote>
</aside>
<p>For what it’s worth I reproduce your Colab result running on my system.</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; torch.version.cuda
'11.6'
&gt;&gt;&gt; torch.cuda.get_device_name()
'GeForce GTX 1050 Ti'
&gt;&gt;&gt; cuda = torch.device ("cuda")
&gt;&gt;&gt; torch.manual_seed (12345)
&lt;torch._C.Generator object at 0x7fd8924da410&gt;
&gt;&gt;&gt; torch.randn (10241, device=cuda)
tensor([ 0.5786, -0.5248, -0.2919,  ..., -0.2040, -1.8688, -0.1346],
       device='cuda:0')
</code></pre>
<p>I don’t have an explanation and it does seem like a minor bug, although<br>
pytorch explicitly warns that it is not guaranteed to be exactly reproducible<br>
across platforms.</p>
<p>(I do note that 10241 is one more than <code>10 * 2**10</code>, although that’s hardly<br>
a good reason to go off the rails …)</p>
<p>Best.</p>
<p>K. Frank</p>

peterdn: <p>Thanks both!</p>
<aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="161398">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>pytorch explicitly warns that it is not guaranteed to be exactly reproducible<br>
across platforms</p>
</blockquote>
</aside>
<p>Is it worth me reporting this as an issue on GitHub, or does the explicit lack of guarantee basically mean this is not-unexpected behaviour?</p>

ptrblck: <p>It’s expected behavior as there is no guarantee for deterministic behavior between different devices and generally setups.</p>