Torch.manual_seed() not working

arnabsinha: <p>Hi everyone,</p>
<p>I am trying to reproduce identical random numbers using torch.manual_seed() but it does not seem to be working. I tried the below method afte referring a few existing forum posts. Please let me know what is wrong.</p>
<pre><code class="lang-auto">import torch
from torch import nn, optim
import numpy as np

device = "cuda" if torch.cuda.is_available() else "cpu"

# torch.manual_seed(42)
import random
random.seed(42)
np.random.seed(42)
torch.cuda.manual_seed(42)
torch.backends.cudnn.deterministic = True

model_linear = LinearRegression()
model_linear.to(device=device)
model_linear.state_dict()

</code></pre>
<p>The last line gives random values every single time to the weights and biases. Please if anyone can help me out.</p>
<p>Python version: 3.12.4<br>
Torch version: 2.4.1</p>

ptrblck: <p>You are initializing the model on the CPU so add <code>torch.manual_seed()</code> which should also seed the device btw.</p>

arnabsinha: <p>I uncommented torch.manual_seed(42) but still my model weights and biases are always giving different initial values.</p>

Tony-Y: <p>You can check torch.manual_seed with torch.nrand like the MPS bug:</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/84288#issue-1356130782">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/84288#issue-1356130782" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue" data-github-private-repo="false">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/84288#issue-1356130782" target="_blank" rel="noopener nofollow ugc">MPS: torch.manual_seed not working on metal (mps)  for  torch.randn</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-08-30" data-time="18:11:39" data-timezone="UTC">06:11PM - 30 Aug 22 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2023-01-03" data-time="16:01:39" data-timezone="UTC">04:01PM - 03 Jan 23 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/Vargol" target="_blank" rel="noopener nofollow ugc">
          <img alt="Vargol" src="https://avatars.githubusercontent.com/u/62868?v=4" class="onebox-avatar-inline" width="20" height="20">
          Vargol
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: random
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: mps
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

Using torch.manual_seed to set a seed for torch.nrand us<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">age not not work, seeding with the same number does not bring back consistent results for  example rung the noddy script

```
import torch

torch.manual_seed(999)
print (torch.randn(3, device='cpu'))
torch.manual_seed(999)
print (torch.randn(3, device='cpu'))
torch.manual_seed(999)
print (torch.randn(3, device='mps'))
torch.manual_seed(999)
print (torch.randn(3, device='mps'))
torch.manual_seed(999)
print (torch.randn(3, device='cpu'))
torch.manual_seed(999)
print (torch.randn(3, device='cpu'))

```


returns

```
tensor([-0.8379,  0.4564,  0.3481])
tensor([-0.8379,  0.4564,  0.3481])
tensor([ 0.2612,  1.2390, -1.1139], device='mps:0')
tensor([-1.4076,  1.3567, -0.4096], device='mps:0')
tensor([-0.8379,  0.4564,  0.3481])
tensor([-0.8379,  0.4564,  0.3481])
```

Note the two MPS lines return different values I'd expect them to be the same, ideally I'd like them to be the same as
the CPU values but appreciate that may not be possible.


BTW I've cut out the following ou of those results in case you were wondered where it was :-)

```
/Volumes/Sabrent Media/Documents/Source/Python/diffuse/lib/python3.10/site-packages/torch/_tensor_str.py:114: UserWarning: The operator 'aten::masked_select' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
  nonzero_finite_vals = torch.masked_select(

```

### Versions

```
Collecting environment information...
PyTorch version: 1.13.0.dev20220828
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.5.1 (arm64)
GCC version: Could not collect
Clang version: 13.1.6 (clang-1316.0.21.2.5)
CMake version: version 3.22.4
Libc version: N/A

Python version: 3.10.4 (main, May 10 2022, 03:52:14) [Clang 13.0.0 (clang-1300.0.29.30)] (64-bit runtime)
Python platform: macOS-12.5.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.2
[pip3] pytorch-lightning==1.4.2
[pip3] torch==1.13.0.dev20220828
[pip3] torch-fidelity==0.3.0
[pip3] torchaudio==0.13.0.dev20220827
[pip3] torchmetrics==0.6.0
[pip3] torchvision==0.14.0.dev20220827
[conda] Could not collect
```


BTW had to fix the collection script to deal with spaces in path names :-)

cc @pbelevich @kulinseth @albanD</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>Could you post the code for <code>LinearRegression</code>?</p>

arnabsinha: <p>Sure.</p>
<pre><code class="lang-auto">device = "cuda" if torch.cuda.is_available() else "cpu"

class LinearRegression(nn.Module):
    '''
        Class to define the neural network using Linear layers. Importing nn.Module is necessary whenever building any NN
    '''

    def __init__(self, *args, **kwargs) -&gt; None:
        super().__init__(*args, **kwargs)
        self.layer1 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)
        self.layer2 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        self.forward1 = self.layer1(x)
        return self.layer2(self.forward1)
</code></pre>

Tony-Y: <p>When <code>manual_seed</code> is working:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; from torch import Generator, randn
&gt;&gt;&gt; g = Generator(device='cpu')
&gt;&gt;&gt; g.manual_seed(42)
&gt;&gt;&gt; randn(3, generator=g, device='cpu') # initially missing
tensor([0.3367, 0.1288, 0.2345])
&gt;&gt;&gt; g.manual_seed(42)
&gt;&gt;&gt; randn(3, generator=g, device='cpu')
tensor([0.3367, 0.1288, 0.2345])
</code></pre>
<p>Can you get a different result on your environment?</p>

arnabsinha: <p>When I run it the first time, I get the same 3 values as you got using this code. When I run randn(3, generator=g, device=‘cpu’) multiple times, I get subsequently different values.</p>
<p>Although I don’t understand how I can fix the different values I am getting in my code.</p>

arnabsinha: <p>I checked again. So torch.manual_seed(42) is giving me the same initial values when I create the model. What I do not understand is if I am changing my device to GPU, why doesn’t torch.cuda.manual_seed(42) work then? What can I do to make it work?</p>

Tony-Y: <p>Could you provide detailed information of your environment?</p>
<ul>
<li>OS, CPU, etc.</li>
<li>How to install PyTorch: conda, pip, source</li>
</ul>

arnabsinha: <p>OS: Windows 11 64-bit operating system<br>
CPU: Intel(R) Core™ i5-8300H CPU @ 2.30GHz   2.30 GHz</p>
<p>For installing Pytorch, used the command listed here: “<a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a>”<br>
Installed Conda from “<a href="https://www.anaconda.com/download/success" class="inline-onebox" rel="noopener nofollow ugc">Download Now | Anaconda</a>” and it came with Python 3.12.4 and pip 24.0</p>

ptrblck: <p>Thanks for the model definition. Your code works as expected and returns the same random values after seeding the code:</p>
<pre data-code-wrap="python"><code class="lang-python">python tmp.py
OrderedDict([('layer1.weight', tensor([[0.7645]], device='cuda:0')), ('layer1.bias', tensor([0.8300], device='cuda:0')), ('layer2.weight', tensor([[-0.2343]], device='cuda:0')), ('layer2.bias', tensor([0.9186], device='cuda:0'))])

python tmp.py 
OrderedDict([('layer1.weight', tensor([[0.7645]], device='cuda:0')), ('layer1.bias', tensor([0.8300], device='cuda:0')), ('layer2.weight', tensor([[-0.2343]], device='cuda:0')), ('layer2.bias', tensor([0.9186], device='cuda:0'))])

python tmp.py 
OrderedDict([('layer1.weight', tensor([[0.7645]], device='cuda:0')), ('layer1.bias', tensor([0.8300], device='cuda:0')), ('layer2.weight', tensor([[-0.2343]], device='cuda:0')), ('layer2.bias', tensor([0.9186], device='cuda:0'))])
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">cat tmp.py 
import torch
import torch.nn as nn

class LinearRegression(nn.Module):
    '''
        Class to define the neural network using Linear layers. Importing nn.Module is necessary whenever building any NN
    '''

    def __init__(self, *args, **kwargs) -&gt; None:
        super().__init__(*args, **kwargs)
        self.layer1 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)
        self.layer2 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        self.forward1 = self.layer1(x)
        return self.layer2(self.forward1)

torch.manual_seed(42)

device = "cuda" if torch.cuda.is_available() else "cpu"
model_linear = LinearRegression()
model_linear.to(device=device)
print(model_linear.state_dict())
</code></pre>
<aside class="quote no-group" data-username="arnabsinha" data-post="9" data-topic="209650">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/arnabsinha/48/25300_2.png" class="avatar"> arnabsinha:</div>
<blockquote>
<p>What I do not understand is if I am changing my device to GPU, why doesn’t torch.cuda.manual_seed(42) work then?</p>
</blockquote>
</aside>
<p>As previously explained:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="209650" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You are initializing the model on the CPU so add <code>torch.manual_seed()</code> which should also seed the device btw.</p>
</blockquote>
</aside>
<p>The initial random values are created on the CPU before you are moving the model to the GPU, so you need to seed the host, too.</p>

Tony-Y: <p>Could you try to test manual_seed after downgrading PyTorch to 2.3.1?</p>
<pre><code class="lang-auto"># CUDA 11.8
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=11.8 -c pytorch -c nvidia
# CUDA 12.1
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=12.1 -c pytorch -c nvidia
# CPU Only
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 cpuonly -c pytorch
</code></pre>

arnabsinha: <p>Thanks for the help! <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>QQ: If I wish to initialize the model on GPU directly, what changes can I make to my code?</p>

ptrblck: <p>You can pass the <code>device</code> argument to the layer initialization directly e.g. via: <code>self.layer1 = nn.Linear(..., device=device)</code>.</p>

To_theMoon: <p>Hi,<br>
I am using torch GCN, and although I set all seed values, I am still getting random node feature generation from the model. Would you mind helping me understand where the issue is coming from?</p>

Tony-Y: <p>Are you using the following model?<br>
<a href="https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.GCN.html#torch-geometric-nn-models-gcn" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.GCN.html#torch-geometric-nn-models-gcn</a></p>

To_theMoon: <p>Thanks a lot for your reply! yes, this is the model I am using. However, in my case, I figured the issue was due to not resetting the parameters in the right place.</p>