Difference between torch.manual_seed and torch.cuda.manual_seed

jef: <p>I often use  <a href="http://pytorch.org/docs/master/torch.html?highlight=manual_seed#torch.manual_seed" rel="nofollow noopener">torch.manual_seed</a> in my code. And I also set the same seed to numpy and native python’s random.</p>
<p>But I noticed that there is also <a href="http://pytorch.org/docs/master/cuda.html?highlight=manual_seed_all#torch.cuda.manual_seed" rel="nofollow noopener">torch.cuda.manual_seed</a>. I definitely use a single GPU.<br>
So what’s happening if I do not set <code>torch.cuda.manual_seed</code>? For example, <code>torch.randn</code> returns same values without <code>torch.cuda.manual_seed</code>. So I want to know what situations I should use cuda’s manual_seed.</p>
<p>So the following code will be better?</p>
<pre><code class="lang-auto">torch.manual_seed(args.seed)
torch.cuda.manual_seed(args.seed)
np.random.seed(args.seed)
random.seed(args.seed)
</code></pre>

albanD: <p>The cuda manual seed should be set if you want to have reproducible results when using random generation on the gpu, for example if you do <code>torch.cuda.FloatTensor(100).uniform_()</code>.</p>

jef: <p>Thank you!<br>
I usually do not write such a code, however, I should call the seed function. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=5" title=":smiley:" class="emoji" alt=":smiley:"></p>

amity137: <p>Hey but shouldn’t torch.manual_seed take care of both as written in <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/randomness.html</a></p>
<p>You can use <a href="https://pytorch.org/docs/stable/torch.html#torch.manual_seed" rel="nofollow noopener"> <code>torch.manual_seed()</code> </a> to seed the RNG for all devices (both CPU and CUDA)</p>

ptrblck: <p>Yes, it the behavior was changed some time ago and was most likely different, when <a class="mention" href="/u/alband">@albanD</a> answered in this thread. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Bruce_Muller: <p>I was just wondering best practice for using seeding. I’m using</p>
<p>torch.manual_seed(args.seed)<br>
torch.cuda.manual_seed(args.seed)<br>
np.random.seed(args.seed)<br>
random.seed(args.seed)</p>
<p>for running experiments on a new loss function, with the changed loss and with standard loss. Is it best to keep using a specific seed value or to vary the seed? I’m thinking some seeds may affect initialisation and therefore get into a better solution, thinking of all you need is a good init…</p>
<p>I’m training two models simultaneously in the same script, so should I have the above seed lines prior to instantiating each model individually to ensure the same initialisation? For a fair comparison of one loss over the other.</p>
<p>Is it wise using a seed for this type of research in general?</p>

Intel_Novel: <p>To be reproducible you may try all this:</p>
<pre><code class="lang-auto">seed_value= 0

# 1. Set `PYTHONHASHSEED` environment variable at a fixed value
import os
os.environ['PYTHONHASHSEED']=str(seed_value)

# 2. Set `python` built-in pseudo-random generator at a fixed value
import random
random.seed(seed_value)

# 3. Set `numpy` pseudo-random generator at a fixed value
import numpy as np
np.random.seed(seed_value)

# 4. Set `pytorch` pseudo-random generator at a fixed value
import torch
torch.manual_seed(seed_value)

</code></pre>