How could I fix the random seed absolutely

coincheung: <p>Hi,</p>
<p>I add these lines at the beginning of my code, and the <code>main.py</code> of my code goes like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np
import random
import my_model
import my_dataset

torch.manual_seed(123)
torch.cuda.manual_seed(123)
np.ranom.seed(123)
random.seed(123)
torch.backends.cudnn.enabled=False
torch.backends.cudnn.deterministic=True

def train():
...

if __name__ == '__main__':
    train()
</code></pre>
<p>I run my program for two times without modifying the code. The loss of the first iter are both <code>9.044713973999023</code>, but the loss of the second iter becomes different with one to be <code>9.045238494873047</code> and the other <code>9.045231819152832</code>.</p>
<p>The reason why I am so concerned about these random behavior is that I find the evaluation results of running the same code for two times can become as large as around 0.5%. I trained my model on cityscapes for 80k iter, and I believe it should be enough for the model to be converged. Thus I do not quite understand why there is such a gap between running two times, and why the loss goes different even at the second iter? How could I fix the random behavior to make two results to be identical as expected please?</p>

colesbury: <p>That should generally be sufficient. A few operations have non-deterministic gradient calculations. For example, the gradient calculation of <code>index_select()</code> with duplicate indices can result in non-determinism due to the ordering of floating point additions.</p>
<p>The description of the operators with non-deterministic behavior is here:<br>
<a href="https://pytorch.org/docs/stable/notes/randomness.html#pytorch" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/randomness.html#pytorch</a></p>
<p>Try to narrow down where the non-determinism is introduced. Use <a href="https://pytorch.org/docs/stable/nn.html?highlight=backward%20hook#torch.nn.Module.register_backward_hook" rel="nofollow noopener">backward hooks</a> to print out where in the network the gradient starts to vary between runs.</p>

coincheung: <p>Thanks for replying!!</p>
<p>I just do not understand, will these randomness bring a performance gap as much as 0.5% when I run the identical code for two times?</p>

coincheung: <p>Hi,</p>
<p>In the code above, did I place the lines of <code>torch.manual_seed ...</code> in the correct locations? I used <code>torch.distributed</code> to manipulate multi-gpu training. Do I need to move these lines after <code>dist.init_process_group</code> or other locations?</p>

colesbury: <p><code>dist.init_process_group</code> should be fine, but if you’re using <code>torch.multiprocessing.spawn</code> or similar you need to make sure that the seeding happens <em>after</em> you spawn</p>

coincheung: <p>Thanks a lot for suggesting hook!!</p>
<p>I tried to use hook to print the mean of the grad tensor, and I find that there is an <code>F.interpolate</code> in my code which brings different gradients each time. Do I have any method to eliminate this randomness? With this randomness,  the result fluctuates violently and I cannot tune the hyperparameters of my model.</p>

Deeply: <p>I am also confused withe too many seeding methods, there are these seeding metods too…</p>
<pre><code class="lang-auto">torch.random.initial_seed()  
torch.cuda.manual_seed_all(seed_value)
torch.cuda.set_rng_state(cuda_rng_state)
torch.set_rng_state(rng_state)
</code></pre>

MariosOreo: <p>Hello,</p>
<p>It seems that non-deterministic of <code>interpolate</code> is a common question.<br>
Here is a <a href="https://discuss.pytorch.org/t/non-deterministic-behavior-of-pytorch-upsample-interpolate/42842">thread</a> in this forums about <code>interpolate</code>, it might give you more information.</p>

coincheung: <p><a class="mention" href="/u/colesbury">@colesbury</a> <a class="mention" href="/u/mariosoreo">@MariosOreo</a> <a class="mention" href="/u/deeply">@Deeply</a><br>
HI,</p>
<p>I come into another problem that I suspect is associated with random behavior. I am training a resnet18 on cifar-10 dataset. The model is simple and standard with only <code>conv2d</code>, <code>bn</code>, <code>relu</code>, <code>avg_pool2d</code>, and <code>linear</code> operators. There still seems to be random behavior problems, even though I have set the seed of random generator. I come to ask is there a random problem with <code>avg_pool2d</code> as the <code>interpolation</code>?</p>
<p>Tell me and I will copy my code here if you feel it is the problem with my code <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">.</p>

MariosOreo: <p>According to the <a href="https://pytorch.org/docs/stable/notes/randomness.html#pytorch" rel="noopener nofollow ugc">doc</a>,</p>
<blockquote>
<p>There are some PyTorch functions that use CUDA functions that can be a source of non-determinism. One class of such CUDA functions are atomic operations, in particular  <code>atomicAdd</code> , where the order of parallel additions to the same value is undetermined and, for floating-point variables, a source of variance in the result.</p>
</blockquote>
<p><strong>AND</strong> a number of operations have backwards that use <code>atomicAdd</code>, such as many forms of pooling, padding, and sampling. There currently is no simple way of avoiding non-deterministic in these functions.</p>