The same seed but different running results on two executions

username: lizhidan
cooked: <p>I just written a simple model to classify cifar10 like below method:<a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#sphx-glr-beginner-transfer-learning-tutorial-py" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#sphx-glr-beginner-transfer-learning-tutorial-py</a></p>
<p>And I ran it twice,with the same seed:</p>
<pre><code class="lang-auto">torch.manual_seed(60)
torch.cuda.manual_seed(60)
</code></pre>
<p>and I set dataset loader <code>shuffle=False</code> without any transformer that may include random variable.<br>
Besides, my network parameters are loaded by an existed weight in order to avoid random assignment.</p>
<p>But in training period, with epoch increasing, the differences between the network weights of two executions were more and more obvious (After the first epoch, the differences are just about 1e-9, second 1e-7 and expended Persistently).<br>
Why that happened? Is there any reason like computation error or there are still some random variables ignored? Thanks in advance.</p>

username: ptrblck
cooked: <p>If you are using cuDNN, you should set the deterministic behavior.<br>
This might make your code quite slow, but might be a good method to check your code and deactivate it later.</p>
<pre><code>torch.backends.cudnn.deterministic = True</code></pre>

username: lizhidan
cooked: <p>Thanks for your reply, but it still can’t solve my problem…</p>

username: yvanscher
cooked: <p>is your weight initialization determined by this random seed? if it’s not that could explain the difference?</p>

username: lizhidan
cooked: <p>emmm I’ve described that my model loaded existed weight trained before so it was fixed without any random factors.</p>

username: ptrblck
cooked: <p>Are you using multiple workers in your <code>DataLoader</code> or any other random functions e.g. from <code>numpy</code>?</p>

username: lizhidan
cooked: <p>Yes I used multiple workers as below:</p>
<pre><code class="lang-auto">train_loader        = DataLoader(cifar10.Cifar10(mode='train',  dataset_size=DATASET_SIZE, binclassify=None), shuffle=False, batch_size=BATCH_SIZE, num_workers=BATCH_SIZE)
test_loader         = DataLoader(cifar10.Cifar10(mode='test', dataset_size=DATASET_SIZE, binclassify=None), shuffle=False, batch_size=BATCH_SIZE, num_workers=BATCH_SIZE)
validation_loader   = DataLoader(cifar10.Cifar10(mode='validation', dataset_size=DATASET_SIZE, binclassify=None), shuffle=False, batch_size=BATCH_SIZE, num_workers=BATCH_SIZE)

</code></pre>
<p>But I didn’t find any place any other random functions being used.</p>

username: ptrblck
cooked: <p>Could you just for the debugging purpose set <code>num_workers=1</code> and see if the first few iterations differ in a similar way?</p>

username: Deeply
cooked: <p>Clearly, these randomizers do not generate the same sequence; here’s an example:</p>
<pre><code class="lang-auto">torch.cuda.manual_seed(60)

torch.cuda.FloatTensor(1).normal_()
Out[94]: tensor([ 0.7700], device='cuda:0')

torch.cuda.FloatTensor(1).normal_()
Out[95]: tensor([ 0.5048], device='cuda:0')

torch.cuda.manual_seed(60)

torch.cuda.FloatTensor(1).normal_()
Out[97]: tensor([ 0.7700], device='cuda:0')

torch.manual_seed(60)
Out[98]: &lt;torch._C.Generator at 0x7f537c0320b0&gt;

torch.randn(1)
Out[99]: tensor([ 0.7534])

torch.randn(1)
Out[100]: tensor([ 1.8541])

torch.manual_seed(60)
Out[101]: &lt;torch._C.Generator at 0x7f537c0320b0&gt;

torch.randn(1)
Out[102]: tensor([ 0.7534])
</code></pre>
<p>Or, I am missing something here?</p>

username: ptrblck
cooked: <p>The seeds work for the CPU and GPU separately, but cannot generate the same random numbers for CPU and GPU.<br>
<code>torch.manual_seed(SEED)</code> will also seed the GPU, but the PRNG used on the GPU and CPU are different. The code should yield deterministic results nevertheless running on the specified device. As far as I know, I is currently not possible to get the same random numbers on different devices. Probably it’s comparable to the seeding in PyTorch vs. numpy. Both will yield deterministic results, but not the same numbers.</p>

username: hughperkins
cooked: <p><a class="mention" href="/u/deeply">@Deeply</a> Yes. I too would prefer that the PRNG is consistent between CPU and GPU; as I allude to in <a href="https://discuss.pytorch.org/t/best-practices-for-seeding-random-numbers-on-gpu/18751/1">Best practices for seeding random numbers on gpu?</a></p>

username: lizhidan
cooked: <p>It still cannot solve it.<br>
Are there any wrong in me check process described below?<br>
I ran my code twice on GPU, after <code>optimizer.step()</code> I used <code>torch.save(model[1]._parameters['weight'].cpu().data, 'w1')</code> to save the weight to my disk (the second time changes w1 to w2). Then I loaded the two weights using <code>torch.load('w1')</code> and <code>torch.load('w2')</code>, subtracting them and check if the results are all 0.</p>

username: ptrblck
cooked: <p>Looks good to me. To search for the problematic part, could you repeat this procedure with random tensors as input, i.e. don’t use your <code>Dataset</code> and <code>DataLoader</code>?<br>
Since you are seeding, the random tensor should be the same in each run.</p>

username: EnCI
cooked: <p>Hi! I’m very new with trying to learn pytorch. When trying to compare models i found that the outputs are different.<br>
Simple CNN:</p>
<p>class NetOne(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NetOne, self).<strong>init</strong>()<br>
self.c1 = 16<br>
self.c2 = 8<br>
self.c3 = 4<br>
self.size = 32<br>
self.fclen1=419<br>
self.fclen2=10<br>
self.conv1 = nn.Conv2d(3, self.c1, 3, padding=1)<br>
self.conv2 = nn.Conv2d(self.c1, self.c2, 3, padding=1)<br>
self.conv3 = nn.Conv2d(self.c2, self.c3, 3, padding=1)<br>
self.bn1 = nn.BatchNorm2d(self.c1)<br>
self.bn2 = nn.BatchNorm2d(self.c2)<br>
self.bn3 = nn.BatchNorm2d(self.c3)<br>
self.fc1 = nn.Linear(self.c3 * self.size * self.size, self.fclen2)<br>
self.fc2 = nn.Linear(self.fclen1, self.fclen2)<br>
self.dropout = nn.Dropout(0.25)</p>
<pre><code>def forward(self, x):
    x = F.relu(self.bn1(self.conv1(x)))
    x = self.dropout(x)
    x = F.relu(self.bn2(self.conv2(x)))
    x = F.relu(self.bn3(self.conv3(x)))
    x = x.view(-1, self.c3 * self.size * self.size)
    x = self.fc1(x)
    return x
</code></pre>
<p>And using it:</p>
<pre><code>   model1 = NetOne()
   model2 = NetOne()
</code></pre>
<p>…<br>
for _, (data, target) in enumerate(train_loader):<br>
if train_on_gpu:<br>
data1, target1 = data.cuda(), target.cuda()<br>
data2, target2 = data.cuda(), target.cuda()<br>
else :<br>
data1, target1 = data, target<br>
data2, target2 = data, target<br>
optimizer1.zero_grad()<br>
optimizer2.zero_grad()<br>
output1 = model1(data1)<br>
output2 = model2(data2)<br>
for i in range (len(s1)):<br>
if s1[i] != s2[i]:<br>
print(‘loss error %d : %f %f’ % (iterNum, s1[i], s2[i]))</p>
<p>gives a lot of lines like:</p>
<p>loss error 0 : -0.132742 0.350880<br>
loss error 1 : -26.758894 -30.239246<br>
loss error 2 : -10.001531 -12.009613</p>
<p>Can you suggest what’s wrong, please.</p>

username: ptrblck
cooked: <p>Based on your posted code you are initializing two models randomly, so the results are expected to be different.<br>
To get the same results, you should either set the seed before creating an instance of the model or load the <code>state_dict</code> from one model into the other (which I would recommend).<br>
Also, since the models are using dropout layers, you would have to call <code>model.eval()</code> on them to disable it.</p>