Randomness in loading the same network

rony2: <p>Hey! I am loading the same network (before even training, right after the initialization) from a pickle file, but it has different parameters every time.<br>
I tried 2 different ways of saving and loading:</p>
<p>net = …<br>
with open(‘train.pickle’, ‘wb’) as f:<br>
pickle.dump([net], f)<br>
then:<br>
with open(‘train.pickle’, ‘rb’) as f:<br>
net1 = pickle.load(f)<br>
with open(‘train.pickle’, ‘rb’) as f:<br>
net2 = pickle.load(f)<br>
net1=net1[0]<br>
net2=net2[0]<br>
print(net1.eval()==net2.eval()) //gives False<br>
if net1.parameters() != net2.parameters():<br>
print(True) //gives True.</p>
<p>I also tried:<br>
torch.save(model, PATH)<br>
model = torch.load(PATH)</p>
<p>How can I fix this?</p>

Alexey_Demyanchuk: <p>Hello <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
The best practice in pytorch is to save a model <code>state_dict</code>. Consider this minimal example.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
m = nn.Linear(10, 2)
state = m.state_dict()
torch.save(state, 'm.pt')

m1 = nn.Linear(10, 2)
m2 = nn.Linear(10, 2)
for p1, p2 in zip(m1.parameters(), m2.parameters()):
    print(torch.all(p1 == p2))
</code></pre>
<p>prints:</p>
<pre><code class="lang-auto">tensor(False)
tensor(False)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">state = torch.load('m.pt')
m1.load_state_dict(state)
m2.load_state_dict(state)
for p1, p2 in zip(m1.parameters(), m2.parameters()):
    print(torch.all(p1 == p2))
</code></pre>
<p>prints:</p>
<pre><code class="lang-auto">tensor(True)
tensor(True)
</code></pre>

rony2: <p>Thank you for your answer.<br>
But, it didn’t work.<br>
It gives true for a very few number of parameters, but not all of them.<br>
I am using google colab btw (if it affects anything).</p>

Alexey_Demyanchuk: <p>I run this example in colab. Gives me all parameters of m1 and m2 are equal after loading the same state dict into them. Is running the exact code from my previous post gives you different results?</p>

rony2: <p>Yes, it works. except that, if you add</p>
<p>s  = sum([np.prod(list(p.size())) for p in m1.parameters()]);<br>
print (‘Number of params: %d’ % s)</p>
<p>It prints 22. Which means, that you have 22 parameters for each of m1 and m2, but only 2 of them are alike. And</p>
<p>if m1.parameters() != m2.parameters():<br>
print(True)</p>
<p>prints True.</p>

Alexey_Demyanchuk: <aside class="quote no-group" data-username="rony2" data-post="5" data-topic="104970">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/e480ec/48.png" class="avatar"> rony2:</div>
<blockquote>
<p>s = sum([np.prod(list(p.size())) for p in m1.parameters()]);<br>
print (‘Number of params: %d’ % s)</p>
<p>It prints 22. Which means, that you have 22 parameters for each of m1 and m2, but only 2 of them are alike. And</p>
</blockquote>
</aside>
<p>I don’t see how you can come to this conclusion from this code. What it is doing. It is just calculating the number of parameters in m1 model and nothing to compare it with m2</p>
<aside class="quote no-group" data-username="rony2" data-post="5" data-topic="104970">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/e480ec/48.png" class="avatar"> rony2:</div>
<blockquote>
<p>m1.parameters() != m2.parameters()</p>
</blockquote>
</aside>
<p>Here you are comparing two generators (and as they are 2 different python objects they are obviously not the same). You can see the type: <code>type(m1.parameters())</code>. That is why you need to iterate and compare parameters themself.</p>
<p>Also in this simple model you can just print parameters for both models and make sure they are the same:</p>
<pre><code class="lang-auto">for p1, p2 in zip(m1.parameters(), m2.parameters()):
    print("m1")
    print(p1)
    print("m2")
    print(p2)
</code></pre>

rony2: <p>Oh my bad. Thanks for your help!</p>