Use of torch.manual_seed still results in different initial weights each time

username: Arun_Vishwanathan
cooked: <p>I am new to PyTorch. As part of my project, I have some code where I construct an <code>OrderedDict</code> incrementally at runtime depending on the layers that are needed as part of the model. Now the way this code is structured is as ,</p>
<pre><code class="lang-auto">growingordereddict = {}
for layers in total_layer_list:
 # depending on the layer create the corresponding nn module and add to ordered dictionary.
 # In Pytorch you need to know the number of in_channels for a module with parameters such as 
#Conv and Dense. In order to obtain the number of in channels, the hack I use is to within the loop,
#construct a temp model with the orderedDict present so far and find the output size. This output 
#size will be the input channel to the next layer

# for example,
# tempmodel = nn.Sequential(growingordereddict)
 # out = tempmodel(torch.randn(1,3,5,5)) #model takes fixed size input
 # size = out.size()

#Use this size when constructing the next nn module as in_channels
</code></pre>
<p>After this loop is done, the final OrderedDict will have all the layers.</p>
<p>At the end outside the loop I do this,</p>
<pre><code class="lang-auto">torch.manual_seed(1)
model = nn.Sequential(growingorderedict)
</code></pre>
<p>However, I notice that on running each time, the initial weights are different. They don’t seem to have a fixed value corresponding to seed 1. Why is my code getting different initial weights for the layer each run, though I use a fixed seed of 1?</p>
<p>I noticed that if I have a sample code with the OrderedDict predefined and manual seed 1, the weights are the same each run. But in the case above where I have a for loop in which I instantiate a model many times to get the output size to construct the next layer, the manual seed set outside the loop to instantiate the final model does not seem to fix the model weights? What is the connection and why is this happening?</p>

username: InnovArul
cooked: <p>It will be helpful to debug if you share the code.</p>

username: Arun_Vishwanathan
cooked: <p>thanks Arul. Unfortunately it is a bit proprietary with many modules and hence it would be difficult to share. What are you particularly looking for? currently when I print <code>model.fc.weight</code> after initializing the <code>model</code> for example if the dense layer is named <code>fc</code>, each run of the program the weights are different.<br>
This does not happen if say I had a predefined ordereddict with that seed. But this I notice only when incrementally adding on to the ordered dictionary and instantiating the model after the for loop.</p>
<p>As a pseudo sample, imagine a model with just Conv2d, flatten and Linear.</p>
<p>the for loop would look like this,</p>
<pre><code class="lang-auto">ordereddict = {} 
prev_output_size = 3
for layer in layer_list:
        ordereddict = OrderedDict(ordereddict.items() + constructmodule(layer,prev_output_size)
        # instantiate a temp model here to know the output size from the layer
        tempmodel = nn.Sequential(ordereddict)
        prev_output_size = tempmodel(torch.rand(1,3,5,5)).size()
</code></pre>
<p><code>constructmodule</code> will return a nn.module depending on the layer.</p>
<p>at the end of the loop I do this,</p>
<pre><code class="lang-auto">torch.manual_seed(1)
finalmodel = nn.Sequential(orderedict)
print(finalmodel.fc.weight) # this is different each time
</code></pre>

username: InnovArul
cooked: <p>In my understanding, ideally, it should not be different assuming that you are recreating <code>growingorderedict</code> after setting the seed.</p>
<p>You could first try to reproduce the behavior without any proprietary modules. If possible, share a short snippet when you can reproduce. Make sure that the random seed is not consumed by any other module after you set the seed.</p>

username: Arun_Vishwanathan
cooked: <p>thanks Arul, please check my updated response above</p>

username: InnovArul
cooked: <p>I see. When you create <code>ordereddict</code>, the weights are already initialized for those modules. <code>nn.Sequential</code> is just a container that holds the modules, but it does nothing to initalize the weights.  The final <code>torch.manual_seed(1)</code> is not having any effect on weights in your code.</p>

username: Arun_Vishwanathan
cooked: <p>ah! thanks, so what would be my way around this? Do I need to create a new ordered dictionary which is a deep copy of the ordereddict to instantiate the final model?</p>

username: InnovArul
cooked: <p>I would say, note down the sizes during the <code>for</code> loop and after the <code>for</code> loop, create a clean model by setting the seed.</p>
<p>or you can reset the parameters at the end:</p>
<aside class="quote" data-post="1" data-topic="20819">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sohrab_salimian/48/2326_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-re-set-alll-parameters-in-a-network/20819">How to re-set alll parameters in a network</a> 
  </div>
  <blockquote>
    How to re-set the weights for the entire network, using the original pytorch weight initialization
  </blockquote>
</aside>


username: Arun_Vishwanathan
cooked: <p>thanks Arul that’s nice to know! I did not realize that when the ordered dictionary is constructed the weights are already initialized. Does that also mean if I move the line <code>torch.manual_seed(1)</code> to the first line of the <code>for</code> loop I should be good? Because in the last iteration of the loop, the ordered dictionary contains all the layers needed for the model.</p>
<p>Secondly, the weight reset is also useful! But in that case, after resetting, how do I set it to what a particular seed would have done? like say after reset, set the weights to what would have been for seed(1)?</p>

username: InnovArul
cooked: <blockquote>
<p>if I move the line  <code>torch.manual_seed(1)</code>  to the first line of the  <code>for</code>  loop I should be good?</p>
</blockquote>
<p>Not really. Again, the module’s weights are initialized as and when they are created. This is not a fool-proof strategy.</p>
<blockquote>
<p>how do I set it to what a particular seed would have done?</p>
</blockquote>
<p>Resetting weights after setting <code>torch.manual_seed()</code> is equivalent to creating a new layer.</p>
<pre><code class="lang-auto">import torch.nn as nn 
import torch 
torch.manual_seed(2) 
x = nn.Conv2d(3,2,3) 
y = nn.Conv2d(3,2,3) 

torch.manual_seed(2) 
y.reset_parameters() 
torch.allclose(x.weight, y.weight)  # True

</code></pre>

username: Arun_Vishwanathan
cooked: <p>brilliant! Thanks Arul!</p>