Different Output for Multiple Linear Layer Output

username: ritchieng
cooked: <p>For example in a  class</p>
<pre><code class="lang-auto">def __init__():
  multi_output = [nn.Linear(50, 2) for i in range(10)]

def forward():
   outputs = [head(x) for head in multi_output]
</code></pre>
<p>The<code>outputs</code> keep changing in values at each run if I feed just a simple input tensor of 0’s.</p>
<p>No dropouts, running on eval().</p>
<p>Why would the ouputs keep changing at each run when I’m using the same model’s weights (<code>network.load_state_dict()</code>)?</p>

username: acobobby
cooked: <p>Hello!<br>
That’s strange…</p>
<p>So, you are calling <code>__init__</code> only once and then apply <code>forward</code> on the same input multiple times without doing any optimization or changing the weights in any way, right?</p>
<p><strong>Edit</strong>: I don’t know if this has something to do with your issue but the class you declared has something wrong: the methods of the class should have <code>self</code> as first argument. Moreover usually you want to subclass <code>nn.Module</code> and calling <code>super</code> in the constructor.</p>

username: ritchieng
cooked: <p>Yes.</p>
<p>I was debugging and I realize the way to have the same results is by using <code>torch.manual_seed(10)</code> which is likely affecting the weight initializations.</p>
<p>But this doesn’t make sense considering I’m loading weights via <code>network.load_state_dict()</code>? Very bizarre.</p>

username: ritchieng
cooked: <p>Yes I’ve done all that (the usual template). I omitted them in the code above so it’s easier to just see my intention of a multi linear layer at the end of the network having different results.</p>

username: acobobby
cooked: <p>Have you checked manually if the weights are still the same after loading of the model? At this point they should be different because you are returning different results, even though I don’t know why they are changed.</p>

username: ritchieng
cooked: <p>Weights are staying the same at each run on checking.</p>

username: acobobby
cooked: <p>I reproduced the error and if I use a <code>nn.ModuleList</code> instead of a simple list to contain the modules the error is fixed.</p>

username: acobobby
cooked: <p>As you can see by running the following example when I load the <code>state_dict</code> the weights are the same as the ones of the original module. By using a simple list to hold the linear modules instead they differ.</p>
<pre><code class="lang-auto">import torch

class M(torch.nn.Module):      
  def __init__(self): 
    super(M, self).__init__() 
    self.multi_output = torch.nn.ModuleList([torch.nn.Linear(50, 2) for i in range(10)]) 

  def forward(self, x): 
    outputs = [head(x) for head in self.multi_output] 
    return outputs 


m = M()
input = torch.zeros(10,50)
print( m(input)[0] )
print( m.multi_output[0].weight )


torch.save( m.state_dict(), 'm.pt' )

m = M()
m.load_state_dict(torch.load('m.pt')) 
print( m(input)[0])
print( m.multi_output[0].weight )
</code></pre>

username: ritchieng
cooked: <p>Shucks you are right! Stupid mistake of mine. Thanks.</p>

username: dato_nefaridze
cooked: <p>only thing i don’t understand is: at first you have (50,2) and then you have (50, 2) again, how is that even possible? there should be (2, x) on the second layer</p>