Why are my models initialized differently?

username: hetekaslan
cooked: <p>Hi all,</p>
<p>I’ve searched the forum for my question but couldn’t find anything similar. Here’s my problem.</p>
<p>I have this architecture with 4 different nn.Sequential blocks (shared below). I’m trying to investigate the impact of latent_size with the values of 4, 8, and 16. As it is seen, “<strong>self.lf_model</strong>” does not depend on the input argument latent_size. However, the initial weights of “<strong>self.lf_model</strong>” changes with the changing latent_size even though I use seeding as torch.manual_seed(15451), random.seed(15451), and, np.random.seed(15451).</p>
<p>class Model(nn.Module):<br>
def <strong>init</strong>(self, latent_size):<br>
super(Model, self).<strong>init</strong>()</p>
<pre><code>    self.encoder    =  nn.Sequential(nn.Linear(in_features=4096, out_features=1024),
                                    nn.Linear(in_features=1024, out_features=256),
                                    nn.Linear(in_features=256, out_features=64),
                                    nn.Linear(in_features=64, out_features=latent_size)
                                    )

    self.lf_model   = nn.Sequential(nn.Linear(in_features=4096, out_features=512), nn.ReLU(), 
                                    nn.Linear(in_features=512, out_features=128), nn.ReLU(),
                                    nn.Linear(in_features=128, out_features=16), nn.ReLU(),
                                    nn.Linear(in_features=16, out_features=2)
                                    )
    

    self.lc_model   = nn.Sequential(nn.Linear(in_features=latent_size+2, out_features=32), nn.Linear(in_features=32, out_features=2))

    self.nlc_model  = nn.Sequential(nn.Linear(in_features=latent_size+2, out_features=32), nn.ReLU(),
                                    nn.Linear(in_features=32, out_features=32), nn.ReLU(),
                                    nn.Linear(in_features=32, out_features=2))
</code></pre>
<p>Why do I get different initial weights for “self.lf_model”? How can I initialize them the same?</p>
<p>Thank you in advance!</p>

username: hetekaslan
cooked: <p>If you would like to see it, here are initial <strong>self.lf_model</strong> initial weights for different <em>latent_size</em>.</p>
<blockquote>
<p>latent_size = 4<br>
model = Model(latent_size).cuda()<br>
print(model.lf_model[0].weight)</p>
</blockquote>
<blockquote>
<p>Parameter containing:<br>
tensor([[ 8.1891e-03,  1.4299e-02, -8.9244e-03,  …, -1.4864e-02,<br>
1.0010e-02, -2.7618e-03],<br>
[ 5.1851e-03, -8.8850e-03,  1.0219e-02,  …,  3.1422e-04,<br>
-2.7850e-03,  3.5177e-03],<br>
[-2.3300e-05, -1.4028e-02, -8.1398e-03,  …,  6.1992e-03,<br>
-1.8799e-03,  3.8405e-03],<br>
…,<br>
[ 2.7780e-03, -3.9676e-03,  1.3176e-02,  …, -3.9793e-03,<br>
1.0665e-02, -9.7567e-03],<br>
[-1.3217e-02,  4.1098e-04,  1.0644e-02,  …, -5.4149e-03,<br>
-1.0527e-02,  1.3252e-02],<br>
[ 8.9856e-03,  9.2430e-03, -6.2179e-04,  …,  3.8803e-03,<br>
-1.3446e-02,  2.1152e-03]], device=‘cuda:0’, requires_grad=True)</p>
</blockquote>
<blockquote>
<p>latent_size = 8<br>
model = Model(latent_size).cuda()<br>
print(model.lf_model[0].weight)</p>
</blockquote>
<blockquote>
<p>Parameter containing:<br>
tensor([[-0.0076, -0.0055,  0.0103,  …, -0.0006,  0.0016,  0.0093],<br>
[-0.0151,  0.0023, -0.0092,  …,  0.0107, -0.0094, -0.0080],<br>
[-0.0088, -0.0056, -0.0116,  …, -0.0095,  0.0092, -0.0143],<br>
…,<br>
[ 0.0132, -0.0096,  0.0010,  …, -0.0121,  0.0117,  0.0089],<br>
[-0.0092,  0.0075,  0.0073,  …, -0.0044, -0.0027, -0.0148],<br>
[-0.0021, -0.0137, -0.0100,  …,  0.0074,  0.0108,  0.0043]],<br>
device=‘cuda:0’, requires_grad=True)</p>
</blockquote>
<blockquote>
<p>latent_size = 16<br>
model = Model(latent_size).cuda()<br>
print(model.lf_model[0].weight)</p>
</blockquote>
<blockquote>
<p>Parameter containing:<br>
tensor([[-3.9591e-03, -1.2846e-02,  9.7075e-03,  …,  1.2239e-02,<br>
4.1199e-03, -8.0544e-03],<br>
[ 6.0928e-03, -6.3468e-03,  2.9797e-03,  …,  1.6150e-03,<br>
4.3071e-03, -6.5914e-03],<br>
[-4.8567e-03,  2.6767e-03,  2.6474e-03,  …, -6.6011e-03,<br>
1.5258e-02,  1.0542e-02],<br>
…,<br>
[-5.6493e-03, -1.1037e-02,  1.4620e-03,  …, -6.7706e-03,<br>
3.3715e-03,  1.0564e-02],<br>
[ 1.0363e-02, -6.8797e-04, -6.7189e-03,  …,  1.3069e-03,<br>
-9.9623e-03, -1.0964e-03],<br>
[ 1.1568e-03, -8.8054e-03, -7.8261e-03,  …,  1.1733e-02,<br>
4.6302e-05, -1.3099e-02]], device=‘cuda:0’, requires_grad=True)</p>
</blockquote>

username: ptrblck
cooked: <p><code>self.encoder</code> uses <code>latent_size</code> as the <code>out_features</code> of the last linear layer and thus to sample the parameters from the pseudorandom number generator (PRNG). Since seeding guarantees to generate the same sequence of random numbers, it’s expected that <code>self.lf_model</code> uses different random values, since the calls into the PRNG changed.</p>

username: hetekaslan
cooked: <p>Thank you for the explanation.</p>
<p>I used a simple workaround. I initialized and saved a model. For each latent_size, I used that base model with one layer edited.</p>