How to fix/define the initialization weights/seed

username: isalirezag
cooked: <p>How I can fix the weight initialization or define my own initialization for weights?<br>
I read <a href="https://discuss.pytorch.org/t/weight-initilzation/157">Weight initilzation</a> , but im still not sure how to do it.<br>
Lets say i implemented this simple network: <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html</a></p>
<p>How can i define the weight to be what i like for the start?</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You have to create the init function and apply it to the model:</p>
<pre><code class="lang-python">def weights_init(m):
    if isinstance(m, nn.Conv2d):
        nn.init.xavier_uniform(m.weight.data)
        nn.init.xavier_uniform(m.bias.data)
    

model = MyModel()
model.apply(weights_init)</code></pre>

username: isalirezag
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you!!! Makes sense.</p>
<p>Just a follow up question.</p>
<p>Lets say your network is like this:</p>
<pre><code class="lang-auto">
class MyModel(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p>and you define the weight like this:</p>
<pre><code class="lang-auto">def weights_init(m):
    if isinstance(m, nn.Conv2d):
        nn.init.xavier_uniform(m.weight.data)
        nn.init.xavier_uniform(m.bias.data)
</code></pre>
<p>Am i putting it in a right place in th following?</p>
<pre><code class="lang-auto">
class MyModel(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def weights_init(m):
    if isinstance(m, nn.Conv2d):
        nn.init.xavier_uniform(m.weight.data)
        nn.init.xavier_uniform(m.bias.data)

   def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p>Or you suggest to put it in another way?<br>
I know it is kinda a stupid question, but the reason that im asking it is that I was wondering if if there is a way to put it inside the <strong>init</strong>(self), but im not sure if it is possible and also not sure if it is even a good idea.<br>
So just was wondering.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>You can define it as a class function, but usually it’s defined outside of your model so that it can be reused by other models.<br>
Make sure to add another condition and init for the linear layers <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"></p>

username: jpeg729
cooked: <p>I notice that the model uses ReLU activations and therefore plain Xavier initialisation will be suboptimal for the main weights.</p>
<p>A better solution would be to supply the correct gain parameter for the activation.<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.init.calculate_gain" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.init.calculate_gain</a></p>
<pre><code class="lang-auto">nn.init.xavier_uniform(m.weight.data, nn.init.calculate_gain('relu'))
</code></pre>
<p>With relu activation this almost gives you the Kaiming initialisation scheme. Kaiming uses either fan_in or fan_out, Xavier uses the average of fan_in and fan_out.</p>
<p>For the biases though, always initialising them to zero seems like a solid default choice.</p>

username: banikr
cooked: <aside class="quote no-group quote-modified" data-username="jpeg729" data-post="5" data-topic="20156">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpeg729/48/3464_2.png" class="avatar"> jpeg729:</div>
<blockquote>
<p>nn.init.calculate_gain(‘relu’)</p>
</blockquote>
</aside>
<p>After doing this do I also have to do torch.cuda.manual_seed to fix the weights?</p>

username: ptrblck
cooked: <p>Setting the seed before initializing the parameters will make sure to use the same pseudo-random values the next time you are executing the script.</p>
<p>I’m not sure, what you are meaning with “fixate the weights”. Could you explain it a bit?</p>

username: banikr
cooked: <p>Hi Ptr,<br>
The network I am using has <code>Dropout</code>, <code>Pooling</code> layers… So when I use weight initialization as follows:</p>
<pre><code class="lang-auto">    def weights_initnot(m):
        xavier=torch.nn.init.xavier_uniform_(m.weight.data,init.calculate_gain('relu'))
        classname = m.__class__.__name__
        if classname.find('Conv') != -1:
            xavier(m.weight.data)
    #         print 'come xavier'
            #xavier(m.bias.data)
        elif classname.find('BatchNorm') != -1:
            m.weight.data.normal_(1.0, 0.02)
            m.bias.data.fill_(0)
        elif classname.find('Linear')!=-1:
            m.weight.data.normal_(0.0, 0.02)
            m.bias.data.fill_(0)
        # elif classname.find('Dropout') != -1:
</code></pre>
<p>I get errors like there is no weight attribute in Dropout or Pool class. which makes sense.</p>
<p>Then I used the following weight init using <code>isinstance</code>… it kinda solved the problem because I am getting similar weights in specific layer weight index with the same seed.</p>
<pre><code class="lang-auto">seed = 1
torch.cuda.manual_seed(seed)


def weights_init(m):
    if isinstance(m, nn.Conv3d):
        torch.nn.init.xavier_uniform_(m.weight.data, init.calculate_gain('relu'))
        # torch.nn.init.xavier_uniform_(m.bias.data)
    elif isinstance(m, nn.BatchNorm3d):
        m.weight.data.normal_(mean=1.0, std=0.02)
        m.bias.data.fill_(0)
    elif isinstance(m, nn.Linear):
        m.weight.data.normal_(0.0, 0.02)
        m.bias.data.fill_(0)

    Net = ResNet3D().to(device)
    Net.apply(weights_init)
</code></pre>
<p>On another note:<br>
Do <code>seed_torch</code> and <code>manual_seed</code> work the same way? I have seen some codes use <code>seed_torch</code>.</p>
<p><strong>Seeking Explanation:</strong> I was talking to one of my friends about the purpose of fix weight initialization. He said it has no big impact because the data we are feeding in are random because of the <code>DataLoader</code>. And we are saving the best-performing weights anyway so fixed weights are not a biggy.</p>
<p>What do you think about it?</p>

username: MERAH_Samia
cooked: <p>And why you’re using the uniform distribution, please</p>

username: ptrblck
cooked: <p><code>xavier_uniform</code> was just used as an example for no particular reason.<br>
You can of course use whatever <code>nn.init</code> method fits your use case best. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: stark
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="20156">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Setting the seed before initializing the parameters will make sure to use the same pseudo-random values the next time you are executing the script.</p>
</blockquote>
</aside>
<p>I want to initialize the weights for every layer <em>(irrespective of the initialization method)</em> using a constant seed value. How exactly it’s done in Pytorch?</p>

username: ptrblck
cooked: <p>If you want to set the same seed before each initialization, you could add <code>torch.manual_seed(SEED)</code> to the <code>weight_init</code> method (before each <code>torch.nn.init</code> call).</p>

username: stark
cooked: <p>I want each linear layer weights/biases to be initialized with the constant values. Following is the <code>weight_init()</code> method the way you suggested:</p>
<pre><code class="lang-auto">def weight_init(m):
    if isinstance(m, torch.nn.Linear):
        torch.manual_seed(786)
        torch.nn.init.xavier_uniform_(m.weight.data)
        torch.nn.init.xavier_uniform_(m.bias.data)
</code></pre>
<p>Applying above to the linear layers in the classifier parts of the VGG19.</p>
<pre><code class="lang-auto">model = VGG19()
model.apply(weight_init)
</code></pre>
<p>Following error is generated each time I call <code>model.apply(weight_init)</code>:</p>
<p><code>ValueError: Fan in and fan out can not be computed for tensor with fewer than 2 dimensions</code></p>
<p>Where am I making a mistake?</p>

username: ptrblck
cooked: <p>You cannot apply <code>xavier_uniform_</code> on the <code>bias</code>, as more than a single dimension is needed in the tensor.</p>

username: stark
cooked: <p>Initializing the bias values differently will ruin my experiment. Is there a workaround other than initializing them with 0’s/1’s?</p>

username: ptrblck
cooked: <p><code>torch.nn.init.zeros_</code> or <code>torch.nn.init.ones_</code> should work.</p>