How are layer weights and biases initialized by default?

knowledge_unlimited: <p>I was wondering how are layer weights and biases initialized by default? E.g. if I create the linear layer torch.nn.Linear(5,100) How are weights and biases for this layer initialized by default?</p>

ptrblck: <p>Linear layers are initialized with</p>
<pre><code class="lang-auto">stdv = 1. / math.sqrt(self.weight.size(1))
self.weight.data.uniform_(-stdv, stdv)
if self.bias is not None:
    self.bias.data.uniform_(-stdv, stdv)
</code></pre>
<p>See also <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py" rel="nofollow noopener">here</a>.</p>

knowledge_unlimited: <p>Thanks! So it depends on the layer you use?</p>

ptrblck: <p>The layers are initialized in some way after creation. E.g. the conv layer is initialized like <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/conv.py#L40" rel="nofollow noopener">this</a>.<br>
However, it’s a good idea to use a suitable init function for your model.<br>
Have a look at the <a href="http://pytorch.org/docs/master/nn.html?highlight=init#torch-nn-init" rel="nofollow noopener">init functions</a>.<br>
You can apply the weight inits like this:</p>
<pre><code class="lang-auto">def weights_init(m):
    if isinstance(m, nn.Conv2d):
        xavier(m.weight.data)
        xavier(m.bias.data)

model.apply(weights_init)
</code></pre>

squirrel: <p>So it won’t throw any error if I forget to initialize some conv layers?</p>

ptrblck: <p>Yes, it won’t throw any errors. Depending on your problem, training could be trickier.</p>

spacemeerkat: <p>Is there a way to alter this code for a situation where you have nn.Conv2d layers whos bias can be on or off depending on their position in the network?</p>
<p>e.g. you have a first Conv2d with a bias term but then a later Conv2d with no bias term.</p>
<p>As the following will return an error:</p>
<pre><code class="lang-auto">    if isinstance(m, nn.Conv2d(bias=True):
        xavier(m.weight.data)
        xavier(m.bias.data)
</code></pre>

ptrblck: <p>You could use a condition to check, if <code>bias</code> was set:</p>
<pre><code class="lang-python">if isinstance(m, nn.Conv2d):
    torch.nn.init.xavier_uniform_(m.weight)
    if m.bias:
        torch.nn.init.xavier_uniform_(m.bias)
</code></pre>

spacemeerkat: <p>If I try that I get the following error when using:</p>
<pre><code class="lang-auto">def weight_init(m):
    if isinstance(m, torch.nn.Conv2d) or isinstance(m, torch.nn.Linear):
        torch.nn.init.xavier_uniform_(m.weight)
        if m.bias: 
            torch.nn.init.xavier_uniform_(m.bias)
</code></pre>
<pre><code class="lang-auto">RuntimeError: bool value of Tensor with more than one value is ambiguous
</code></pre>

ptrblck: <p>Sorry for the misleading code. It should be: <code>if m.bias is not None:</code><br>
Also, <code>xavier_uniform</code> will fail on <code>bias</code>, as it has less than 2 dimensions, so that <code>fan_in</code> and <code>fan_out</code> cannot be computed.</p>
<pre><code class="lang-python">if isinstance(m, nn.Conv2d):
    torch.nn.init.xavier_uniform_(m.weight)
    if m.bias is not None:
        torch.nn.init.zeros_(m.bias)
</code></pre>

spacemeerkat: <p>No no not at all, I should have been able to work that out for myself <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=5" title=":joy:" class="emoji" alt=":joy:"></p>
<p>Okay that seems to be okay except that I get the following error from the .zeros</p>
<p><code>AttributeError: module 'torch.nn.init' has no attribute 'zeros_'</code></p>
<p>Perhaps it’s an outdated atribute?</p>

ptrblck: <p>I think it was introduced in the latest release, i.e. <code>0.4.1</code>.<br>
I would recommend to update to it or in case it’s not possible at the moment due to whatever reason, you could use:</p>
<pre><code class="lang-python">with torch.no_grad():
    m.bias.zero_()
</code></pre>

spacemeerkat: <p>Ahh I’m using 0.4.0 so I will update to the newest version.</p>
<p>That code you sent works on 0.4.0 though which is great, thanks for your help as always! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

spacemeerkat: <p>Hi ptrblck,</p>
<p>I just wanted to follow up on this: If you were to use <code>nn.Conv2d( ... ,bias = True)</code> presumably the weight would be zeroed would it not? Because <code>True</code> is <code>!= None</code> in Python language…</p>
<p>Therefore, you must either use bias = False, or don’t insert and bias information to nn.Conv2d? Does this sound right?</p>

ptrblck: <p>If you don’t want to use the bias, you should set <code>bias=False</code> during the instantiation of the layer.<br>
Are you somehow referring to its initialization? In my example I set the bias to zeros, if it’s available.<br>
It’s still a learnable and used parameter in case you are wondering if the bias is useless afterwards.</p>

spacemeerkat: <p>Sorry, my mistake I understand now. I meant bias = False in my first sentence above but I was concerned that because in Python False is not None, that it would somehow try to attribute some bias initialisation to the layer even if you set it to False. But I assume False trumps the weight initialisation so that you are left with no bias, which is what you want.</p>

ptrblck: <p>Sorry for the confusion. In the construction of the conv layer you pass <code>bias</code> as a bool value (<a href="https://github.com/pytorch/pytorch/blob/c95fa4b9046085aefa91e76845f20918bd1054ef/torch/nn/modules/conv.py#L35">code</a>).<br>
If it is set to <code>True</code> (or anything that returns <code>True</code> in the line of code), <code>self.bias</code> will be initialized to the <code>nn.Parameter</code>.</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , in this script, the xavier algorithm will apply on all layers or only in nn.Conv2d?</p>

ptrblck: <p>It depends on the condition you are using.<br>
In the scripts I’ve posted in this thread, I’ve used <code>if isinstance(m, nn.Conv2d)</code>, so it’ll be just used for <code>nn.Conv2d</code> layers.</p>
<p>You can of course add more conditions to it for other layers/parameters etc.</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, Thank you. I have a UNet model, then do I need to put if condition for each layer?</p>