Why do we need to specify non-linearity while initializing the weights of convolutional layers with kaiming He initialization?

samra-irshad: <p>Hi guys, I am initializing the weights of CNN with kaiming he initializer but I dont understand why we have to specify the type of non-linear activation function in <code>nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')</code> . I am comfused about this non-linearity parameter, does it mean all the layers which I am initializing with kaiming he weights will have ‘relu’ as their activation function or it has something to do with the initialization itself?</p>

Usama_Hasan: <p><a class="mention" href="/u/samra-irshad">@samra-irshad</a>, If you look at the equation proposed to calculate the standard deviation by he et al, in the paper <a href="https://arxiv.org/abs/1502.01852" class="inline-onebox" rel="noopener nofollow ugc">[1502.01852] Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification</a>.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bf6d614b44689fda173bac66c1d3d2a9a2dc52bb.png" alt="0*DwUan_QhBFIKHFfy" data-base62-sha1="rjrxDx0BviDrOOaq73sJ7YpXqVd" width="258" height="71"></p>
<p><strong>where a = the negative slope of the rectifier used after this layer</strong>.<br>
So I guess this to initialize the distribution, where mean is zero and std is calculated using the above formula.<br>
<a href="https://pytorch.org/docs/stable/_modules/torch/nn/init.html#kaiming_uniform_" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/<em>modules/torch/nn/init.html#kaiming_uniform</em></a></p>

samra-irshad: <p>So does it mean the layers I initialize using <code>nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')</code> are all going to have <code>ReLU</code> as their activation function? In this case, I have to exclude the last layer of my model from being initialized through this method that is going to map probabilities over classes via <code>softmax</code></p>

Usama_Hasan: <aside class="quote no-group" data-username="samra-irshad" data-post="3" data-topic="105496">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samra-irshad/48/27993_2.png" class="avatar"> samra-irshad:</div>
<blockquote>
<p>So does it mean the layers I initialize using <code>nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')</code> are all going to have <code>ReLU</code> as their activation function?</p>
</blockquote>
</aside>
<p>I don’t think that is the case, as I mentioned above it calculates the standard deviation of the weights considering the next non-linearity which has to be applied, It doesn’t apply relu to them.<br>
You should check the implementation in pytorch, I have shared the link above.<br>
Take this example,</p>
<pre><code class="lang-auto">w = torch.empty(3, 5)
nn.init.kaiming_normal_(w, mode='fan_out', nonlinearity='relu')
&gt;&gt;tensor([[-0.4137,  0.3216,  0.0705, -0.4403,  0.4050],
        [-0.5409,  0.3364, -0.7153,  0.2617,  0.5652],
        [ 0.2512,  0.6643, -0.9265, -0.2095, -0.9202]])

</code></pre>
<p>If you look at the above output, you’ll see a lot of negative’s as weights, which if the ReLU was applied will not be the case.</p>

ado_sar: <p>The thing I can’t understand is that in the <a href="https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_" rel="noopener nofollow ugc">docs</a>, the default value for <code>nonlinearity</code> is <code>leaky_relu</code>, while the default value for <code>a</code> is <code>0.01</code></p>