Somthing about Linear.py in nn moudles

username: mutouqizi
cooked: <p>In official documentation, the weights of Linear layer are initialized by the formula:1/in_features , but l find the formula in linear.py is another one ,as show in picture.I want to know why " a=math.sqrt(5)" ,when i choose ReLU as activate function.WHY?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38f467e8d250a5a2578cf5976064e75d2d6363bb.png" data-download-href="https://discuss.pytorch.org/uploads/default/38f467e8d250a5a2578cf5976064e75d2d6363bb" title="捕获2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38f467e8d250a5a2578cf5976064e75d2d6363bb.png" alt="捕获2" data-base62-sha1="87QnhFbzSGObQxIycYlzffrIThh" width="690" height="205" data-dominant-color="F8F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">捕获2</span><span class="informations">850×253 7.19 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: SumanthRH
cooked: <p>The initialization done is same as in the docs. The implementation simply uses the <code>kaiming_uniform_</code>  function (<a href="https://pytorch.org/docs/stable/nn.init.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.init — PyTorch 1.7.0 documentation</a>) and passes the right parameter to get a uniform distribution in (-1/<code>sqrt(in_features)</code>,1/<code>sqrt(in_features)</code>) . In Kaiming uniform initialization, there is an extra <code>sqrt(3)</code>. So they’ve made the <code>gain</code> value 1/<code>sqrt(3)</code> using</p>
<pre><code class="lang-auto">gain = torch.nn.init.calculate_gain(nonlinearity='leaky_relu', param=math.sqrt(5))
</code></pre>
<p>Check <a href="https://github.com/pytorch/pytorch/blob/4ff1823fac8833d4bdfe333c49fd486143175395/torch/nn/init.py#L347" class="inline-onebox" rel="noopener nofollow ugc">pytorch/init.py at 4ff1823fac8833d4bdfe333c49fd486143175395 · pytorch/pytorch · GitHub</a>.</p>
<p>Seems like a workaround to avoid writing another function that might duplicate code in <code>kaiming_uniform_</code></p>

username: mutouqizi
cooked: <p>I can understand the operation of  initializing weights. What i can’t understand is that in docs( <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear" rel="noopener nofollow ugc">torch.nn.linear</a>) it says  the weights of Linear layer are initialized by the formula:<code>1/in_features</code>. But in linear.py, it use <code>kaiming_uniform_</code>,which youcan find in this link :<a href="https://github.com/pytorch/pytorch/blob/4ff1823fac8833d4bdfe333c49fd486143175395/torch/nn/modules/linear.py#L35" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/4ff1823fac8833d4bdfe333c49fd486143175395/torch/nn/modules/linear.py#L35</a></p>

username: mutouqizi
cooked: <p>And here’s the weirdest thing for me is the function <code>init.kaiming_uniform_(self.weight, a=math.sqrt(5))</code>,which means the weights for the full connection layer are initialized by default using the formula corresponding to LeakyRelu.I want to know why it choose <code>a=math.sqrt(5)</code> instead of the negative_slope chosed by user.</p>

username: SumanthRH
cooked: <p>Hey. I realise now that my answer wasn’t clear earlier. They’ve used the <a href="https://pytorch.org/docs/stable/_modules/torch/nn/init.html#kaiming_uniform_" rel="noopener nofollow ugc">Kaiming uniform initialisation</a>, yes but they’ve chosen the parameter <code>a</code> such that <code>gain</code> is set to be <code>1/sqrt(3)</code>. The way they’ve done that is using the <code>calculate_gain</code> function (<a href="https://github.com/pytorch/pytorch/blob/4ff1823fac8833d4bdfe333c49fd486143175395/torch/nn/init.py#L67" class="inline-onebox" rel="noopener nofollow ugc">pytorch/init.py at 4ff1823fac8833d4bdfe333c49fd486143175395 · pytorch/pytorch · GitHub</a>). With <code>nonlinearity=leaky_relu</code>, the gain factor is <code>math.sqrt(2.0 / (1 + param ** 2))</code>. Choose <code>a=param=math.sqrt(5)</code>. Then the gain factor is <code>1/sqrt(3)</code>.</p>

username: mutouqizi
cooked: <p>Maybe I didn’t make my question clear. I have understood the operation of Kaiming initialization. What I don’t understand is why when i use ReLu as the activation function , the full connection layer uses the default scheme: <code>init.kaiming_uniform_(self.weight, a=math.sqrt(5))</code>to initialize its weights, which is different from its <a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#linear" rel="noopener nofollow ugc">docs</a> and l have check its resnet.py(<a href="https://github.com/pytorch/vision/blob/146dd8556c4f3b2ad21fad9312f4602cd049d6da/torchvision/models/resnet.py#L186" class="inline-onebox" rel="noopener nofollow ugc">vision/resnet.py at 146dd8556c4f3b2ad21fad9312f4602cd049d6da · pytorch/vision · GitHub</a>), in which ReLu is used as activation function, but FC is initialied by deauflt, too.<br>
I think the initialization parameter <code>a</code> should be set to <code>0</code> when ReLu is used as activation function, but now it is <code>sqrt(5)</code>,which shoulid be used in LeakyReLU.</p>

username: mutouqizi
cooked: <p><code>init.kaiming_uniform_(self.weight, a=math.sqrt(5))</code> in Linear.py.Check it in:<a href="https://github.com/pytorch/pytorch/blob/4ff1823fac8833d4bdfe333c49fd486143175395/torch/nn/modules/linear.py#L86" class="inline-onebox" rel="noopener nofollow ugc">pytorch/linear.py at 4ff1823fac8833d4bdfe333c49fd486143175395 · pytorch/pytorch · GitHub</a></p>

username: googlebot
cooked: <p>Linear class has no knowledge about activations, so this reset_parameters function had to be universal. It is also invoked from the constructor, thus it has no arguments, you’re supposed to manually re-initialize parameters after construction if defaults don’t work well.</p>

username: SumanthRH
cooked: <p>As <a class="mention" href="/u/googlebot">@googlebot</a> mentions, this initialization is universal, invoked in the constructor of <strong>any</strong> linear layer. The <code>nonlinearity</code> argument passed to the <code>kaiming_uniform_</code> function is <em>not</em> based on the nonlinearity that follows the Linear layer. The whole idea is to use the <code>kaiming_uniform_</code> function to the get the uniform initialization we want, by passing the right set of arguments (<code>a</code> and <code>nonlinearity</code>).</p>
<p>You can maybe think about this as PyTorch devs trying to avoid writing any code that duplicates statements in <code>kaiming_uniform_</code>.</p>

username: mutouqizi
cooked: <p>Thank you very much for your constant reply. I have understood it.</p>