What is gain value in nn.init.calculate_gain(nonlinearity, param=None) and how to use these gain number and why?

username: Manik_Hossain
cooked: <p>what is gain value in nn.init.calculate_gain(nonlinearity, param=None) and how to use these gain number and why ?<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/58c3c203a60070cc2f174aff17cc7d3bb164e1f2.png" data-download-href="https://discuss.pytorch.org/uploads/default/58c3c203a60070cc2f174aff17cc7d3bb164e1f2" title="gain"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/58c3c203a60070cc2f174aff17cc7d3bb164e1f2_2_690x475.png" alt="gain" data-base62-sha1="cFfqse1Jxxj6rGvhnqZaalawwoy" width="690" height="475" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/58c3c203a60070cc2f174aff17cc7d3bb164e1f2_2_690x475.png, https://discuss.pytorch.org/uploads/default/original/2X/5/58c3c203a60070cc2f174aff17cc7d3bb164e1f2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/58c3c203a60070cc2f174aff17cc7d3bb164e1f2.png 2x" data-dominant-color="FAFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gain</span><span class="informations">851×587 23.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>The <code>gain</code> is a scaling factor for some init functions, e.g. <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.init.xavier_uniform_">xavier init</a>.</p>
<p>It’s used to scale the standard deviation with respect to the applied non-linearity.<br>
Since the non-linearity will affect the std dev of the activation, you might run into some issues, e.g. vanishing gradients. The gain with respect to the non-linearity should give you “good” statistics of your activations.</p>

username: Manik_Hossain
cooked: <p>what is    fan_in,       fan_out   here<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ed44f7c2d5d67f7fd8deb2c74a611657ef762338.png" data-download-href="https://discuss.pytorch.org/uploads/default/ed44f7c2d5d67f7fd8deb2c74a611657ef762338" title="t"><img src="https://discuss.pytorch.org/uploads/default/original/2X/e/ed44f7c2d5d67f7fd8deb2c74a611657ef762338.png" alt="t" data-base62-sha1="xQYVfp4AyhcE3Nnz4Na8wLGtN2w" width="690" height="200" data-dominant-color="F0F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">t</span><span class="informations">877×255 24.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>It’s basically the number of input and output activations.<br>
For a linear layer layer it would be just <code>fan_in = in_features</code> and <code>fan_out = out_features</code>:</p>
<pre><code class="lang-python">print(torch.nn.init._calculate_fan_in_and_fan_out(nn.Linear(10, 2).weight))
</code></pre>
<p>In the case of <code>nn.Conv2d</code> it’s a bit more complicated, as you need to calculate receptive field using the kernel size, the <code>in_channels</code>, and number of filters:</p>
<pre><code class="lang-python">print(torch.nn.init._calculate_fan_in_and_fan_out(nn.Conv2d(3, 1, 3, 1, 1).weight))
</code></pre>
<p>You can find the formula <a href="https://github.com/pytorch/pytorch/blob/df8c5a3572c5d9f9d9724ce880a9f511b9af95f9/torch/nn/init.py#L178">here</a>.</p>

username: Manik_Hossain
cooked: <p>so if i initiate the weight of a conv2d or conv3d i will use gain=1<br>
if i am right then</p>
<ol>
<li>where’s the use of  relu or thanh gain value as these functions have no weight to initialize ?</li>
<li>or i will multiply gain value of conv2d and relu (1*sqrt(2)) for this case bellow<br>
F.relu(self.conv2d())    (just i think this)<br>
am i right?</li>
</ol>

username: ptrblck
cooked: <p>You are right, that most activation functions don’t have parameters, so that you don’t have to initialize them.<br>
However, you use the gain corresponding to the non-linearity you’ve used in your model.<br>
E.g. for a model like conv -&gt; relu -&gt; conv -&gt; relu, you could use:</p>
<pre><code class="lang-python">nn.init.xavier_uniform_(m, gain=nn.init.calculate_gain('relu'))
</code></pre>
<p>to initialize the weights of the conv layers.</p>

username: DonnyHe
cooked: <p>Hey btrblck - sorry for replying to this old thread, but do you know any reference on how the gains are calculated? I couldn’t find it anywhere in the code base</p>

username: Zihao_Qin
cooked: <p>Hi Donny, do you have any idea now? I’m just thinking of this question now</p>

username: Zihao_Qin
cooked: <p>I think we can do experiments to test and get the result (the constant) using this code, here the example is tanh():</p>
<pre><code class="lang-auto">x = torch.randn(10000)
out = torch.tanh(x)

gain = x.std() / out.std()
print('gain:{}'.format(gain))

</code></pre>

username: DonnyHe
cooked: <p>hey zihao - excuse my slow reply.</p>
<p>i used the following references to find out where the formula came from more or less <a href="https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="noopener nofollow ugc">https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf</a>, <a href="https://arxiv.org/pdf/1502.01852.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1502.01852.pdf</a>.</p>
<p>your estimation does not work for all cases. for example for relu, the post activation value y will be a noncentered variable, and then if you multiply by a weight matrix with \sigma_w, it’s variance will be var(w) (var(y) + mean(y)^2), which will need to be accounted for, and you have no way of knowing what E(y) will be, as its value depends on the layer. Hence your way of estimation will not work for every layer, this contradicts the fact that information gain is applied uniformly across all layers.</p>
<p>As a result, I recommend reading the papers (section 4.2 and 2.2 respectively) as they derive and explain how they come up with the scaling factor which is information gain here quite well</p>

username: Zihao_Qin
cooked: <aside class="quote no-group quote-post-not-found" data-username="Zihao_Qin" data-post="12" data-topic="28131" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zihao_qin/48/42411_2.png" class="avatar"> Zihao_Qin:</div>
<blockquote>
<p>Thank you Donny, that helps a lot !</p>
</blockquote>
</aside>
<p>Thank you Donny, that helps a lot !</p>

username: j93hahn
cooked: <p>What is the expected gain for the exponent activation function? Also, I am curious to understand how gain is calculated to begin with?</p>

username: wy18811521307
cooked: <p>When browsing others’ code on GitHub, I noticed that most people don’t pass the gain parameter when using Xavier or Kaiming initialization functions, yet the models still converge normally. Does this mean that specifying the gain parameter using the calculate_gain function is a best practice, but not strictly necessary?</p>