Xavier Initialization PyTorch vs MxNet

username: jean-marc
cooked: <p>I am porting an MxNet paper implementation to PyTorch</p>
<pre><code class="lang-auto">mx.init.Xavier(rnd_type="uniform", factor_type="avg", magnitude=0.0003)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">torch.nn.init.xavier_uniform_(array, gain=0.0003) 
</code></pre>
<p>Should be pretty much the same, right?</p>
<p>But the docs and source code show another “definition” of <strong>magnitude</strong> and <strong>gain</strong></p>
<p>Even when scaling gain and magnitude correctly, I am still getting different ranges of numbers.</p>
<p>Both starting from an <strong>empty</strong> array and initializing it.</p>
<p>The image can show the docs of both <a href="https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.xavier_uniform_" rel="noopener nofollow ugc">PyTorch</a> and <a href="https://mxnet.apache.org/api/python/docs/api/initializer/index.html#mxnet.initializer.Xavier" rel="noopener nofollow ugc">MxNet</a>.</p>
<p>Am I missing something?<br>
How can I make sure that both PyTorch and MxNet functions are initializing a specific input array in the same way?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2c68994615e0add9f9aaa55600e428b14b80aee.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2c68994615e0add9f9aaa55600e428b14b80aee" title="Screenshot 2020-02-28 at 16.02.19"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c68994615e0add9f9aaa55600e428b14b80aee_2_690x206.png" alt="Screenshot 2020-02-28 at 16.02.19" data-base62-sha1="u4BDgeygnUTa6UnSBjIgDurr9F4" width="690" height="206" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c68994615e0add9f9aaa55600e428b14b80aee_2_690x206.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c68994615e0add9f9aaa55600e428b14b80aee_2_1035x309.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2c68994615e0add9f9aaa55600e428b14b80aee_2_1380x412.png 2x" data-dominant-color="EDEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-02-28 at 16.02.19</span><span class="informations">3584×1072 600 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>I can’t find the usage of <code>magnitude</code> in the docs.<br>
Based on the default value, and <a href="https://beta.mxnet.io/_modules/mxnet/initializer.html#Xavier">the source code</a> it seems to be the <code>3</code> inside the sqrt.</p>
<p>If that’s the case, you won’t get the same output for <code>gain, magnitude = 0.0003</code>.</p>
<p>For <code>magnitude = 0.0003</code>, you would need to use <code>gain = sqrt(0.0003 / 6)</code>.</p>

username: jean-marc
cooked: <p>Based on the source code of <strong>MxNet</strong>:</p>
<pre><code class="lang-auto">factor = (fan_in + fan_out) / 2.0
...
scale = np.sqrt(self.magnitude / factor)
</code></pre>
<p>Their docs show the function:<br>
<code>c = \\sqrt{\\frac{3.}{0.5 * (n_{in} + n_{out})}}</code><br>
Which could possibly mean that the default magnitude=3 is the one in <code>frac{3}</code><br>
So they have in the upper part of the fraction (inside sqrt) <strong>3(magnitude) * 2(the 0.5 in the denominator)</strong></p>
<p>In <strong>PyTorch</strong> <code>gain</code> acts differently.<br>
If we take it inside the function we will have <code>gain^2 * 6</code> which should be equal to the MxNet <code>magnitude*2</code> which makes <code>gain = sqrt(magnitude/3)</code> meaning that our <strong>magnitude of</strong> <strong>0.0003</strong> would be a <strong>gain of</strong> sqrt(0.0003/3) = <strong>0.01</strong></p>
<p>Which is still weird given that I am getting different ranges of results.</p>
<p>It may be a question more suitable for the MxNet forum so I will ask it there, make sure to reach a solution, then go back here to post either a link to the solution or my own explanation of it.</p>
<p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for taking the time to check it out!</p>