Pytorch Weight Initialisation Layer by layer

bluesky314: <p>I was seeing how the means and standard deviations of the weights of a CNN change as the model progresses and observed that the standard deviations of the later layers is initilzed much less that the first by default:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/27524060e31e521890f4cacf7827a983852fbbfb.png" data-download-href="https://discuss.pytorch.org/uploads/default/27524060e31e521890f4cacf7827a983852fbbfb" title="02%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/27524060e31e521890f4cacf7827a983852fbbfb_2_690x298.png" alt="02%20PM" data-base62-sha1="5BQQlAOACLAUskKkAndtizGhU5J" width="690" height="298" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/27524060e31e521890f4cacf7827a983852fbbfb_2_690x298.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/27524060e31e521890f4cacf7827a983852fbbfb_2_1035x447.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/2/27524060e31e521890f4cacf7827a983852fbbfb.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">02%20PM</span><span class="informations">1214×526 110 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Left is the means and right is the stds</p>
<p>This was for a small 4 layer network, even when I made a larger network I observed the same, that the later layers were intilizied to have much lower variance:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/65494bc76514db9b2e79569315055426c0c6e7cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/65494bc76514db9b2e79569315055426c0c6e7cb" title="40%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/65494bc76514db9b2e79569315055426c0c6e7cb_2_690x224.png" alt="40%20PM" data-base62-sha1="es1g9ljAjOOjjkWHlYW1ilAjiLp" width="690" height="224" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/65494bc76514db9b2e79569315055426c0c6e7cb_2_690x224.png, https://discuss.pytorch.org/uploads/default/optimized/2X/6/65494bc76514db9b2e79569315055426c0c6e7cb_2_1035x336.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/6/65494bc76514db9b2e79569315055426c0c6e7cb_2_1380x448.png 2x" data-dominant-color="E8E8E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">40%20PM</span><span class="informations">1920×624 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Usually the standard deviation of the second layer is half that of the first layer</p>
<p>As you can see the stds systematically decrease. Can someone explain why this is done? I know we want our ouputs to have mean 0 and std 1 but how is this helping that?</p>
<p>Code:<br>
Create any CNN in pytorch and then run</p>
<pre><code>model=Conv()

def stats(x): return x.mean(),x.std()


for i,e in enumerate(model.children()):
  print(i,e)
  print('-----Layer Mean and std',stats(e.weight))
</code></pre>
<h1><a name="update-1" class="anchor" href="#update-1"></a>Update</h1>
<p>Even with I manually initilise the layers I get the same behavior!</p>
<pre><code class="lang-auto">model=Conv()

for l in model.children():
  try:
    init.kaiming_normal_(l.weight)
    l.bias.data.zero_()
  except:
    pass

def stats(x): return x.mean(),x.std()


for i,e in enumerate(model.children()):
  print(i,e)
  print('-----Layer Mean and std',stats(e.weight))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/6c7681fbee3f83c8e0ab77d796effe890400cf77.png" data-download-href="https://discuss.pytorch.org/uploads/default/6c7681fbee3f83c8e0ab77d796effe890400cf77" title="05%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/6c7681fbee3f83c8e0ab77d796effe890400cf77_2_690x217.png" alt="05%20PM" data-base62-sha1="ftvt2SWsYqMJa8FzvYjPd6MND5J" width="690" height="217" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/6c7681fbee3f83c8e0ab77d796effe890400cf77_2_690x217.png, https://discuss.pytorch.org/uploads/default/optimized/2X/6/6c7681fbee3f83c8e0ab77d796effe890400cf77_2_1035x325.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/6/6c7681fbee3f83c8e0ab77d796effe890400cf77_2_1380x434.png 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">05%20PM</span><span class="informations">1874×592 99.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is pretty unsettling! pytorch is keeping track of the layer index. Why is this happening?</p>