Change batch norm gammas init

username: polo5
cooked: <p>Is there a way to change the initialization of batchnorm gammas of my model, in a way that works for both the <code>affine=True</code> and <code>affine=False</code> settings? For pytorch version &gt;= 1.2.0 these are set to one, but I would like to change that.</p>
<p>The issue is that when <code>affine=False</code> the batchnorm gammas are set to <code>None</code> and it’s not clear if that’s okay to play with.</p>

username: ptrblck
cooked: <p>You should be able to use <code>torch.nn.init</code>, if you check for <code>None</code> before calling an init method on the affine parameters:</p>
<pre><code class="lang-python">def bn_weight_init(m):
    if m.weight is not None and m.bias is not None:
        torch.nn.init.normal_(m.weight)
        torch.nn.init.normal_(m.bias)

bn = nn.BatchNorm2d(3, affine=True)
print(bn.weight, bn.bias)
bn.apply(bn_weight_init)
print(bn.weight, bn.bias)

bn = nn.BatchNorm2d(3, affine=False)
print(bn.weight, bn.bias)
bn.apply(bn_weight_init)
print(bn.weight, bn.bias)
</code></pre>

username: polo5
cooked: <p>Thanks for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> However my objective is to change gammas and betas even when they are not learnable (<code>affine=False</code>).</p>
<p>By deafault <code>affine=False</code> means that gammas are set to 1s, and are not parameters. I want to change this to some other value than 1 and still have BN gammas not learnable.</p>
<p>The best would be to rewrite my own BN module but I can’t figure out where the gammas are set to 1 in the case of <code>afffine=False</code> since they are set to None.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Ah OK, in that case I would use <code>affine = True</code>, set the desired values via <code>bn.weight.fill_()</code> or <code>.copy_()</code>, and freeze the affine parameters by setting their <code>requires_grad</code> attribute to <code>False</code>.<br>
Would that work for you?</p>

username: polo5
cooked: <p>Thanks. My issue is that I have a really annoying use case (in meta learning) where this hack doesn’t work. All my weights must be concatenated into a single trainable tensor, and passed to the network as an argument when doing the forward pass. So this trick doesn’t work because that would mean having some elements in my weight vector be non-trainable.</p>
<p>In short I need gammas not to be parameters when <code>affine=False</code> otherwise they break everything… Do you think tweaking the BN layer source code is required here? I can’t figure out why <code>gamma=None</code> gets interpreted as <code>gamma=1</code> in the BN function.</p>

username: ptrblck
cooked: <p>There are internal checks as seen <a href="https://github.com/pytorch/pytorch/blob/acbca57d18336405fc66bb63f28bbe8eb47d937c/aten/src/ATen/native/Normalization.cpp#L86-L87">here</a>, which replace the <code>None</code> values with the default ones.</p>
<p>Thanks for the information about the use case.<br>
I’m unfortunately unsure, what the best way would be (probably reimplementing the batchnorm layer manually).<br>
However, this hacky trick might also work for you:</p>
<pre><code class="lang-python">bn = nn.BatchNorm2d(3, affine=False)
delattr(bn, 'weight')
delattr(bn, 'bias')
bn.register_buffer('weight', torch.ones(3) * 2)
bn.register_buffer('bias', torch.ones(3) * 2)
</code></pre>
<p>Note that this is completely untested code and might break in various ways, but you could try it and see, if that would be working.</p>

username: polo5
cooked: <p>Thanks a lot! I’ll have a go and give up if it doesn’t work, since changing C code sounds like it’s above my (PhD) paygrade… <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>

username: ptrblck
cooked: <p>In the worst case, you can use my <a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">manual and slow reference implementation</a> to hack around. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>