Adaptive Normalization

username: bala
cooked: <p>I want to implement adaptive normalization as suggested in the paper Fast Image Processing with Fully- Convolutional networks. The normalization is defined as a<em>x + b</em>BN(x) where a and b are learnable scalar parameters and BN is the 2d batch normalization operator.  This normalizer needs to be invoked during training after every leaky_relu activated 2d convolution layer. How do I go about coding this normalizer?</p>

username: SimonW
cooked: <p>Something like the following should work. There might be some errors as I didn’t look up the exact API, but those should be easy to fix.</p>
<pre><code class="lang-auto">class ABN2d(nn.Module):
  def __init__(self):
    super().__init__()
    self.bn = nn.BatchNorm2d(affine = False)
    self.register_parameter('a', Variable(torch.*Tensor(1, 1, 1, 1), requires_grad = True))
    self.register_parameter('b', Variable(torch.*Tensor(1, 1, 1, 1), requires_grad = True))

  def forward(self, x):
    return a * x + b * self.bn(x)
</code></pre>

username: ptrblck
cooked: <p>I was implementing it myself and compared my approach to yours.</p>
<p>Here is my module:</p>
<pre><code>class AdaptiveBatchNorm2d(nn.Module):
    def __init__(self, num_features, eps=1e-5, momentum=0.1, affine=True):
        super(AdaptiveBatchNorm2d, self).__init__()
        self.bn = nn.BatchNorm2d(num_features, eps, momentum, affine)
        self.a = nn.Parameter(torch.FloatTensor(1, 1, 1, 1))
        self.b = nn.Parameter(torch.FloatTensor(1, 1, 1, 1))

    def forward(self, x):
        return self.a * x + self.b * self.bn(x)
</code></pre>
<p>Could you explain the difference between your approach registering the Variables as parameters and using <code>nn.Parameter</code>.<br>
I also don’t know, why you set <code>affine</code> to False.<br>
The original BN implementation introduced two parameters (gamma and beta)<br>
(Btw. is there any option to write greek letters? \beta seems not to work)</p>
<p>As I understand the new formulation would be:</p>
<pre><code>a * x + b * gamma * x_norm + b * beta
</code></pre>
<p>Thus using <code>affine=False</code>, you would lose the <code>b * beta</code> term or am I missing something?</p>

username: SimonW
cooked: <p><code>nn.Parameter</code> and <code>register_parameter</code> should be equivalent. Turns out that I should use <code>Parameter</code> instead of <code>Variable</code> in above code.<img src="//discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=5" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"></p>
<p>I set affine to false because b, beta and gamma will be all trainable. So the formulation is equivalent but with less params except for gradient values.</p>

username: bala
cooked: <p>But the pytorch doc says if affine = False, gamma and beta are not learnt during training?</p>

username: SimonW
cooked: <p>If affine is false, there won’t be any gamma or beta.</p>
<p>I just realized that if affine = True, then it is possible to learn of an bias offset. If you also want that, you may either use affine = True at the cost of unnecessary parameter, or add another parameter. <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Brando_Miranda
cooked: <p>what exact paper are you referring to? is it this one <a href="https://ieeexplore.ieee.org/document/5596746" class="inline-onebox" rel="noopener nofollow ugc">Adaptive Normalization: A novel data normalization approach for non-stationary time series | IEEE Conference Publication | IEEE Xplore</a></p>

username: vinaykumar2491
cooked: <aside class="quote no-group" data-username="Brando_Miranda" data-post="7" data-topic="9157">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/48/14355_2.png" class="avatar"> Brando_Miranda:</div>
<blockquote>
<p>what exact paper are you referring to?</p>
</blockquote>
</aside>
<p>This is the paper title: “<strong>Fast Image Processing with Fully Convolution Networks</strong>” <a href="https://arxiv.org/pdf/1709.00643.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1709.00643.pdf</a></p>