Is it a pytorch bug with eps set in BatchNorm

username: xiefeiwhu
cooked: <p>In my network, I first set eps in BN is:</p>
<p>class BNPReLU(nn.Module):<br>
def <strong>init</strong>(self, nIn):<br>
super().<strong>init</strong>()<br>
self.bn = nn.BatchNorm2d(nIn, eps=1e-3)<br>
self.acti = nn.PReLU(nIn)</p>
<pre><code>def forward(self, input):
    output = self.bn(input)
    output = self.acti(output)

    return output
</code></pre>
<p>Then, I add another BN like this:</p>
<p>Class Network(nn.Module):<br>
self.bn_relu = BNPReLU(nIn)<br>
self.primary_conv = nn.Sequential(<br>
nn.Conv2d(inp, init_channels, kernel_size, stride, kernel_size//2, bias=False),<br>
nn.BatchNorm2d(init_channels),<br>
nn.ReLU(inplace=True) if relu else nn.Sequential(),<br>
)</p>
<p>But, when run my code in model.train(). the eps is set 1e-3 in all places. So, I debug this network and find the eps is set correctly in network initialization, but when go into forward(), all the eps is set 1e-3. I don’t know why</p>

username: albanD
cooked: <p>Hi,</p>
<p>Are you sure you’re not setting it somewhere?<br>
This prints the right thing for me:</p>
<pre><code class="lang-python">import torch
from torch import nn

a = nn.BatchNorm2d(10, eps=1e-3)
b = nn.BatchNorm2d(10)

print(a.eps)
print(b.eps)

inp = torch.rand(2, 10, 3, 3)

a(inp)
b(inp)

print(a.eps)
print(b.eps)
</code></pre>

username: xiefeiwhu
cooked: <p>You can’t test it like this. As I said, the eps is right in initialization, but in the forward function, they are all set to 1e-3.<br>
Code is like this:</p>
<pre><code class="lang-auto">class BNPReLU(nn.Module):
    def __init__(self, nIn):
        super().__init__()
        self.bn = nn.BatchNorm2d(nIn, eps=1e-3)
        self.acti = nn.PReLU(nIn)

    def forward(self, input):
        output = self.bn(input)
        output = self.acti(output)

        return output


class Main_net(nn.Module):
    def __init__(self, classes=2):
        super().__init__()
        self.bn_prelu_1 = BNPReLU(32)
        self.decode1 = nn.Sequential(nn.Conv2d(67, 16, kernel_size=1, stride=1, bias=False),
                                     nn.BatchNorm2d(16, eps=1e-5),
                                     nn.ReLU())
         print('self in initialization:', self)

     def forward(self, input):
        print('self in forward:', self)
        output0 = self.bn_prelu_1(input)
        output = self.decode1(output0)
</code></pre>
<p>In the main.py,  I set like this:</p>
<pre><code class="lang-auto">    model = Main_net(class=2)
    model.train()
    output = model(input)
</code></pre>
<p>The result: because i am a new user and only upload one picture, so I merge these results in one<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c535163c1ac6ad5a5ad141efdecbd79e48b1c5d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2c535163c1ac6ad5a5ad141efdecbd79e48b1c5d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c535163c1ac6ad5a5ad141efdecbd79e48b1c5d_2_690x490.jpeg" alt="image" data-base62-sha1="6k7w1zuKY4DBQiARX7OmCvNpsgB" width="690" height="490" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c535163c1ac6ad5a5ad141efdecbd79e48b1c5d_2_690x490.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c535163c1ac6ad5a5ad141efdecbd79e48b1c5d_2_1035x735.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2c535163c1ac6ad5a5ad141efdecbd79e48b1c5d_2_1380x980.jpeg 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1418×1008 180 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>By the way, I also set the eps other value, such as 1e-4 in BNPReLU and nn.BatchNorm2d. But in the forward process, the eps is all set to 1e-3. Very weird.</p>

username: albanD
cooked: <p>That gives the right result as well:</p>
<pre><code class="lang-python">import torch
from torch import nn, optim

class BNPReLU(nn.Module):
    def __init__(self, nIn):
        super().__init__()
        self.bn = nn.BatchNorm2d(nIn, eps=1e-3)
        self.acti = nn.PReLU(nIn)

    def forward(self, input):
        output = self.bn(input)
        output = self.acti(output)

        return output


class Main_net(nn.Module):
    def __init__(self, classes=2):
        super().__init__()
        self.bn_prelu_1 = BNPReLU(32)
        self.decode1 = nn.Sequential(nn.Conv2d(32, 16, kernel_size=1, stride=1, bias=False),
                                     nn.BatchNorm2d(16, eps=1e-5),
                                     nn.ReLU())
        print('self in initialization:', self)

    def forward(self, input):
        print('self in forward:', self)
        output0 = self.bn_prelu_1(input)
        output = self.decode1(output0)

mod = Main_net()
inp = torch.rand(1, 32, 20, 20)
mod(inp)
</code></pre>
<p>Gives me:</p>
<pre><code class="lang-auto">self in initialization: Main_net(
  (bn_prelu_1): BNPReLU(
    (bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
    (acti): PReLU(num_parameters=32)
  )
  (decode1): Sequential(
    (0): Conv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
  )
)
self in forward: Main_net(
  (bn_prelu_1): BNPReLU(
    (bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
    (acti): PReLU(num_parameters=32)
  )
  (decode1): Sequential(
    (0): Conv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU()
  )
)
</code></pre>
<p>You get something different?</p>

username: xiefeiwhu
cooked: <p>Sorry, I initialize the BN with 1e-3 in the main code and forget it, my fault.</p>