How to get the batch mean and variance inside BatchNorm?

chengyangfu: <p>Hi,<br>
I am trying to implement a variation of BatchNorm and want to use the current code first.<br>
My problem is how to find the batch_mean and batch_std inside the current BatchNorm module.</p>
<p>In the old implementation torch/legacy/nn/BatchNormalization.py:<br>
There are save_mean and save_std parameters are generated after forward pass.  However, these two parameters are removed in the current version. Is there any way I can achieve these two parameters in the current version?</p>
<pre><code>Thanks!</code></pre>

ptrblck: <p>You could get the running mean and var (exponentially weighted average from last batches) with <code>bn.running_mean</code> and <code>bn.running_var</code>, if that is what you need.</p>

chengyangfu: <p>Hi Thanks!<br>
But actually I want the batch_mean and batch_var which were used during the training.<br>
I am implementing the batch renomalization paper <a href="https://arxiv.org/abs/1702.03275" rel="nofollow noopener">https://arxiv.org/abs/1702.03275</a>. Both batch_mean,  and running_mean are used during training. Is there any way to access both variables? I am pretty sure they are stored in some way to accelerate the backward propagation.</p>

ptrblck: <p>Ah ok! Sorry, I misunderstood your question.<br>
I think the current <code>gamma</code> and <code>beta</code> values are stored in <code>bn.weight</code> and <code>bn.bias</code>, respectively.</p>

chengyangfu: <p>Unfortunately, these are not what I am looking for too.</p>
<pre><code>def batch_norm(input, running_mean, running_var, weight=None, bias=None,
               training=False, momentum=0.1, eps=1e-5):
    f = torch._C._functions.BatchNorm(running_mean, running_var, training, momentum, eps, torch.backends.cudnn.enabled)
    return f(input, weight, bias)
</code></pre>
<p>Inside the batch_norm function,  <code>torch._C._functions.BatchNorm</code> calculates the batch_mean and batch_var first, and then use them to normalize the batch and update the <code>running_mean</code> and <code>running_var</code>. These two parameters are stored in inside the function. I am wondering is there any easy way I can access this two parameters?</p>

ptrblck: <p>Ah yeah, I see. My fault <img src="//discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"></p>

XinDongol: <p>Have you found the way?</p>

chengyangfu: <p>No, there is no easy to do it. Due to the performance, the batch mean/std calculation is inside the cuDNN and used to update the running mean/std directly.<br>
The simple way to do it is running a batch norm module but set it in the evaluation mode. Then you can get the batch mean/std easily in the training.</p>

XinDongol: <p>Thanks. You can also use this to compute them by yourself.</p>
<pre><code class="lang-python">self.mu = tmp.transpose(0,1).contiguous().view(c,-1).mean(1)
self.var = tmp.transpose(0,1).contiguous().view(c,-1).var(1)
</code></pre>

gcucurull: <p>What I do is to use a <code>hook</code> to inspect the input and output to the batchnorm layer, and I compute the mean and variance of the input to the layer (which should be roughly the same to the one computed by <code>torch._C._functions.BatchNorm</code>).</p>
<p>Example:</p>
<pre><code class="lang-auto"># 1. define model
model = ...

# 2. register the hook
model.bn_layer.register_forward_hook(printbn)

# 3. the hook function will be called here
model.forward(...)
</code></pre>
<p>and the <code>hook</code> can be defined as:</p>
<pre><code class="lang-auto">def printbn(self, input, output):
    print('Inside ' + self.__class__.__name__ + ' forward')
    mean = input[0].mean(dim=0)
    var = input[0].var(dim=0)
    print(mean)
</code></pre>

wangdongxu: <pre><code class="lang-python">batch_mean = torch.mean(input, dim=[0, 2, 3])
batch_var = torch.var(input, dim=[0, 2, 3], unbiased=False)
</code></pre>

OMAR_ENNAYRI: <p><span class="hashtag-raw">#Creating</span> the model with 3 convultional layers and 3 pooling layers, 1 fully connected layers.<br>
class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(in_channels=3, out_channels=53, kernel_size=4, stride=2, padding=3)<br>
self.pool = nn.MaxPool2d(2, 2)<br>
self.conv2 = nn.Conv2d(53, 106, kernel_size=4, stride=2, padding=3)<br>
self.conv3 = nn.Conv2d(106, 212, kernel_size=4, stride=2, padding=3)<br>
self.fc1 = nn.Linear(212<em>3</em>3, 1908)<br>
self.fc2 = nn.Linear(1908, 8)</p>
<pre><code>def forward(self, x):

    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = self.pool(F.relu(self.conv3(x)))
    x = torch.flatten(x, 1) 
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x
</code></pre>
<p>device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”) # PyTorch v0.4.0<br>
model = Net().to(device)<br>
summary(model, (3, 144, 144))</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b451be165a68cfe728ee10a2f0a3c13fa4cc3b4e.png" data-download-href="https://discuss.pytorch.org/uploads/default/b451be165a68cfe728ee10a2f0a3c13fa4cc3b4e" title="pythorch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b451be165a68cfe728ee10a2f0a3c13fa4cc3b4e.png" alt="pythorch" data-base62-sha1="pJb4MUBgzMQDcLXEDwXO6LTNTZA" width="690" height="248" data-dominant-color="F0D0D1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pythorch</span><span class="informations">949×342 15.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The <code>in_features</code> of <code>self.fc1</code> doesn’t match the feature size of <code>x</code>, so you would need to fix it.<br>
I also don’t know how your post is related to this topic.</p>

OMAR_ENNAYRI: <p>what is the correction for it, how will i match the features with the size x</p>