Minimum value of Intermediate BatchNorm layer output clamps to 0

god_sp33d: <p>Hi,</p>
<p>I am doing an experiment where I use the output of intermediate values of batchnorm layer from resnet50. So, I forward hook at different layers and work on the output. But, the weird thing I observed is the min value of the batchnorm output clamps to zero at all the layers. To reproduce the result I wrote a toy program:</p>
<pre><code class="lang-auto">import torch
from torchvision import models, datasets
import torchvision
import datasets
from tqdm import tqdm, trange


activation_student = {}
def get_activation_student(name):
    def hook(self, input_, output):
        activation_student[name]=  output.data
    return hook

amazonData = datasets.amazon_load('train', 16)
model = models.resnet50(pretrained=True)
for n, m in model.named_modules():
    if n == 'layer2.1.bn1' or 'layer3.2.bn2' or 'layer4.1.bn3':
        m.register_forward_hook(get_activation_student(n))

for i in trange(5, leave=False):
    source_x, source_y = next(iter(amazonData))
    out = model(source_x)
    print("min: ", torch.min(activation_student['layer2.1.bn1']))
    print("min: ", torch.min(activation_student['layer3.2.bn2']))
    print("min: ", torch.min(activation_student['layer4.1.bn3']))

</code></pre>
<p>output:<br>
min: tensor(0.)<br>
min: tensor(0.)<br>
min: tensor(0.)</p>
<p>can someone help me in fixing this, or is this expected ?</p>

god_sp33d: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> any idea?</p>
<p>P.S. I apologize for tagging.</p>

ptrblck: <p>The activation values are a bit tricky in this use case.<br>
As you can see in <a href="https://github.com/pytorch/vision/blob/8a64dbcd2ecb5d55b45e1946dd8dac56307e60be/torchvision/models/resnet.py#L96" rel="nofollow noopener">this line of code</a>, the <code>Bottleneck</code> module uses <code>self.relu</code> directly after the batch norm layers.<br>
Since <code>self.relu</code> is defined as an inplace operation, the output of the batchnorm layers will be manipulated inplace, which is why you actually see the “output” of the <code>self.relu</code> layer.</p>

god_sp33d: <p>Hi,</p>
<p>Thanks for your reply. I have gone through the docs. Had it been ‘layer2.1.bn3’ then I am convinced with your reason as there is a relu operation following it but after the batchnorm layer ‘layer2.1.bn1’ there is no relu operation. In resnet “conv-bn-conv-bn-conv-bn-relu” is the pattern followed and I am looking at the output of the first batchnorm where there is no ReLU(inplace=True). I hope I made it clear ?</p>
<p>But still, I get minimum value as “0” ? or am I missing something ?</p>

ptrblck: <p>The output of the <code>Bottleneck</code> layer might be misleading:</p>
<pre><code class="lang-python">model.layer2[1]
Out[42]: 
Bottleneck(
  (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace)
)
</code></pre>
<p>This output does not mean, that these modules will be executed sequentially, as they are not wrapped in an <code>nn.Sequential</code> module.<br>
In fact the output just lists all modules as they were initialized in <code>__init__</code>, while the <code>forward</code> method defines the execution.<br>
As you can see from the link I’ve posted, <code>self.relu</code> is used three times inside the <code>Bottleneck</code> module (every time operating on the activation inplace).</p>

god_sp33d: <p>That was very clear and to the point. Thanks a lot.</p>
<p>One final question, is there a way I could hook somehow and get the output of the batchnorm layer and not the relu output ? I cannot use conv layer’s output because my loss function involves a margin  where I can set the margin to one std. dev if only I get output of batchnorm layer.</p>
<p>Thanks again.</p>

ptrblck: <p>The clean approach would be to derive a custom class from <code>ResNet</code> and use vanilla <code>nn.ReLU</code> modules instead of the inplace version.</p>
<p>The hacky way would be to just replace the “unwanted” inplace ReLUs:</p>
<pre><code class="lang-python">model = models.resnet50(pretrained=True)
model.layer2[1].relu = nn.ReLU()
model.layer3[2].relu = nn.ReLU()
model.layer4[1].relu = nn.ReLU()
for n, m in model.named_modules():
    if n == 'layer2.1.bn1' or 'layer3.2.bn2' or 'layer4.1.bn3':   
        m.register_forward_hook(get_activation_student(n))
</code></pre>

god_sp33d: <p><img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> Thanks my issue is solved. I will mark the above comment as solution.</p>