Why we skip initialize running mean and running var while using pretrained resnet50?

two_Two: <p>Hi everyone  I am new to pytorch and there’s one issue that really confuses me</p>
<p>When I try to use transfer learning  and take the resnet50 as base from this link</p>
<p><a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py" rel="nofollow noopener">vision/torchvision/models/resnet.py </a></p>
<p>And download the weight from <a href="https://download.pytorch.org/models/resnet50-19c8e357.pth" rel="nofollow noopener">resnet50-19c8e357.pth</a></p>
<p>Here’s the problems:</p>
<p>While I am initializing the weights from pretrained model I first check if <strong>“all model’s layer match pretrained model’s key”</strong> ,so here’s what I do:</p>
<pre><code class="lang-auto">model_resnet = resnet50() #this is built completely same as  the link above
resnet50_weights = torch.load("resnet50-19c8e357.pth") 

#get all layer name from model_resnet
names = {}
for name,param in model_resnet.named_parameters():
    names[name] = 0

#to see if there's anything missing
for key in resnet50_weights:
    if key not in names:
        print(key)
</code></pre>
<p>And the output be like:</p>
<pre><code class="lang-auto">bn1.running_mean
bn1.running_var
layer1.0.bn1.running_mean
layer1.0.bn1.running_var
layer1.0.bn2.running_mean
layer1.0.bn2.running_var
layer1.0.bn3.running_mean
layer1.0.bn3.running_var
layer1.0.downsample.1.running_mean
layer1.0.downsample.1.running_var
layer1.1.bn1.running_mean
layer1.1.bn1.running_var
layer1.1.bn2.running_mean
layer1.1.bn2.running_var
layer1.1.bn3.running_mean
layer1.1.bn3.running_var
layer1.2.bn1.running_mean
layer1.2.bn1.running_var
layer1.2.bn2.running_mean
layer1.2.bn2.running_var
layer1.2.bn3.running_mean
layer1.2.bn3.running_var
layer2.0.bn1.running_mean
layer2.0.bn1.running_var
layer2.0.bn2.running_mean
layer2.0.bn2.running_var
layer2.0.bn3.running_mean
layer2.0.bn3.running_var
layer2.0.downsample.1.running_mean
layer2.0.downsample.1.running_var
layer2.1.bn1.running_mean
layer2.1.bn1.running_var
layer2.1.bn2.running_mean
layer2.1.bn2.running_var
layer2.1.bn3.running_mean
layer2.1.bn3.running_var
layer2.2.bn1.running_mean
layer2.2.bn1.running_var
layer2.2.bn2.running_mean
layer2.2.bn2.running_var
layer2.2.bn3.running_mean
layer2.2.bn3.running_var
layer2.3.bn1.running_mean
layer2.3.bn1.running_var
layer2.3.bn2.running_mean
layer2.3.bn2.running_var
layer2.3.bn3.running_mean
layer2.3.bn3.running_var
layer3.0.bn1.running_mean
layer3.0.bn1.running_var
layer3.0.bn2.running_mean
layer3.0.bn2.running_var
layer3.0.bn3.running_mean
layer3.0.bn3.running_var
layer3.0.downsample.1.running_mean
layer3.0.downsample.1.running_var
layer3.1.bn1.running_mean
layer3.1.bn1.running_var
layer3.1.bn2.running_mean
layer3.1.bn2.running_var
layer3.1.bn3.running_mean
layer3.1.bn3.running_var
layer3.2.bn1.running_mean
layer3.2.bn1.running_var
layer3.2.bn2.running_mean
layer3.2.bn2.running_var
layer3.2.bn3.running_mean
layer3.2.bn3.running_var
layer3.3.bn1.running_mean
layer3.3.bn1.running_var
layer3.3.bn2.running_mean
layer3.3.bn2.running_var
layer3.3.bn3.running_mean
layer3.3.bn3.running_var
layer3.4.bn1.running_mean
layer3.4.bn1.running_var
layer3.4.bn2.running_mean
layer3.4.bn2.running_var
layer3.4.bn3.running_mean
layer3.4.bn3.running_var
layer3.5.bn1.running_mean
layer3.5.bn1.running_var
layer3.5.bn2.running_mean
layer3.5.bn2.running_var
layer3.5.bn3.running_mean
layer3.5.bn3.running_var
layer4.0.bn1.running_mean
layer4.0.bn1.running_var
layer4.0.bn2.running_mean
layer4.0.bn2.running_var
layer4.0.bn3.running_mean
layer4.0.bn3.running_var
layer4.0.downsample.1.running_mean
layer4.0.downsample.1.running_var
layer4.1.bn1.running_mean
layer4.1.bn1.running_var
layer4.1.bn2.running_mean
layer4.1.bn2.running_var
layer4.1.bn3.running_mean
layer4.1.bn3.running_var
layer4.2.bn1.running_mean
layer4.2.bn1.running_var
layer4.2.bn2.running_mean
layer4.2.bn2.running_var
layer4.2.bn3.running_mean
layer4.2.bn3.running_var
</code></pre>
<p>As far as I know,when we freeze the pretrained model,one of issue is that we need to be careful about either using batch statistics or running mean&amp;var that calculated while using transfer learning ,I think we should use the running mean&amp;var as we “freeze” the model like the link below also mentioned:<br>
<a href="http://blog.datumbox.com/the-batch-normalization-layer-of-keras-is-broken/" rel="nofollow noopener">The Batch Normalization layer of Keras is broken</a></p>
<p>Anyone has any idea how do we get running_mean&amp;var?</p>

ptrblck: <p>You are only saving the parameters in these lines of code:</p>
<pre><code class="lang-python">names = {}
for name,param in model_resnet.named_parameters():
    names[name] = 0
</code></pre>
<p>while the running estimates are stored as <code>buffers</code>.<br>
You could append these buffers using:</p>
<pre><code class="lang-python">for name, buf in model_resnet.named_buffers():
    names[name] = 0
</code></pre>
<p>and run your code again.</p>

two_Two: <p>Wow,Thanks , it works!<br>
However as I initialize model by using</p>
<pre><code class="lang-auto">#fc layer has been deleted here
for key in resnet50_weights:
    model_resnet.state_dict()[key] = resnet50_weights[key]
</code></pre>
<p>And I do a little test to check initialization is done properly.</p>
<pre><code class="lang-auto">print(model_resnet.state_dict()["layer4.2.bn3.running_mean"]==resnet50_weights["layer4.2.bn3.running_mean"])
</code></pre>
<p>And the output be like:</p>
<pre><code class="lang-auto">tensor([False, False, False,  ..., False, False, False])
</code></pre>
<p>Which implies initialization is wrong ,But why does it fail?</p>

ptrblck: <p>I would recommend to load the <code>state_dict</code>, which will automatically map the right parameters and buffers via:</p>
<pre><code class="lang-python">model.load_state_dict(pretrained_model.state_dict())
</code></pre>
<p>or do you really need to manually load each parameter and buffer?</p>

two_Two: <p>I use this to load the pretrained weights:</p>
<p><code>resnet50_weights = torch.load("resnet50-19c8e357.pth")</code></p>
<p>so</p>
<pre><code class="lang-auto">model_resnet.load_state_dict(resnet50_weights.state_dict())
</code></pre>
<p>would raise error:</p>
<pre><code class="lang-auto">'collections.OrderedDict' object has no attribute 'state_dict'
</code></pre>
<p>As for    <code>Do you really need to manually load each parameter and buffer?</code><br>
Actually no, I am new to pytorch is there’s a better way to do what I want ,I would always prefer it! But for curiosity sake ,I still wonder why  it fails…</p>

ptrblck: <p>Based on the error message, it seems <code>resnet50_weight</code> might already be the <code>state_dict</code>. Could you try to load them as <code>model_resnet.load_state_dict(resnet50_weights)</code>?</p>

two_Two: <p>Thanks!It works but what if I want to initialize the model except the fc layer ,I<br>
I think in this case we have to assign each parameter and buffer or is there an alternative?</p>

ptrblck: <p>I would just load the complete model and just reinitialize the last layer afterwards. It seems to be simpler than the other way around. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

two_Two: <p>What if I want to discard the fc layer? How can I delete fc layer after using</p>
<pre><code class="lang-auto">model_resnet.load_state_dict(resnet50_weights)
</code></pre>
<p>I think this might work</p>
<pre><code class="lang-auto">model_resnet = nn.Sequential(*list(model_resnet.children())[:-2])
</code></pre>

ptrblck: <p>Your approach might work for this model.<br>
However, if your model’s <code>forward</code> method is a bit more complicated than a simple sequence of modules, you could also set <code>model_resnet.fc = nn.Identity()</code>.</p>

two_Two: <p>HI, just making sure if everything is right</p>
<p>in <code>forward</code> function in resnet50 I delete the <code>flatten()</code></p>
<pre><code class="lang-auto">def forward(self, x):
   x = self.conv1(x)
   x = self.bn1(x)
   x = self.relu(x)
   x = self.maxpool(x)
   x = self.layer1(x)
   x = self.layer2(x)
   x = self.layer3(x)
   x = self.layer4(x)
   x = self.avgpool(x)
   #x = torch.flatten(x)
   x = self.fc(x)
       return x
</code></pre>
<p>And after constructing the model ,I do what you said</p>
<pre><code class="lang-auto">model_resnet = resnet50()
resnet50_weights = torch.load("resnet50-19c8e357.pth")
model_resnet.load_state_dict(resnet50_weights)
model_resnet.fc = nn.Identity()
model_resnet.avgpool = nn.Identity()
</code></pre>
<p>And without <code>flatten</code>, I think the output shape would right now be same as last residual block output shape,if you find anything wrong above,please do correct me .Thanks!</p>

ptrblck: <p>The <code>Identity</code> assignment would be a hack in case you don’t want to manipulate the <code>forward</code>.<br>
Since you are changing it, just comment out the calls to <code>self.avgpool</code> and <code>self.fc1</code> and check the shape of the output after passing some random input to the model.</p>

two_Two: <p>OK, I’ve got desired shape,Thanks!!!<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
BTW, I was browsing around and I found that in this link<br>
<a href="https://discuss.pytorch.org/t/bytetensor-to-floattensor-is-slow/3672/3">ByteTensor to FloatTensor is slow? - PyTorch Forums</a><br>
Why do people always convert tensor’s type to float while calculating, would that be faster?</p>

ptrblck: <p>The default <code>dtype</code> is float32, which works faster in CUDA than float64.<br>
Usually you don’t need the precision and range of FP64, and can stick to FP32.<br>
However, Tensorcores in newer NVIDIA GPUs can accelerate FP16 operations.<br>
Since the value range and precision might be too narrow for some models using this number format, we developed <a href="https://github.com/NVIDIA/apex">NVIDIA/apex</a> containing mixed precision recipes and are currently working on upstreaming it into PyTorch in <a href="https://github.com/pytorch/pytorch/issues/25081">this issue</a>.</p>

two_Two: <p>Yeah,would definitely follow it,but what about int,are there reasons not using int?Thanks!</p>

ptrblck: <p>I’m not familiar with training methods using int values, since the gradients would also be int, wouldn’t they?<br>
Based on this assumption, it sounds not really useful, but I haven’t looked for the latest research papers on this topic.<br>
Inference however is possible using int values and might give you a performance gain on specialized hardware. Have a look at <a href="https://pytorch.org/docs/stable/quantization.html">Quantization</a> to get more information.</p>

two_Two: <p>Yeah would definitely check out,And just a little to ask:The situation is I would like to freeze the pretrained model,as the link below has already showed how you can completely freeze bn layer while training:<br>
<a href="https://discuss.pytorch.org/t/bug-weird-behavior-between-evaluation-and-training-mode/13297/17">freeze bn </a></p>
<p>But i what’s confuses me is these lines of codes:</p>
<pre><code class="lang-auto">if freeze_bn_affine:
    '''Freezing Weight/Bias of BatchNorm2D'''
    m.weight.requires_grad = False
    m.bias.requires_grad = False
</code></pre>
<p>So since I have already set the bn layers to eval() stage,why do I still need this?<br>
And after this should I need to set the <code>affine= False</code> manually ?</p>

ptrblck: <p><code>train</code> and <code>eval</code> will change the behavior of the running estimates (using batch stats and updating running stats in <code>train</code>, while using running estimates in <code>eval</code>).<br>
This is unrelated to the trainable affine parameters (<code>weight</code> and <code>bias</code>), so you should freeze these parameters, if you don’t want to train them any further.</p>

two_Two: <p>So after setting weights and bias <code>requires_grad=False</code> I do not have to do <code>BatchNorm2d(3,affine=False)</code>every single time ,right?</p>

ptrblck: <p>Your line of code would re-initialize the batchnorm layer, which seems to be wrong, as you’ll lose all trained parameters and estimated stats.<br>
So no, don’t use this line of code, just freeze the parameters using their <code>requires_grad</code> flag.</p>