How to do finetune traning with BatchNorm2d inside?

JohnHerry: <p>python3.10 and torch2.0.1+cu118<br>
I have a model to finetune, the model has encoder and decoder part.  I want only to finetune the decoder part.  So I use code follows to freeze layers except the decoder</p>
<pre><code class="lang-auto">decoder_layer_tag = ["decoder", "dec_lstm"]
for name, para in pretrained_base_model.named_parameters():
    name_tag = name.split(".")[0]
    if name_tag not in decoer_layer_tag:
        para.require_grad = False
</code></pre>
<p>the training process is OK,  but during the inference, I found that the  encoder output from the pretrained_base_model  is different from the encoder output of the finetuned model.<br>
I found that the different output comes from the output of submodule inside encoder like this:</p>
<pre><code class="lang-auto"> sequential([torch.nn.Conv2d(), torch.nn.BatchNorm2d(), torch.nn.ReLU(inplace=True)])
    
</code></pre>
<p>the output of torch.nn.BatchNorm2d is different.  I compared the weight here in these two models, they are the same. So I guess that even I had freeze the weight inside the BatchNorm2d,  its self.training still be true and something inside had changed. I am not formilla with this BatchNorm2d inside.  so need some help about it here.</p>

JohnHerry: <p>Is that wokrs?</p>
<pre><code class="lang-auto">class Model(nn.Moduel):
     def __init__(self):
          self.encoder = EncodeCls()
          self.decoder = DecoderCls()
          ....
     def forward(self,  x, ....):
          self.encoder.eval()
          x = self.encoder(x)
          x' = self.decoder(x)
          return x'
</code></pre>

JohnHerry: <p>It seems that inside the BatchNorm2d in the encoder part , those parameters are changed during finetune:<br>
conv.1.runing_mean<br>
conv1.runing_war<br>
conv1.num_batches_tracked<br>
…</p>

JohnHerry: <p><a href="https://stackoverflow.com/questions/70259900/pytorch-running-mean-running-var-and-num-batches-tracked-are-updated-during-tra" rel="noopener nofollow ugc">it workes</a>,  It is strange of the BatchNorm2d design when it’s weights are frozen and its registed variables are still update with the training process.</p>

ptrblck: <p>This behavior is expected, since the <code>running_</code> stats are not trainable parameters (and they thus also won’t receive any gradients), but buffers which are updated in the forward pass if the model is in training mode. Calling <code>model.eval()</code> will not “freeze” any parameters, but just change the models behavior (e.g. dropout will be disabled and batchnorm layers will use their running stats to normalize the incoming input activation).</p>

JohnHerry: <p>Yes, model.eval() will call  model.train(False) recursively, we can call this as sequential.eval(), and as a result, the update of registed_buffers in the BatchNorm2d are stopped. that is the state what we really want when we do freeze.<br>
if we do not call eval() to stop updating, the conv weights inside BatchNorm2d are stay unchanged, but its output are different during training bacased of the update of registed_buffers.<br>
I guess we need a module member just named  freeze(), when we call it , all module parameters or buffer values are stay unchanged all the time, even when the traning=True.  Now there is no such member, so we should take care that the being “eval()” d submodule will be opened for training again after a validation interval.</p>