Weird BN configuration

hongildong971: <p>Hello, I am currently researching on non-stationary data. And I finally found a high performance configuration for BN layers. But, the process of those looks quite weird. So I coudn’t figure it out why it shows the high performance.</p>
<p>I’m trying to finetune the BN parameters of the pre-trained model with simple entropy minimization loss.</p>
<p>In this process, I configured the BN layers as <code>track_running_stats=False</code>. Then, I didn’t initialize the running estimates as <code>running_mean=None</code> and <code>running_var=None</code>.</p>
<p>With this configurations, how does the BN layers work for input data in train/eval mode?</p>
<p>I’ve observed that the output of the model in train/eval mode are different in last prediction stage.</p>
<pre><code class="lang-auto">############### Configuration
model.train()
    
model.requires_grad_(False)

for m in model.modules():
if isinstance(m, (nn.BatchNorm1d, nn.BatchNorm2d)):
    m.requires_grad_(True) ###### Affine parameters
    
    m.track_running_stats = False
    # m.running_mean = None
    # m.running_var = None

############### Fintuning
model.train()

optimizer.zero_grad()

outputs = model(x)

loss = softmax_entropy(outputs).mean(0)

loss.backward()

optimizer.step()


############### Predictions
"""
outputs1 and outputs2 are different even for the same input.
"""
model.train()
outputs1 = model(x)

model.eval()
outputs2 = model(x)
</code></pre>

ptrblck: <aside class="quote no-group" data-username="hongildong971" data-post="1" data-topic="194646">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hongildong971/48/63023_2.png" class="avatar"> hongildong971:</div>
<blockquote>
<p>With this configurations, how does the BN layers work for input data in train/eval mode?</p>
</blockquote>
</aside>
<p>From the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html">docs</a>:</p>
<blockquote>
<p>track_running_stats (bool) – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics, and initializes statistics buffers running_mean and running_var as None. When these buffers are None, this module always uses batch statistics. in both training and eval modes. Default: True</p>
</blockquote>

hongildong971: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for your answering.</p>
<p>Following the documents, it initializes the <code>running_mean</code> and <code>running_var</code> as <code>None</code> when we set the <code>track_running_stats=False</code> of BN layers at first time.</p>
<p>But, I’m trying to configure the BN layers after the pre-training stage, which is done with <code>if isinstance(...)</code> in the for loop. So, it doesn’t enter to the <strong>init</strong> function, so that the <code>running_mean</code> and <code>running_var</code> don’t change.</p>
<p>I just change the <code>track_running_stats</code> argument. Thus, it only affects the forward process, I guess.</p>
<p>In addition, I found that the performance also changes if I set <code>m.running_mean=None</code> and <code>m.running_var=None</code> when configuring BN layers.</p>

ptrblck: <p>Are you manipulating <code> track_running_stats</code> after initializing the layer? If so, why are you not setting it during the initialization directly? If you want to keep using the batch stats for normalize the inputs you could leave the layer in training mode.</p>

hongildong971: <p>I just wanna try to fine-tune the parameters in the BN layers. So, I was trying to continuously update the parameters based on the pre-trained weight after loading the pre-trained model.<br>
In addition, manipulating <code>track_running_stats</code> after initializing the layer is common in test-time adaptation algorithm. So, I followed those works.</p>
<p>Ref 1. <a href="https://github.com/DequanWang/tent/blob/master/tent.py" class="inline-onebox" rel="noopener nofollow ugc">tent/tent.py at master · DequanWang/tent · GitHub</a></p>
<p>Ref 2. <a href="https://github.com/TaesikGong/NOTE/blob/main/learner/note.py" class="inline-onebox" rel="noopener nofollow ugc">NOTE/learner/note.py at main · TaesikGong/NOTE · GitHub</a></p>
<p>Unlike those, I just wanna fix the running estimates over all modes of the BN layers<br>
(because the data I’ve used are temporally correlated, so that estimating batch statistics leads to the accumulation of inappropriate statistics)<br>
However, previous works just keep track for input data or don’t track and initialize the <code>running_mean/var</code> as <code>None</code>.</p>
<p>But, in my case, I observed the severe performance degradation when I initialize the <code>running_mean/var</code> as <code>None</code>.<br>
In contrast to that,  I observed the significant performance improvement when I didn’t initialize the <code>running_mean/var</code> as <code>None</code>.</p>
<p>So, I wanna know how the BN layers work in train/eval mode when I manipulate those like <code>track_running_stats=False</code> and don’t initialize <code>running_mean/var</code></p>
<pre><code class="lang-auto">for m in model.modules():
  if isinstance(m, (nn.BatchNorm1d, nn.BatchNorm2d)):
      m.requires_grad_(True)
      
      m.track_running_stats = False
      ### no init running estimates
      # m.running_mean = None
      # m.running_var = None
</code></pre>