How to unnormalize output of Batch Norm

Dark: <p>Hi there,<br>
I am planning on using the Batch Norm layer between the two other layers of my model. After training the model I am not sure how to 'un’normalize the output when I want to receive the predicted values?</p>
<p>If it would be simply normalization on some values - I would just calculate and save the variance and mean, then use these values for 'un’normalizing the output of the model - when I want to get the predictions.</p>
<p>In my view, in the case of batch normalization - since the normalization is done based on the batches and these batches are different at every epoch, these mean and variance values will be different at each epoch - so I dont see a way to save those values for using it for unnormalizing the predicted values…</p>

ptrblck: <p>To get the predictions of the model (assuming you are using a validation or test dataset), you would not “unnormalize” the data, but would use the running stats instead by calling <code>model.eval()</code>.<br>
During training the internal stats (stored in <code>.running_mean</code> and <code>.running_var</code> in batchnorm layers) will be updated with a specified <code>momentum</code> and the current batch stats.</p>

Dark: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your reply.<br>
I had one more question regarding the topic following your answer:<br>
Let’s suppose that my forward method looks like this:</p>
<pre><code class="lang-python">def forward(self, x, hs):
          out, hs = self.lstm(x, hs)
          out = self.batchnormalization(out)
          out = self.fullyconnected(out)
          return out, hs
</code></pre>
<p>Then in my training and validation code:</p>
<pre><code class="lang-python">for e in range(epochs):
        training_loss = 0
        validation_loss = 0
        model.train()
        for x,y training_data:
            train_h  = model.init_hidden(batch_size) 
            train_h = tuple([h.data for h in train_h])  
            out, train_h = model(x, train_h)
            **loss = criterion(out,y)** 
            training_loss += loss.item()
            opt.zero_grad()
            loss.backward()
            optimizer.step()
        training_losses.append(training_loss/len(training_data))

              # validation
                model.eval()
                with torch.no_grad():
                    for x, y in validation_data:
                        val_h  = model.init_hidden(batch_size) 
                        val_h = tuple([h.data for h in val_h])
                        out, val_h = model(x, val_h)
                        **loss = criterion(out, y)**
                        validation_loss += loss.item()
                validation_losses.append(validation_loss/len(validation_data))
</code></pre>
<p>Then in my training and validation code how should I use the running stats that you mentioned? Since as far as I understand, I cant just do loss = criterion(out, y), right? (because the ‘out’ was calculated using the BatchNorm)</p>

ptrblck: <p>The <code>model.train()</code> and <code>model.eval()</code> calls will switch between the training mode (normalizing the input batch with its own stats and updating the running stats) and evaluation mode (normalizing the input batch/sample with the running stats). You don’t need to apply the running stats manually.</p>

Dark: <p>Got it, thanks a lot!</p>