Performance difference when using BatchNorm1d vs StandardScaler

ic_fly2: <p>I have a basic classification model:</p>
<pre><code class="lang-auto">class BNN(nn.Sequential):
    def __init__(self):
        super().__init__()
        self.add_module("Linear", nn.Linear(len(columns), 40))
        self.add_module("ReLU", nn.ReLU(inplace=True))
        self.add_module("Linear2", nn.Linear(40, 1))
        self.add_module("Sigmoid", nn.Sigmoid())
</code></pre>
<p>and if i train and then evaluate the model with data that I have applied the sklearn preprocessor to:</p>
<pre><code class="lang-auto">scaler = preprocessing.StandardScaler()
train_scaled = scaler.fit_transform(train[columns])
test_scaled = scaler.transform(test[columns])
evaluate_scaled = scaler.transform(evaluate[columns])
</code></pre>
<p>the performance is fine / great.<br>
However if I turn the model into:</p>
<pre><code class="lang-auto">class BNN(nn.Sequential):
    def __init__(self):
        super().__init__()
        self.add_module("Norm", nn.BatchNorm1d(len(columns))) # Layer in question
        self.add_module("Linear", nn.Linear(len(columns), 40))
        self.add_module("ReLU", nn.ReLU(inplace=True))
        self.add_module("Linear2", nn.Linear(40, 1))
        self.add_module("Sigmoid", nn.Sigmoid())
</code></pre>
<p>and don’t preprocess the data, the performance degrades very significantly.<br>
Is this possible because the batchedNorm only sees batches and would be expected to under perform (a bit) or am I using it wrong? Do i need to do something special because I have balanced training data (upsampled) but imbalanced test and evaluation data (1% positive class)</p>
<p>How would a network that just learned the BatchNorm1d layer look?</p>

ic_fly2: <p>I’ve made a simple test and it confirms what I though. How do i train just a Normalisation layer?</p>
<pre><code class="lang-auto">

scaler = preprocessing.StandardScaler()
sk_train_scaled = scaler.fit_transform(train[columns])
sk_test_scaled = scaler.transform(test[columns])
sk_evaluate_scaled = scaler.transform(evaluate[columns])

train_scaled = train[columns].values
test_scaled = test[columns].values
evaluate_scaled = evaluate[columns].values


x_train = torch.tensor(train_scaled, dtype=torch.float)
y_train = torch.tensor(train[['y']].values, dtype=torch.float)

x_test = torch.tensor(test_scaled, dtype=torch.float)
y_test = torch.tensor(test[['y']].values, dtype=torch.float)

x_evaluate = torch.tensor(evaluate_scaled, dtype=torch.float)
y_evaluate = torch.tensor(evaluate[['y']].values, dtype=torch.float)


# In[7]:


batch_size = int(len(x_train)/100)
dataset = data.TensorDataset(x_train.to(DEVICE), y_train.to(DEVICE))
loader = data.DataLoader(dataset, batch_size=batch_size)


# In[8]:


class TorchScaler(nn.Sequential):
      def __init__(self):
        super().__init__()
        self.add_module("Norm", nn.BatchNorm1d(len(columns)))  


# In[32]:


ts = TorchScaler()


# In[33]:


ts.train()


# In[35]:


sk_train_scaled.std(), sk_train_scaled.mean()


# In[36]:


tx = ts.forward(x_train)
tx.std(), tx.mean()


# In[37]:


ts.eval()


# In[38]:


tx2 = ts.forward(x_train)
tx2.std(),  tx2.mean()


# In[39]:


sk_test_scaled.std(), sk_test_scaled.mean()


# In[40]:


txt = ts.forward(x_test)
txt.std(),  txt.mean()


</code></pre>
<p>and these are the outputs:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aaf514b9cd2fcf9d7072cd513f89e5325466863.png" data-download-href="https://discuss.pytorch.org/uploads/default/7aaf514b9cd2fcf9d7072cd513f89e5325466863" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7aaf514b9cd2fcf9d7072cd513f89e5325466863.png" alt="image" data-base62-sha1="hvjRMNPkPDIQwrAPPYNM3vIvGkX" width="662" height="500" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">852×643 16.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>