Pytorch implemented MLP worse than sklearn MLPRegressor

bpfbpfbpf: <p>Hello Pytorch!</p>
<p>I am new to pytorch, and I’m trying to translate my sklearn MLPRegressor model into pytorch. The in-sample R-squared is better than sklearn, however, the out-of-sample R-squared is horrible.</p>
<p>My database details are as follows:</p>
<ol>
<li>Sample size: ~60k</li>
<li>Feature size: 52 (including binary features)</li>
</ol>
<p>I already did standardization for the features.</p>
<p>MLPRegressor model structure is:</p>
<pre><code class="lang-auto">hyperparams = {
    "hidden_layer_sizes": (64, 32, 16),
    "activation": "relu",
    "solver": "adam",
    "learning_rate_init": 0.01,
    "alpha": 0.01,
    "early_stopping": True,
    "batch_size": 500,
    "max_iter": 1000,
}
MLP_regr = MLPRegressor(**hyperparams)
</code></pre>
<p>the in-sample R2 for this model is ~0.6 and out-of-sample R2 is 0.2.</p>
<p>My PyTorch MLP implementation structure is</p>
<pre><code class="lang-auto">class MLP(nn.Module):
    def __init__(self, in_dim, out_dim=1):
        super(MLP, self).__init__()
        self.layers = nn.Sequential(
            nn.Linear(in_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, 16),
            nn.ReLU(),
            nn.Linear(16, 1)
        )
        
    def forward(self, x):
        return self.layers(x)
</code></pre>
<p>and the training procedure is:</p>
<pre><code class="lang-auto">num_epochs = 1000
batch_size = 500
loss_func = nn.MSELoss()
optimizer = torch.optim.Adam(mlp_torch.parameters(), lr=0.01, weight_decay=0.01)
trainloader = DataLoader(train_dataset, batch_size=batch_size)
testloader = DataLoader(test_dataset, batch_size=batch_size)
mlp_torch = MLP(in_dim=X_train_torch.size(1))

### training step
in_sample_r2_ = []
for epoch in range(num_epochs): 
    in_sample_r2_temp = []
    for id_batch, (X_batch, y_batch) in enumerate(trainloader):
        optimizer.zero_grad()
        y_pred = mlp_torch(X_batch)
        loss = loss_func(y_pred, y_batch.unsqueeze(1))
        loss.backward()
        optimizer.step()
        
        # store in-sample R-squared
        in_sample_r2_temp.append(r2_score(y_batch.detach().numpy(), y_pred.T[0].detach().numpy()))
    in_sample_r2_.append(np.mean(in_sample_r2_temp))
    
    if epoch % 50 == 1:
        print(f"Epoch {epoch}: {loss.item()}, averaged in-sample R-squared is: {np.mean(in_sample_r2_temp)}")
</code></pre>
<p>and the plot of in-sample R2 for my implementation looks like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/56ef52624d130dabdd6be815e4bbfc93292b0b36.png" data-download-href="https://discuss.pytorch.org/uploads/default/56ef52624d130dabdd6be815e4bbfc93292b0b36" title="Screenshot 2023-04-24 at 2.41.14 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56ef52624d130dabdd6be815e4bbfc93292b0b36_2_690x456.png" alt="Screenshot 2023-04-24 at 2.41.14 PM" data-base62-sha1="cp3O2p3GCDCJrNMtU1CmHQQ8GhM" width="690" height="456" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56ef52624d130dabdd6be815e4bbfc93292b0b36_2_690x456.png, https://discuss.pytorch.org/uploads/default/original/3X/5/6/56ef52624d130dabdd6be815e4bbfc93292b0b36.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/6/56ef52624d130dabdd6be815e4bbfc93292b0b36.png 2x" data-dominant-color="E9E5E5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-04-24 at 2.41.14 PM</span><span class="informations">734×486 28.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>After 1000 epochs, my torch MLP achieves 0.61 in-sample R2, and according to the plot, I personally think it learns well. But when I test my model on the testing sets and calculate the out-of-sample R2, it surprisingly gives me a <strong>-0.6</strong>. Here is the code for testing step:</p>
<pre><code class="lang-auto">### testing step
with torch.no_grad():
    out_sample_r2_ = []
    for id_batch, (X_test, y_test) in enumerate(testloader):
        y_pred = mlp_torch(X_test)
        out_sample_r2 = r2_score(y_test.detach().numpy(), y_pred.T[0].detach().numpy())
        out_sample_r2_.append(out_sample_r2)

print(f"Out-of sample R-squared: {np.mean(out_sample_r2_)}")
</code></pre>
<p>I am very confused about this result, it doesn’t seem like overfit and it has the similar result as sklearn does, but why the OOS R-squared too bad? Or did I make mistake to calculate the result? Thank you!!</p>
<p>Best,<br>
-P</p>

eqy: <p>I’m not sure the hyperparameters are identical across the two setups. For example, with <code>early_stopping</code> set to <code>True</code> the <code>MLPRegressor</code> will set aside some fraction of the training data for validation and potentially terminate trainning before <code>max_iter</code>. It looks like it would do the same thing according to the <code>tol</code> hyperparameter (which looks like it would have the default value i nyour case).</p>
<p>If possible, I would check how many iterations the <code>MLPRegressor</code> is actually training for and try limiting your PyTorch implementation to doing the same thing and check if that makes things more compatible. Otherwise, you could try implementing the early stopping/tolerance checks in your PyTorch implementation.</p>

bpfbpfbpf: <p>Hi eqy,</p>
<p>Thank you! The early stopping matters, (and I am surprised PyTorch doesn’t have early stopping function), I implemented my early stop class and now it outperforms sklearn a lot.</p>
<p>Best,<br>
Peiliang</p>