Model stuck at high loss value after the first epochs

username: Nidhal_Baccouri
cooked: <p>Hi I’m confused, I’m using pytorch for my project but my Model is not learning well. the loss start with a very high value (40000), then after the first 5-10 epochs the loss decrease rapidly to 6000-7000 and then it stuck there, no matter what I make. I tried even to change to skorch instead of pytorch so that I can use cross validation functionality but that also didn’t help. I tried different optimizers and added layers and neurons to the network but that didn’t help, it stuck at 6000 which is a very high loss value. I’m doing regression here, I have 10 features and I’m trying to predict one continuous value. that should be easy to do that’s why it is confusing me more.</p>
<p>here is my network:</p>
<blockquote>
<p>class BearingNetwork(nn.Module):<br>
def <strong>init</strong>(self, n_features=X.shape[1], n_out=1):<br>
super().<strong>init</strong>()<br>
self.model = nn.Sequential(</p>
<pre><code>        nn.Linear(n_features, 512), 
        nn.BatchNorm1d(512),
        nn.LeakyReLU(),
        nn.Linear(512, 64),
        nn.BatchNorm1d(64),
        nn.LeakyReLU(),
        nn.Linear(64, n_out),
    )
</code></pre>
</blockquote>
<p>and here are my settings:</p>
<blockquote>
<p>R2 = EpochScoring(r2_score, lower_is_better=False, name=‘R2’)<br>
explained_var_score = EpochScoring(EVS, lower_is_better=False, name=‘EVS Metric’)<br>
custom_score = make_scorer(RMSE)   # making root mean squared error as a custom metric<br>
rmse = EpochScoring(custom_score, lower_is_better=True, name=‘rmse’)</p>
<p>bearing_nn = NeuralNetRegressor(</p>
<pre><code>BearingNetwork,
criterion=nn.MSELoss,
optimizer=optim.Adam,
optimizer__amsgrad=True, # I also tried Adam without the amsgrad
max_epochs=5000,
batch_size=128,
lr=0.001,
train_split=skorch.dataset.CVSplit(10),
callbacks=[R2, explained_var_score, rmse, Checkpoint(), EarlyStopping(patience=100)],
device=device
</code></pre>
<p>)</p>
</blockquote>
<p>I also standardize the Input values. my Input have the shape:</p>
<blockquote>
<p>torch.Size([39006, 10])</p>
</blockquote>
<p>and shape of output is:</p>
<blockquote>
<p>torch.Size([39006, 1])</p>
</blockquote>
<p>I’m using 128 as my Batch_size but I also tried other values like 32, 64, 512 and even 1024. Although normalizing output is not necessary but I also tried that and It didn’t work when I predict values, the loss is high. Please someone help me on this, I would appreciate every helpful advice</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1b3865b0c02821ed6d47c7aec9c217593b2d75b.png" data-download-href="https://discuss.pytorch.org/uploads/default/e1b3865b0c02821ed6d47c7aec9c217593b2d75b" title="loss_epochs"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1b3865b0c02821ed6d47c7aec9c217593b2d75b_2_378x500.png" alt="loss_epochs" data-base62-sha1="wcE3MypFi5SDy5v661RPNqZTWZl" width="378" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1b3865b0c02821ed6d47c7aec9c217593b2d75b_2_378x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/1/e1b3865b0c02821ed6d47c7aec9c217593b2d75b_2_567x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/1/e1b3865b0c02821ed6d47c7aec9c217593b2d75b.png 2x" data-dominant-color="F0F0F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_epochs</span><span class="informations">722×954 258 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>