Constant Predictions in Non-Linear Model Despite Training Progress

MrSchiller: <p>Hello,</p>
<p>I’m working on a non-linear problem using a simple Neural Network model, and I’m encountering a strange issue with my predictions. The model is designed to predict a single output parameter (<code>delta_sigma</code>) based on two input parameters (<code>sigma_t</code> and <code>delta_epsilon</code>). While training, I’ve normalized the data and tested different hyperparameters, but I’m still getting constant prediction values, no matter the inputs.</p>
<h3><a name="p-471755-problem-description-1" class="anchor" href="#p-471755-problem-description-1"></a>Problem Description:</h3>
<ul>
<li>The model appears to learn initially, as I see the loss function decrease over time. Both the training loss and validation loss decrease significantly, which seems normal.</li>
<li>However, after some time, the predictions stabilize and remain constant (even though I change the input values). For example, it’s usually around a value like <code>1000</code>, but sometimes it fluctuates. But right now, the output is just constant.</li>
<li>I’ve tried experimenting with different hyperparameters, like changing the learning rate or network architecture, but it hasn’t resolved the issue. The model keeps outputting the same value for every input, regardless of changes.</li>
</ul>
<h3><a name="p-471755-steps-ive-taken-2" class="anchor" href="#p-471755-steps-ive-taken-2"></a>Steps I’ve Taken:</h3>
<ul>
<li>I started with a simple code and gradually added more complexity to explore potential causes, such as overfitting or issues with normalization.</li>
<li>I’ve also seen posts mentioning data processing issues as potential causes of this problem, but I can’t seem to pinpoint what might be wrong with my data processing pipeline.</li>
<li>The dataset is small, and even though the loss looks good (both value loss and overall loss), the final predictions are all identical.</li>
</ul>
<h3><a name="p-471755-request-3" class="anchor" href="#p-471755-request-3"></a>Request:</h3>
<ul>
<li>I’d appreciate any help in identifying what might be going wrong.</li>
<li>I’m attaching a link to my <a href="https://github.com/BesT1CooKie1/pina_oedometer/blob/master/README.md" rel="noopener nofollow ugc">GitHub repository</a> where you can find the essential details in the README (the notebook specifically explains the process and shows what happens in the last iteration).</li>
<li>The files themselves aren’t necessary to look at; the README should be enough to understand the issue.</li>
</ul>
<h3><a name="p-471755-important-code-blocks-4" class="anchor" href="#p-471755-important-code-blocks-4"></a>Important Code Blocks:</h3>
<p>Here are some key pieces of the code that might help in understanding the issue:</p>
<h4><a name="p-471755-model-definition-5" class="anchor" href="#p-471755-model-definition-5"></a>Model Definition:</h4>
<pre><code class="lang-auto">class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(LSTMModel, self).__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)
        self.relu = nn.ReLU()

    def forward(self, x):
        _, (hn, _) = self.lstm(x)
        out = self.fc(hn[-1])
        return out
</code></pre>
<p>This LSTM-based architecture is simple and designed to predict <code>delta_sigma</code> from the inputs <code>sigma_t</code> and <code>delta_epsilon</code>.</p>
<h4><a name="p-471755-data-normalization-6" class="anchor" href="#p-471755-data-normalization-6"></a>Data Normalization:</h4>
<pre><code class="lang-auto">def min_max_normalize(tensor):
    """
    Normalizes the tensor using global min_val and max_val.
    """
    global min_val, max_val
    if min_val is None or max_val is None:
        min_val = torch.min(tensor)
        max_val = torch.max(tensor)
    return (tensor - min_val) / (max_val - min_val)
</code></pre>
<p>Data normalization ensures that all inputs are on the same scale. However, the output still seems to stay constant despite this normalization.</p>
<h4><a name="p-471755-training-loop-with-early-stopping-7" class="anchor" href="#p-471755-training-loop-with-early-stopping-7"></a>Training Loop with Early Stopping:</h4>
<pre><code class="lang-auto">best_val_loss = float('inf')
patience = 20
patience_counter = 0
epochs = 10000

for epoch in range(epochs):
    model.train()
    running_loss = 0.0

    for i, (x_batch, y_batch) in enumerate(train_dataloader):
        optimizer.zero_grad()
        outputs = model(x_batch)
        loss = criterion(outputs, y_batch)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

    avg_train_loss = running_loss / len(train_dataloader)

    if improvement_block:
        model.eval()
        val_loss = 0.0
        with torch.no_grad():
            for x_batch_val, y_batch_val in val_dataloader:
                val_outputs = model(x_batch_val)
                loss = criterion(val_outputs, y_batch_val)
                val_loss += loss.item()

        avg_val_loss = val_loss / len(val_dataloader)

        if avg_val_loss &lt; best_val_loss:
            best_val_loss = avg_val_loss
            patience_counter = 0
        else:
            patience_counter += 1

        if patience_counter &gt;= patience:
            print(f"Early stopping after {epoch + 1} epochs due to no improvement.")
            break

    if (epoch + 1) % 10 == 0:
        print(f'Epoch {epoch+1}/{epochs}, Train Loss: {avg_train_loss:.6f}, Validation Loss: {avg_val_loss:.6f}')
</code></pre>
<p>This loop includes early stopping based on validation loss, but even with this setup, I end up with constant predictions after training.</p>
<h4><a name="p-471755-prediction-block-8" class="anchor" href="#p-471755-prediction-block-8"></a>Prediction Block:</h4>
<pre><code class="lang-auto">predicted_delta_sigma, true_delta_sigma = predict_oedometer(
    model,
    example_sigma_t_input,
    example_delta_epsilon_input,
    min_val,
    max_val,
    normalize=normalize
)
</code></pre>
<p>Here, I’m making predictions with the trained model. Despite a well-decreasing loss, the predictions end up being constant.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b2cfdf472c915450da681bebbce72bb9eb1c613.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4b2cfdf472c915450da681bebbce72bb9eb1c613" title="loss_pred-vs-true"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b2cfdf472c915450da681bebbce72bb9eb1c613_2_690x279.jpeg" alt="loss_pred-vs-true" data-base62-sha1="aJ2cISqIKDmZ3nryCZYvBlJE6lR" width="690" height="279" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b2cfdf472c915450da681bebbce72bb9eb1c613_2_690x279.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b2cfdf472c915450da681bebbce72bb9eb1c613_2_1035x418.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b2cfdf472c915450da681bebbce72bb9eb1c613_2_1380x558.jpeg 2x" data-dominant-color="EEF4F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_pred-vs-true</span><span class="informations">1640×664 101 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h3><a name="p-471755-conclusion-9" class="anchor" href="#p-471755-conclusion-9"></a>Conclusion:</h3>
<p>I’m still relatively new to the field and am learning together with my professors, but I’ve hit a wall. I would be really grateful for any insights or suggestions on where I might be going wrong.</p>
<p>Thanks in advance for your help!</p>

KFrank: <p>Hi Lukas!</p>
<aside class="quote no-group" data-username="MrSchiller" data-post="1" data-topic="220791" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/7993a0/48.png" class="avatar"> MrSchiller:</div>
<blockquote>
<p>Despite a well-decreasing loss, the predictions end up being constant.</p>
</blockquote>
</aside>
<p>Looking at the “Prediction vs. Real” chart you’ve posted, it looks like your model is learning<br>
to predict the mean of your “Real” target values (but not the actual values on a per-sample<br>
basis).</p>
<p>This would explain your sensibly falling loss as you train – the loss gets smaller as your<br>
model better learns the mean value in question.</p>
<p>I would suggest trying to get your model to overfit.  Take, for example, a single batch of five<br>
or ten samples, and train repeatedly (perhaps for quite a long time) with that one batch.  Does<br>
your model learn to predict the correct target values just for the batch?  (Don’t expect such an<br>
overfit model to “generalize” to the rest of your dataset.)  If you can get your model to overfit,<br>
can you get it to overfit on a larger, but still small set of samples?</p>
<p>If you can get your model to overfit, it is able to learn to predict per-sample values, rather than<br>
just the overall mean.  I would then try training on your full training set for a much longer time.</p>
<p>It is plausible that it’s easy for your model to learn the overall mean, but harder to learn the<br>
per-sample values.  Learning the mean could give you a nicely falling loss curve that then<br>
plateaus, and you simply might not be training (anywhere near long) enough to see the “real,”<br>
more difficult per-sample learning start to kick in.</p>
<p>You say that you normalize your data, but from the looks of your chart, it appears that you are<br>
not normalizing your target values to have mean zero.  I would recommend normalizing your<br>
target values to mean zero and standard deviation one.  Now a randomly-initialized model will<br>
start out making predictions whose mean value is about zero, so your model can focus on<br>
learning the sample-to-sample differences (rather than just the mean).</p>
<p>If you can’t get your model to overfit a (very) small dataset, then you should check your code<br>
carefully for bugs.  (You might consider whether there is some peculiarity in your architecture<br>
that impedes training, although this seems less likely.)</p>
<p>For example, if your model were somehow disconnected in the middle so that the inputs never<br>
made it to the output, the <code>bias</code> in the final layer could, nonetheless, learn to predict the overall<br>
mean of the target values.</p>
<p>Best.</p>
<p>K. Frank</p>

MrSchiller: <p>Hello Frank,<br>
thank you for your detailed analysis and time to check out everything! I will try to fix my code with your suggestions. I will give you an update how it went.</p>
<p>Thank you very much the support, have a great day!</p>
<ul>
<li>Lukas</li>
</ul>