Linear regression loss function value does not decrease

mtx: <p>Hi everyone!<br>
I’m pretty new to Machine Learning and I was trying to implement Linear Regression to predict house prices.<br>
The issue that I’m facing is that during training, my loss function does not decrease or it decreases just a bit. It’s stuck on value <code>165283216.0</code> which is not satisfying at all…</p>
<p>Here’s the code:</p>
<pre><code class="lang-auto"># house_features is pandas dataframe with columns: 'district', 'rooms', 'square_meters'
# house_prices is a dataframe with just one column: 'price'
X_train, x_test, Y_train, y_test = train_test_split(house_features, house_prices, test_size=0.2, random_state=42)

# Here I'm converting data to tensors
dtype = torch.float
X_train_tensor = torch.tensor(X_train.values, dtype=dtype)
x_test_tensor = torch.tensor(x_test.values, dtype=dtype)

Y_train_tensor = torch.tensor(Y_train.values, dtype=dtype)
y_test_tensor = torch.tensor(y_test.values, dtype=dtype)

# Configuration values
input_features_amount = 3
output_amount = 1
hidden_layer_size = 10

loss_function = torch.nn.MSELoss()
learning_rate = 1e-4

model = torch.nn.Sequential(torch.nn.Linear(input_features_amount, hidden_layer_size),
                            torch.nn.Sigmoid(),
                            torch.nn.Linear(hidden_layer_size, output_amount))

loss_list = []
EPOCHS = 10_000
# Here I'm reshaping tensor so that the shapes would match - I was receiving an error without this line
Y_train_tensor = torch.reshape(Y_train_tensor, (X_train_tensor.shape[0], 1))

for epoch in range(EPOCHS):
    y_pred = model(X_train_tensor)
    loss = loss_function(y_pred, Y_train_tensor)
    
    if epoch % 1000 == 0:
        print(epoch, loss.item())
    
    loss_list.append(loss.item())
    model.zero_grad()
    loss.backward()
    
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad
</code></pre>
<p>I attach also a screenshot from Jupyter Notebook with training loss function values. Does anyone know what might be the reason of this issue?<br>
I went through a couple of similar posts but I wasn’t able to implement any reasonable fix.</p>
<p>About dataset, I built it myself by scrapping some page with house rental offers and it consists of ~1000 records.</p>
<p>Thanks in advance!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/3/63c2614a7beda07bbc63d6b005b78e2e1a057927.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/63c2614a7beda07bbc63d6b005b78e2e1a057927" title="Screen Shot 2023-03-01 at 20.05.08"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63c2614a7beda07bbc63d6b005b78e2e1a057927_2_690x318.jpeg" alt="Screen Shot 2023-03-01 at 20.05.08" data-base62-sha1="eevJ4bxmrB9SAZcvGYJPta6AJ2T" width="690" height="318" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63c2614a7beda07bbc63d6b005b78e2e1a057927_2_690x318.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63c2614a7beda07bbc63d6b005b78e2e1a057927_2_1035x477.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63c2614a7beda07bbc63d6b005b78e2e1a057927_2_1380x636.jpeg 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-03-01 at 20.05.08</span><span class="informations">1920×887 62 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

The_Snail: <p>Did you scale the features in any way? Usually neural networks won’t learn well if the features are not scaled.</p>

mtx: <p>I didn’t before, but now I used Standarization:</p>
<pre><code class="lang-auto">scaler = StandardScaler()
X_train_norm = scaler.fit_transform(X_train)
</code></pre>
<p>It got a little bit better, but the result is still not satisfying <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74a1ac87ef61dc900f09ee049d8723ab646bca00.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/74a1ac87ef61dc900f09ee049d8723ab646bca00" title="Screen Shot 2023-03-01 at 21.53.46"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74a1ac87ef61dc900f09ee049d8723ab646bca00_2_690x314.jpeg" alt="Screen Shot 2023-03-01 at 21.53.46" data-base62-sha1="gDLLPiMsBXmxoOTTgeOz9lqjpyU" width="690" height="314" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74a1ac87ef61dc900f09ee049d8723ab646bca00_2_690x314.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74a1ac87ef61dc900f09ee049d8723ab646bca00_2_1035x471.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74a1ac87ef61dc900f09ee049d8723ab646bca00_2_1380x628.jpeg 2x" data-dominant-color="F5F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2023-03-01 at 21.53.46</span><span class="informations">1920×875 61.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

The_Snail: <p>I can see now that you also unnecesserily do the parameters update manually at the end of the code. Have a look at pytorch tutorials to see how to contruct proper training loop.</p>

ptrblck: <p>Your current code works ~alright for overfitting random data:</p>
<pre data-code-wrap="python"><code class="lang-python"># Configuration values
input_features_amount = 3
output_amount = 1
hidden_layer_size = 10

X_train_tensor = torch.randn(16, input_features_amount)
Y_train_tensor = torch.rand(16, 1) * 100

loss_function = torch.nn.MSELoss()
learning_rate = 1e-3

model = torch.nn.Sequential(torch.nn.Linear(input_features_amount, hidden_layer_size),
                            torch.nn.Sigmoid(),
                            torch.nn.Linear(hidden_layer_size, output_amount))

loss_list = []
EPOCHS = 100_000

for epoch in range(EPOCHS):
    y_pred = model(X_train_tensor)
    loss = loss_function(y_pred, Y_train_tensor)
    
    if epoch % 1000 == 0:
        print(epoch, loss.item())
    
    loss_list.append(loss.item())
    model.zero_grad()
    loss.backward()
    
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad
            
plt.plot(y_pred.detach().numpy())
plt.plot(Y_train_tensor.numpy())
</code></pre>
<p>Output:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2ea92d0d642dc74c2827d6b380251647c2040015.png" alt="image" data-base62-sha1="6EMq8jzB32DAuIkDYbMzUzhseLH" width="368" height="248"></p>
<p>Note that I’ve already increased the learning rate and number of epochs by 10x and the model is still not overfitting the 16 samples.<br>
Using <code>nn.ReLU</code> instead of <code>nn.Sigmoid</code> seems to help and indeed using a built-in optimizer, as suggested by <a class="mention" href="/u/the_snail">@The_Snail</a>, might help.</p>