Question: I am mastering pytorch here, and decided to implement very simple 1 to 1 linear regression, from height to weight. Got dataset: https://www.kaggle.com/datasets/mustafaali96/weight-height but any other would do nicely. Lets import libraries and information about females: import torch from torch.utils.data import Dataset from torch.utils.data import DataLoader import pandas as pd import matplotlib.pyplot as plt import numpy as np df = pd.read_csv('weight-height.csv',sep=',') #https://www.kaggle.com/datasets/mustafaali96/weight-height height_f=df[df['Gender']=='Female']['Height'].to_numpy() weight_f=df[df['Gender']=='Female']['Weight'].to_numpy() plt.scatter(height_f, weight_f, c ="red",alpha=0.1) plt.show()  Which gives nice scatter of measured females:  So far, so good. Lets make Dataloader: class Data(Dataset): def __init__(self, X: np.ndarray, y: np.ndarray) -> None: # need to convert float64 to float32 else # will get the following error # RuntimeError: expected scalar type Double but found Float self.X = torch.from_numpy(X.reshape(-1, 1).astype(np.float32)) self.y = torch.from_numpy(y.reshape(-1, 1).astype(np.float32)) self.len = self.X.shape[0] def __getitem__(self, index: int) -> tuple: return self.X[index], self.y[index] def __len__(self) -> int: return self.len traindata = Data(height_f, weight_f) batch_size = 500 num_workers = 2 trainloader = DataLoader(traindata, batch_size=batch_size, shuffle=True, num_workers=num_workers)  ...linear regression model... class linearRegression(torch.nn.Module): def __init__(self, inputSize, outputSize): super(linearRegression, self).__init__() self.linear = torch.nn.Linear(inputSize, outputSize) def forward(self, x): out = self.linear(x) return out model = linearRegression(1, 1) criterion = torch.nn.MSELoss() optimizer = torch.optim.SGD(model.parameters(), lr=0.00001)  .. lets train it: epochs=10 for epoch in range(epochs): print(epoch) for i, (inputs, labels) in enumerate(trainloader): outputs=model(inputs) loss = criterion(outputs, labels) optimizer.zero_grad() loss.backward() optimizer.step()  gives 0,1,2,3,4,5,6,7,8,9 now lets see what our model gives: range_height_f=torch.linspace(height_f.min(),height_f.max(),150) plt.scatter(height_f, weight_f, c ="red",alpha=0.1) pred=model(range_height_f.reshape(-1, 1)) plt.scatter(range_height_f, pred.detach().numpy(), c ="green",alpha=0.1)  ...  Why does it do this? Why wrong slope? consistently wrong slope, I might add Whatever I change, optimizer, batch size, epochs, females to males.. it gives me this very wrong slope, and I really don't get - why? Edit 1: Added loss, here is plot  Edit 2: Have decided to explore a bit, and made regression with skilearn: from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression X_train, X_test, y_train, y_test = train_test_split(height_f, weight_f, test_size = 0.25) regr = LinearRegression() regr.fit(X_train.reshape(-1,1), y_train) plt.scatter(height_f, weight_f, c ="red",alpha=0.1) range_pred=regr.predict(range_height_f.reshape(-1, 1)) range_pred plt.scatter(range_height_f, range_pred, c ="green",alpha=0.1)  which gives following regression, which looks nice:  t = torch.from_numpy(height_f.astype(np.float32)) p=regr.predict(t.reshape(-1,1)) p=torch.from_numpy(p).reshape(-1,1) w= torch.from_numpy(weight_f.astype(np.float32)).reshape(-1,1) print(criterion(p,w).item())  However in this case criterion=100.65161998527695 Pytorch in own turn converges to about 210 Edit 3 Changed optimisation to Adam from SGD: #optimizer = torch.optim.SGD(model.parameters(), lr=0.00001) optimizer = torch.optim.Adam(model.parameters(), lr=0.5)  lr is larger in this case, which yields interesting, but consistent result. Here is loss: , And here is proposed regression:  And, here is log of loss criterion as well for Adam optimizer: 

Answer: I think your issue stems from the data not being centered around zero. See this thread for another example where "centering" the data prior to training has a huge effect on the convergence of SGD optimization.  Update (Dec 29the, 2022): TL;DR It's all about normalization/initialization. In detail: Your data is not centered around 0 and it is not scaled "nicely". This makes it very difficult to SGD (and all other variants of it) to struggle with optimization. In this answer I showed how centering the training data (subtracting mean and deciding by the std) solves this problem. Here I'll show you how to leave your data as-is, but change the initialization of the weights to solve your problem. let m_x, s_x be the mean and std of X, and m_y, s_y be the mean and std of y. When pytorch init the weights, a and b, for the linear layer y = aX + b it assumes X and y have zero mean and unit variance. This is NOT the case here. Far from it. Therefore, we need to re-adjust the initial a and b accordingly. Here's the math for it:  And the code: mu_x, sig_x, mu_y, sig_y = traindata.X.mean().item(), traindata.X.std().item(), traindata.y.mean().item(), traindata.y.std().item() # just for fun, here are the values: # (63.7087, 2.6962, 135.8601, 19.0225) # start a fresh model and adjust its initial values: model = linearRegression(1, 1) model.linear.weight.data *= (sig_x / sig_y) model.linear.bias.data = sig_y * (-(mu_x/sig_x)+(mu_y/sig_y)) # now you are good to go! continue optimizing like you originally did: # init an optimizer optimizer = torch.optim.SGD(model.parameters(), lr=0.00001) # optimize for 10 epochs (now you don't need this much, you can even increase the learning rate...) epochs=10 for epoch in range(epochs): print(epoch) for i, (inputs, labels) in enumerate(trainloader): outputs=model(inputs) loss = criterion(outputs, labels) optimizer.zero_grad() loss.backward() optimizer.step()  The loss curve looks like this:  And the optimizer converged to In []: loss.item() Out[]: 100.9453125  Similar to that of sklearn.linear_model.LinearRegression. Plotting the prediction on the data: