Various factors could be affecting this scenario. Some of them are as follows: The learning rate could be slow. This is why the loss does not decrease in few starting epochs. Regularization parameter could be high. This could be stuck at the local minima.