Training Neural Networks for price prediction with TensorFlow | by Jan Majewski | Towards Data Science

Using Deep Neural Networks for regression problems might seem like overkill (and quite often is), but for some cases where you have a significant amount of high dimensional data they can outperform any other ML models.
When you learn about Neural Networks you usually start with some image classification problem like the MNIST dataset — this is an obvious choice as advanced tasks with high dimensional data is where DNNs really thrive.
Surprisingly, when you try to apply what you learned on MNIST on a regression tasks you might struggle for a while before your super-advanced DNN model is any better than a basic Random Forest Regressor. Sometimes you might never reach that moment...
In this guide, I listed some key tips and tricks learned while using DNN for regression problems. The data is a set of nearly 50 features describing 25k properties in Warsaw. I described the feature selection process in my previous article: feature-selection-and-error-analysis-while-working-with-spatial-data so now we will focus on creating the best possible model predicting property price per m2 using the selected features.
The code and data source used for this article can be found on GitHub.
When training a Deep Neural Network I usually follow these key steps:
A) Choose a default architecture — no. of layers, no. of neurons, activation
B) Regularize model
C) Adjust network architecture
D) Adjust the learning rate and no. of epochs
E) Extract optimal model using CallBacks
Usually creating the final model takes a few runs through all of these steps but an important thing to remember is: DO ONE THING AT A TIME. Don’t try to change architecture, regularization, and learning rate at the same time as you will not know what really worked and probably spend hours going in circles.
Before you start building any DNNs for regression tasks there are 3 key things you must remember:
Standarize your data to make training more efficient
Use RELU activation function for all hidden layers — you will be going nowhere with default sigmoid activation
Use Linear activation function for the single-neuron output layer
Another important task is selecting the loss function. Mean Squared Error or Mean Absolute Error are the two most common choices. As my goal to minimize the average percentage error and maximize the share of all buildings within 5% error I choose MAE, as it penalizes outliers less and is easier to interpret — it pretty much tells you how many $$/m2 on average each offer is off the actual value.
There is also a function directly linked to my goal — Mean Absolute Percentage Error, but after testing it against MAE I found the training to be less efficient.
We start with a basic network with 5 hidden layers and a decreasing number of neurons in every second layer.
tf.keras.backend.clear_session()tf.random.set_seed(60)model=keras.models.Sequential([        keras.layers.Dense(512, input_dim = X_train.shape[1], activation='relu'),      keras.layers.Dense(512, input_dim = X_train.shape[1], activation='relu'),      keras.layers.Dense(units=256,activation='relu'),      keras.layers.Dense(units=256,activation='relu'),        keras.layers.Dense(units=128,activation='relu'),    keras.layers.Dense(units=1, activation="linear"),],name="Initial_model",)model.summary()
We use Adam optimizer and start with training each model for 200 epochs — this part of the model configuration will be kept constant up to point 7.
optimizer = keras.optimizers.Adam()model.compile(optimizer=optimizer, warm_start=False,             loss='mean_absolute_error')history = model.fit(X_train, y_train,                    epochs=200, batch_size=1024,                    validation_data=(X_test, y_test),                     verbose=1)
Our first model turned out to be quite a failure, we have horrendous overfitting on Training data and our Validation Loss is actually increasing after epoch 100.
Drop out is probably the best answer to DNN regularization and works with all types of network sizes and architectures. Applying Dropout randomly drops a portion of neurons in a layer in each epoch during training, which forces the remaining neurons to be more versatile — this decreases overfitting as one Neuron can no longer map one specific instance as it will not always be there during training.
I advise reading the original paper as it describes the idea very well and does not require years of academic experience to understand it — Dropout: A Simple Way to Prevent Neural Networks from Overfitting
tf.keras.backend.clear_session()tf.random.set_seed(60)model=keras.models.Sequential([        keras.layers.Dense(512, input_dim = X_train.shape[1], activation='relu'),      keras.layers.Dropout(0.3),        keras.layers.Dense(512, activation='relu'),      keras.layers.Dropout(0.3),keras.layers.Dense(units=256,activation='relu'),     keras.layers.Dropout(0.2),        keras.layers.Dense(units=256,activation='relu'),     keras.layers.Dropout(0.2),        keras.layers.Dense(units=128,activation='relu'),    keras.layers.Dense(units=1, activation="linear"),],name="Dropout",)
The (0.x) after Dropout specifies what share of Neurons you want to drop, which translates into how much you want to regularize. I usually start with dropout around (0.3–0.5) in the largest layer and then reduce its rigidness in deeper layers. The idea behind such approach is that neurons in deeper networks tend to have more specific tasks and therefore dropping too many will increase bias too much.
Analyzing learning curve for the modified model we can see that we are going in the right direction. First of all we managed to make progress from the Validation Loss of the previous model (marked by the grey threshold line), secondly, we seem to replace overfitting with a slight underfit.
When working with several layers with RELU activation we have a significant risk of dying neurons having a negative effect on our performance. This can lead to underfitting we could see in the previous model as we might actually not be using a large share of our neurons, which basically reduced their outputs to 0.
Batch Normalization is one of the best ways to handle this issue — when applied we normalize activation outputs of each layer for each batch to reduce the effect of extreme activations on parameter training, which in turn reduces the risk of vanishing/exploding gradients. The original paper describing the solution is more complicated to read than the previous one referenced but I would still suggest giving it a try — Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
tf.keras.backend.clear_session()tf.random.set_seed(60)model=keras.models.Sequential([        keras.layers.Dense(512, input_dim = X_train.shape[1], activation='relu'),     keras.layers.BatchNormalization(),    keras.layers.Dropout(0.3),        keras.layers.Dense(512, activation='relu'),      keras.layers.BatchNormalization(),    keras.layers.Dropout(0.3),keras.layers.Dense(units=256,activation='relu'),     keras.layers.BatchNormalization(),    keras.layers.Dropout(0.2),        keras.layers.Dense(units=256,activation='relu'),     keras.layers.BatchNormalization(),    keras.layers.Dropout(0.2),        keras.layers.Dense(units=128,activation='relu'),    keras.layers.Dense(units=1, activation="linear"),],name="Batchnorm",)
Adding Batch Normalization helped us to bring some of the neurons back to life, which increased our model variance changing underfitting to slight overfitting — training neural networks is often a game of cat and mouse, balancing between optimal bias and variance.
Another good news is that we still are improving in terms of a validation error.
Leaky RELU activation function is a slight modification of RELU function, which allows some negative activations to leak through, further reducing the risk of dying neurons. Leaky RELU usually takes longer to train, which is why we will train this model for another 100 epochs.
tf.keras.backend.clear_session()tf.random.set_seed(60)model=keras.models.Sequential([        keras.layers.Dense(512, input_dim = X_train.shape[1]),     keras.layers.LeakyReLU(),    keras.layers.BatchNormalization(),    keras.layers.Dropout(0.3),        keras.layers.Dense(512),      keras.layers.LeakyReLU(),    keras.layers.BatchNormalization(),    keras.layers.Dropout(0.3),keras.layers.Dense(units=256),     keras.layers.LeakyReLU(),    keras.layers.BatchNormalization(),    keras.layers.Dropout(0.2),        keras.layers.Dense(units=256),     keras.layers.LeakyReLU(),    keras.layers.BatchNormalization(),    keras.layers.Dropout(0.2),        keras.layers.Dense(units=128),    keras.layers.LeakyReLU(),     keras.layers.Dense(units=1, activation="linear"),],name="LeakyRELU",)
It seems Leaky RELU reduced the overfitting and gave us a healthier learning curve, where we can see the potential for improvement even after 300 epochs. We nearly reached the lowest error from previous model, but we managed to do that without overfitting, which leaves us space for increasing variance.
At this point, I am happy enough with the basic model to make the network larger by adding another hidden layer with 1024 neurons. The new layer also has the highest dropout rate. I also experimented with dropout rates for lower levels due to change in the overall architecture.
tf.keras.backend.clear_session()tf.random.set_seed(60)model=keras.models.Sequential([        keras.layers.Dense(1024, input_dim = X_train.shape[1]),     keras.layers.LeakyReLU(),    keras.layers.BatchNormalization(),    keras.layers.Dropout(0.4),        keras.layers.Dense(512),      keras.layers.LeakyReLU(),    keras.layers.BatchNormalization(),    keras.layers.Dropout(0.3),keras.layers.Dense(512),      keras.layers.LeakyReLU(),    keras.layers.BatchNormalization(),    keras.layers.Dropout(0.3),        keras.layers.Dense(units=256),     keras.layers.LeakyReLU(),    keras.layers.BatchNormalization(),    keras.layers.Dropout(0.2),        keras.layers.Dense(units=256),     keras.layers.LeakyReLU(),    keras.layers.BatchNormalization(),    keras.layers.Dropout(0.01),keras.layers.Dense(units=128),    keras.layers.LeakyReLU(),     keras.layers.Dropout(0.05),    keras.layers.Dense(units=1, activation="linear"),],name="Larger_network",)
Expanding network architecture seems to be going in the right direction, we increased variance slightly getting learning curve, which is close to optimal balance. We also managed to get our Validation Loss nearly on par with the overfitted BatchNorm model.
Once we are happy with the network architecture, Learning Rate is the most important hyperparameter, which needs tuning. I decided to use learning rate decay, which allows me to train my model faster at the beginning and then decrease the learning rate with further epochs to make training more precise.
optimizer = keras.optimizers.Adam(lr=0.005, decay=5e-4)
Selecting the right starting rate and decay can be challenging and takes some trial and error. In my case it turned out that the default Adam learning rate in Keras, which is 0.001 was a bit high. I started with a Learning rate of 0.005 and over 400 epochs decreased it to 0.001.
Tuning Learning Rate helped us to finally improve our validation error result, while still keeping the learning curve healthy without too much risk of overfitting — there might even be some space for training the model for another 100 epochs.
The last task remaining before choosing our best model is to use CallBacks to stop training at the optimal epoch. This allows us to retrieve the model at the exact epoch, where we reached minimall error. The big advantage of this solution is that you do not really need to worry if you want to train for 300 or 600 epochs — if your model starts overfitting the Call Back will get you back to the optimal epoch.
checkpoint_name = 'Weights\Weights-{epoch:03d}--{val_loss:.5f}.hdf5' checkpoint = ModelCheckpoint(checkpoint_name, monitor='val_loss', verbose = 1, save_best_only = True, mode ='auto')callbacks_list = [checkpoint]
You need to define your callbacks: checkpoint_name specifying where and how you want to save weights for each epoch, checkpoint specifies how the CallBack should behave —I advise monitoring val_loss for improvement and saving only if the epoch made some progress on that.
history = model.fit(X_train, y_train,                    epochs=500, batch_size=1024,                    validation_data=(X_test, y_test),                     callbacks=callbacks_list,                     verbose=1)
Then all you need to do is to add callbacks while fitting your model.
Using Callbacks allowed us to retrieve the optimal model trained at epoch 468 — the next 30 epochs did not improve as we started to overfit the train set.
It took us 7 steps in order to get to the desired model output. We managed to improve at nearly every step, with a plateau between batch_norm and 1024_layer model, when our key goal was to reduce overfitting. To be honest refining these 7 steps, probably took me 70 steps so bear in mind that training DNNs is an interative process and don’t be put off if your improvement stagnates for a few hours.
Finally, how did our best DNN perform in comparison to a base Random Forest Regressor trained on the same data in the previous article?
In two key KPIs our Random Forest scored as follows:
Share of forecasts within 5% absolute error = 44.6%
Mean percentage error = 8.8%
Our best Deep Neural Network scored:
Share of forecasts within 5% absolute error = 43.3% (-1.3 p.p.)
Mean percentage error = 9.1% (+0.3 p.p.)
Can we cry now? How is it possible that after hours of meticulous training our advanced neural network did not beat a Random Forest? To be honest there are two key reasons:
A sample size of 25k records is still quite small in terms of training DNNs, I choose to give this architecture a try as I am gathering new data every month and I am confident that within a few months I will reach samples closer to 100k, which should give DNN the needed edge
The Random Forest model was quite overfitted and I am not confident that it would generalize well too other properties, despite high performance on validation set — at this point, I would probably still use the DNN model in production as more reliable.
To summarize — I would advise against starting the solving of a regression problem with DNN. Unless you are working with hundreds of k samples on a really complex project, a Random Forest Regressor will usually be much faster to get initial results — if they prove to be promising you can proceed to DNN. Training efficient DNN takes more time and if your data sample is not large enough it might never reach Random Forest performance.
[1]: Nitish Srivastava. (June 14 2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting
[2]: Sergey Ioffe. (Mar 2 2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift