Deep Neural Networks for Regression Problems | by Mohammed AL-Ma'amari | Towards Data Science

Neural networks are well known for classification problems, for example, they are used in handwritten digits classification, but the question is will it be fruitful if we used them for regression problems?
In this article I will use a deep neural network to predict house pricing using a dataset from Kaggle .
You can download the dataset from Here
I highly recommend you to try running the code using my notebook on Google colab [Here]
1- Process the dataset2- Make the deep neural network3- Train the DNN4- Test the DNN5- Compare the result from the DNN to another ML algorithm
First of all, we will import the needed dependencies :
We will not go deep in processing the dataset, all we want to do is getting the dataset ready to be fed into our models .
We will get rid of any features with missing values, then we will encode the categorical features, that’s it.
Load train and test data into pandas DataFrames
Combine train and test data to process them together
combined.describe()
let’s define a function to get the columns that don’t have any missing values
Get the columns that do not have any missing values .
Let’s see how many columns we got
[out]:Number of numerical columns with no nan values : 25 Number of nun-numerical columns with no nan values : 20
The correlation between the features
From the correlation heat map above, we see that about 15 features are highly correlated with the target.
One Hot Encode The Categorical Features :
We will encode the categorical features using one hot encoding.
[out]:There were 45 columns before encoding categorical features There are 149 columns after encoding categorical features
Now, split back combined dataFrame to training data and test data
Define a sequential model
Add some dense layers
Use ‘relu’ as the activation function for the hidden layers
Use a ‘normal’ initializer as the kernal_intializer
Initializers define the way to set the initial random weights of Keras layers.
We will use mean_absolute_error as a loss function
Define the output layer with only one node
Use ‘linear ’as the activation function for the output layer
[Out]:_________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= dense_1 (Dense)              (None, 128)               19200      _________________________________________________________________ dense_2 (Dense)              (None, 256)               33024      _________________________________________________________________ dense_3 (Dense)              (None, 256)               65792      _________________________________________________________________ dense_4 (Dense)              (None, 256)               65792      _________________________________________________________________ dense_5 (Dense)              (None, 1)                 257        ================================================================= Total params: 184,065 Trainable params: 184,065 Non-trainable params: 0 _________________________________________________________________
Define a checkpoint callback :
[out]:Train on 1168 samples, validate on 292 samplesEpoch 1/5001168/1168 [==============================] - 0s 266us/step - loss: 19251.8903 - mean_absolute_error: 19251.8903 - val_loss: 23041.8968 - val_mean_absolute_error: 23041.8968  Epoch 00001: val_loss did not improve from 21730.93555 Epoch 2/500 1168/1168 [==============================] - 0s 268us/step - loss: 18180.4985 - mean_absolute_error: 18180.4985 - val_loss: 22197.7991 - val_mean_absolute_error: 22197.7991  Epoch 00002: val_loss did not improve from 21730.93555...Epoch 00500: val_loss did not improve from 18738.19831<keras.callbacks.History at 0x7f4324aa80b8>
We see that the validation loss of the best model is 18738.19
We will submit the predictions on the test data to Kaggle and see how good our model is.
Not bad at all, with some more preprocessing, and more training, we can do better.
Now, let us try another ML algorithm to compare the results.
We will use random forest regressor and XGBRegressor.
Split training data to training and validation data
We will try Random forest model first:
Random forest validation MAE =  19089.71589041096
Make a submission file and submit it to Kaggle to see the result :
Now, let us try XGBoost model :
[out]: XGBoost validation MAE =  17869.75410958904
Make a submission file and submit it to Kaggle to see the result :
Isn’t that a surprise, I really did not think that neural networks will beat random forests and XGBoost algorithms, but let us try not to be too optimistic, remember that we did not configure any hyperparameters on random forest and XGBoost models, I believe if we did so, these two models would outscore neural networks.
We load and processed the dataset
We got familiar with the dataset by plotting some histograms and a correlation heat map of the features
We used a deep neural network with three hidden layers each one has 256 nodes.
We used a linear activation function on the output layer
We trained the model then test it on Kaggle.
We also tested two other models
Our deep neural network was able to outscore these two models
We believe that these two models could beat the deep neural network model if we tweak their hyperparameters.
Try to put more effort on processing the dataset
Try other types of neural networks
Try to tweak the hyperparameters of the two models that we used
If you really want to get better at regression problems, follow this tutorial:
[Stacked Regressions : Top 4% on LeaderBoard | Kaggle]
Regression Tutorial with the Keras Deep Learning Library in Python
You can follow me on Twitter @ModMaamari
AI Generates Taylor Swift’s Song Lyrics
Introduction to Random Forest Algorithm with Python
Machine Learning Crash Course with TensorFlow APIs Summary
How To Make A CNN Using Tensorflow and Keras ?
How to Choose the Best Machine Learning Model ?