Adventure into Machine Learning using Python
By: Bee Guan Teo
Tags: Supervised Learning, Machine Learning, Python, Data Science

Hands-on

Now, we have some ideas about machine learning and we are ready to move on to the practical aspect to start training a machine learning model.

1. Objective

The objective of this project is to perform value estimation of house prices in Sindian District., New Taipei City, Taiwan. To achieve the objective, we are going to train a machine learning model to predict house prices.

2. Acquisition of dataset

We will use the real estate valuation data which can be obtained from the UCL Machine Learning Repository. The estate data is stored in an Excel file.

In general, there are two main sources of data for machine learning: 1) our own collection, 2) public data. We tend to use public data for learning or experimental purpose as this saves us lots of time from collecting data on our own. Thanks to the generosity of the data science community, we can easily obtain the public data from sites such as Kaggle, UCL Machine Learning Repository, Open Data, to name just a few.

3. Setting up machine learning workspace

With our raw data ready, we can start setting up our machine learning workspace.

3.1 Hardware requirement

We don’t need a high-cost computer installed with NVIDIA GPU to run this machine learning project. A PC or Mac machine with processor i5 or i7 and 8GB of RAM memories is good enough to meet the hardware requirement for this simple project.

3.2 Software requirement

We are going to use Python-based machine learning and data processing libraries for this project and therefore our computer shall be ready with:

Python 3 (https://www.python.org/)

Pandas (https://pandas.pydata.org/)

Matplotlib (https://matplotlib.org/)

Numpy (https://numpy.org/)

Scikit-learn (https://scikit-learn.org/stable/)

Jupyter Notebook / Jupyter Lab https://jupyter.org/)

The easiest way to get all these Python libraries is to install Anaconda package on our computer. Anaconda offers most of the Python-based machine learning and data processing libraries in one package.

4. Exploring data

This is important to gain a general understanding of our data before we can use them to train our machine learning model. To explore our data, we are going to use Jupyter Notebook as our editor and start writing Python code.

Firstly, we can create a notebook and use Pandas to read our real estate data. (Please note I have renamed the Excel file to “Real_estate_data.xlsx”).

The snippet of code to read data from Excel file

We use the Pandas head() and info() methods to have a quick overview of our data.

The snippet of code to view the first five records

The first five records

The head() method shows the first five records.

The snippet of code to get a quick description of data

Description of data

Pandas info() method gives us a quick description of data. For example:

total number of entries

total number of columns

data type in each column

There are only 414 entries in our real estate data. This is considered a very small dataset based on the machine learning standard. However, it’s still fine to use it as sample data to kickstart a simple project.

This is also worth highlighting that the house price in the data is based on the price per unit area in Taiwan currency (NT$).

The feature in every column is numerical. However, not all the features are useful to train a machine learning model. For instance, the first column “No” serves nothing more than a record index. Obviously, it doesn’t show any strong correlation with a house price. Hence, we will simply filter out this feature through a feature selection process later.

Besides, many column names of our data are too lengthy and this will pose some difficulties to use Pandas to extract the individual columns to analyze them. We will have to rename the columns to more concise names to ease our data processing job.

Problematic column names

5. Preprocessing data

At this stage, we are going to use Python Pandas library again to filter and clean our data before it can be used to train our machine learning model.

5.1 Rename columns

We start our data preprocessing step by renaming all the column names of our real estate data except column “No”. We can use the Pandas rename() method for this purpose.

The snippet of code to rename columns

To rename the column, we just simply create a column mapper, new_names (Line 1–7). We map a new column name to an original name and then set the entire column mapper to parameter “columns” (Line 9).

Next, we use the head() method again to visualize the first five records (Line 10) and we shall see the output as below:

Renamed columns

5.2 Feature selection

As mentioned, not all the features from our data shall be used to train the machine learning model. Only those of the features that show a good correlation with house prices will be selected for machine learning. Besides, the feature selection is also important to save computational cost to train a machine learning model and to improve the model performance.

Pandas dataframe offers a handy method, corr(), to compute the standard correlation coefficient (Pearson’s r) between every pair of the features that exist in our data.

The snippet of code to compute the standard correlation coefficient

Result of standard coefficients

To interpret the result as shown above, we need to understand several important facts:

The correlation coefficient will always range from -1 to 1

If the value is close to 1, it means there is an existence of a strong positive correlation between a particular pair of features(e.g. conv_stores vs house_price)

correlation between a particular pair of features(e.g. conv_stores vs house_price) If the value is close to -1, it means there is an existence of a strong negative correlation between a particular pair of features (e.g. dist_mrt vs house_price)

correlation between a particular pair of features (e.g. dist_mrt vs house_price) If the value is close to 0, it means there is no linear correlation between that particular pair of features (e.g. No vs house_price).

The result shows that the feature conv_stores and dist_mrt heavily impact the house price as they have the highest positive and negative correlation, respectively. This outcome makes sense as a house price is usually dependant on the availability of surrounding facilities. One of the important facilities is the convenience shop. On another hand, the real estate values are greatly reduced when the properties located far from public transport service (That’s why the dist_mrt shows a strong negative correlation with house_price).

Besides, we should not underestimate the impact of the geographic location which are represented by features lat and lng. The correlation analysis of lat and lng suggests that the house price tends to be higher in the North-East region. One way to visualize the distribution of house prices against the geographic location is to plot a scatterplot using Python Matplotlib library.

The snippet of code to plot scatterplot of distribution of house prices against the geographic locations

The scatterplot showing the distribution of house prices

The color map in the scatter plot reveals most of the high-cost estate properties are centered at the red circle region. Although this is not always true that all the high-cost estates situated in the region, it still shows a general density pattern where the price of an estate property tends to be higher compared with other regions.

Besides, the feature house_age can also be one significant factor that affects property prices although the impact is not comparable to dist_mrt and conv_stores.

On another hand, feature No and trans_date show the poor correlation with house price (close to zero).

Curse of dimensionality

While there are at least five features from the dataset which are significant enough to train our model, we have a very small dataset with only 414 entries. According to the curse of dimensionality, the more features we use to train our model, the dataset we need tends to grow exponentially. In general, the dataset we need for

1 feature = 10,

2 features = 10 * 10 = 100,

3 features = 10 * 10 * 10 = 1000 and so on….

Given a limited dataset with only hundreds of entries, we can only pick two of the most relevant features to train our model. Hence, let us pick the features with the highest positive and negative correlation: conv_stores and dist_mrt.

In short, only two features are selected as the input features for machine learning based on the result of correlation analysis and consideration of the curse of dimensionality.

We are going to drop features No, trans_date, lat, lng, and house_age from our dataset using the Pandas drop() method. Besides, we also need to exclude the “house_price” from our input feature list.

The snippet of code to drop columns from the original dataset

Remaining columns used as input features

5.3 Data conversion

At this stage, we have done feature selection but we still haven’t ready to use the input features to train our model. There are still several preprocessing steps required to complete before we can start training our model.

We need to convert our input features into a Numpy array. Besides, we will also need to extract the “house_price” from the original dataset and use it as the output feature or label. The output feature should also be converted to Numpy array as well.

To convert the input feature and output feature into the Numpy array we can use Pandas data frame values property.

The snippet of code to convert input features and output feature to Numpy array

Conventionally, the input features shall be stored in a variable named, X and output feature in a variable named, y.

5.4 Feature scaling

If we observe our input features, X, again, we will find that they are numerical values with different scales.

dist_mrt (min: 23.38–max: 6488)

conv_store(min: 0–max: 10)

It is important to rescale our input features to enable them to share the same scale. Feature scaling is important in many machine learning algorithms in order to obtain the optimized output. There are several ways of feature scaling and the approach we are going to use is standardization. Standardization is a process of features rescaling so that all the features have a standard normal distribution with a mean of 0 and a standard deviation of 1.

Scikit-learn offers a useful module, StandardScaler class to perform standardization on the input features.

The snippet of code to perform data scaling using standardization

5.5 Splitting dataset into training and test sets

Next, we are going to split our input features, X, and output feature, y, into two separate sets: a training set and a test set. The training set as its name suggested is used to train the machine learning model whereas the test set is to evaluate the trained model in a later stage.

Scikit-learn provides another useful module, train_test_split, that enables us to easily split our data into a training set and a test set with few lines of codes.

The snippet of code to splite dataset into training set and test set

Here we use the common naming convention to create several variables named X_train, X_test, y_train, and y_test, to hold the arrays of the training set and the test set of input features and output feature. Please note the parameter, test_size=0.2, in the train_test_split() function will set apart 80% of data as training set and another 20% as test set.

6. Training model

At last, we are ready to train our first machine learning model. Let’s try with a simple model, Linear Regression Model. Linear regression is a modeling approach to define a linear relationship between a dependent variable and one or more independent variables.

We can use the Scikit-Learn’s LinearRegression module, to create a linear regression model and fit our training data with it.

The snippet of code to train linear model

In a few seconds, a trained linear model is produced and we can test the model with our test set using the predict() method.

The snippet of code to use test the trained model using test set

The predicted values from the test set are as follows:

Predicted house prices per unit area

7. Evaluating Model

Now we have our first model ready and we have used it to predict the house price per unit area for the test set. However, how do we know the accuracy of the model’s prediction? How well the linear model predicts the house price?

Let’s take a look at a grouped bar chart that shows the comparison of true values (y_test)and their corresponding predicted values (y_pred).

The snippet of codes to plot a group bar chart to compare the true y and predicted y

Grouped bar chart showing True Y and Predicted Y (Blue: True Y, Orange: Predicted Y)

The grouped bar chart above gives us a general idea that some predicted values are very close to their corresponding true values but some are far apart from each other.

Now, we are going to use a quantitative approach to measure the performance of the linear model through cost function. A cost function is a formula to quantify the discrepancy between the true value (y_test)and the predicted value (y_pred).

In a regression problem like predicting the house price, there are two common options we can consider: Root Mean Squared Error (RMSE) or Mean Absolute Error (MAE).

Both cost functions above calculate the average of the difference between the true values and the predicted values. However, the RMSE is sensitive to the outlier that could possibly exist in our dataset and affect the evaluation result. Hence, we pick MAE as our choice.

Fortunately, we can easily use Scikit-Learn module again to call for the mean_absolute_error() method to evaluate the model.

The snippet of codes to use MAE to measure the performance of the linear model

Result of MAE (Linear Regression Model)

The resulting MAE metric is around 6.344. This means the average absolute difference between the predicted house price per unit area and the true value is around NT$6.344 (This can be an indication that we might consider trying another model to gain a better outcome).

8. Trying a new model

In this section, we are going to train and evaluate a Decision Tree model. All we need is just to repeat the similar steps as presented in Section 6 & 7 above.

We can use DecisionTreeRegressor module from Scikit-Learn to create a Decision Tree model and fit our training data with it. Once again, we use MAE cost function to measure the model’s performance.

The snippet of codes to train a Decision Tree and evaluate the model

Result of MAE (Decision Tree Model)

The resulting MAE metric is around 4.84. This shows that the Decision Tree model offers a more accurate prediction of the house price compared with the Linear Regression model.

If we are still not satisfied with the current model’s accuracy, there are few approaches we can consider: