Geospatial Feature Engineering: Feature Extraction via Clustering
By: Emmanuel Nunez
Tags: Feature Engineering, Machine Learning, Regression Modeling, Geospatial Features, Geospatial Data

In machine learning, we often encounter geographical or geospatial data such as latitude and longitude to be used as features. Some other geospatial data may include city, area code and other objects which may enrich latent relationships between the variables.

In this tutorial, it is assumed that the latitude and longitude coordinates are provided and we would like to extract features out from these coordinates. For additional feature engineering techniques, you may find this useful.

Scenario

A housing data is provided and the goal is to predict the housing price given all the features (columns).

Data

A housing data was provided which includes latitude and longitude.

Housing Features with Latitude and Longitude

The direct encoding of these coordinates along with the rest of the features resulted to an 85% “R squared” score, which is not bad. However, I wanted to improve the performance of the model by extracting features from the location points.

Clustering via Kmeans

There are other methods to extract features from the location coordinates. One may use geoencoders to extract city name, distance from the city and others. However, this may be computationally expensive and time consuming, depending on the volume of the data.

Another method is clustering via Kmeans. This method offers a quick and easy implementation of feature extraction. Below is the sample code of implementation.

Clustering via Kmeans

In the code above, I arbitrarily chose 50 clusters which turned to be good enough number, fit the data and plotted it along with the centers of the clusters. The second to the last line is the actual feature extraction using the “fit_transform” method, which gives a cluster-distance transformed data with dimensions, “n_samples by n_clusters”. The table below is the sample of transformed data.

Geospatial Transformed, Cluster-Distance Data

The plot below shows the actual positions of the houses along with the center of the clusters.

Housing Coordinates with Cluster Centers

Model Performance

In this task, there were two models used; Sklearn’s ElasticNetCV and XGBoostRegressor from XGBoost platform. The latter yielded better results with 90% “R squared” score which significantly improved after the geospatial feature extraction was applied.

Evaluation Results

Feature ranking shows that these spatial features played an important part in improving the performance of the model.

Feature Ranks

Conclusion:

Feature engineering is one of the crucial steps in creating a good performing model. When dealing with geospatial data, clustering is a simple and quick way to extract features which can significantly improve the performance of the model.

Possible Optimizations

One possible way to optimize this method is to find the best number k, clusters in kmeans using the “Elbow" method. However, I will leave this exercise for you to try in your projects.

Let me know if you have any questions. Just leave a comment below. ☺

Here is the link to the complete code, https://github.com/eanunez/tutorials/blob/master/XGBoostRegressor_housing.ipynb