Water Pumps maintenance prediction : data-science illustrated
Tags: Python, Data Science, Machine Learning

Encoding

Trimming high-cardinality categorical variables

For a categorical variable to have a reliable predictive power it needs to be observed a significant amount of time ; otherwise we cannot be confident in its impact. Of course many Machine Learning algorithms can discard by themselves unreliable predictors to insure robustness but it still helps if we can take care of it up-stream.

In the following code snippet I group all rare (i.e. being observed less than 40 times in a dataset of >50,000 records ) categorical values into a single “rare” value. We achieve this with the “dataframe.value_counts” method that we used previously to find very high frequency values.

Remark : To dig further into the topic of robust estimation you can read about LASSO and Ridge regularization technics.

Encoding categorical variables

Once we are done cleaning/trimming the categorical columns we can turn them into numbers so as to feed them to our choice of predictive model.

How do we turn values such as USA, Spain, Russia, Australia into numbers? Well we don’t! Actually we turn categorical values into vectors. This can be done with an explicit encoding or an implicit encoding. As this post goal is to present generic ingredients for Machine Learning recipes I only present an explicit method ; one-hot-encoding.

One-hot-encoding first maps each value to an index integer and then maps this integer to a vector that is equal to 1 on the corresponding index and 0 everywhere else. For exemple with USA, Spain, Russia, Australia we could have :

USA => 0 => (1,0,0,0)

Spain => 1 => (0,1,0,0)

Russia => 2 =>(0,0,1,0)

Australia => 3 =>(0,0,0,1)

In Pandas we perform this conversion using the “pandas.get_dummies” function as follow :

Remark : If you have a large number of possible unique values the vector space will be very large and very sparse BUT we previously took care to trimming rare values.

Remark : Implicit encoding let the training algorithm determine the best vector representation for each value, this is usually called an embedding and this is quite common in neural-network modeling of text. For more information you can start with this blog post.

Encoding numerical variables

At this stage it is not absolutely necessary to further transform the numerical variables but it can help a lot the model estimation phase (further down) to re-scale them. Indeed it is empirically observed (and theoretically understood) that the optimization phase of the model calibration to the data works best if all the predictive variables are scaled to a common range of values. Given categorical variables are encoded 0/1 we scaled everything on a scaled of a few units. The usual scaling methods are :

Normal : (X[i]-mean(X))/stdev(X)

Min/Max : (X[i]-min(X))/(max(X)-min(X))

Quantile : Percentile(X[i], X)

Here is an example of Quantile encoding using “pandas.qcut” function :

Encoding date variables

Add new features for year, month, day of week by splitting the date stamp using the “dataframe.apply” method and custom Python “lambda” functions :

[more advanced] Encoding GPS data

GPS coordinates have one advantage over regional division : they take into account spatial ordering, something that is lost in region names. Furthermore region names are not great appropriate representations are all the area that are close to the region borders. However region names have one advantage : sparsity.

One way to get the best of coordinates and regional clustering is to :

overlay a grid over the map and

“weigh” the relevance of each grid node for each record

In the below exemple I overlaid a grid of 4x5=20 nodes. Lets label the nodes N(1), …, N(20) (the ordering does not matter). Then we add to our dataset 20 extra “weight” features/columns defined as follow :

Weight(GPS[i], N(1)) = kernel( distance(GPS[i], N(1)) )

…

Weight(GPS[i], N(20)) = kernel( distance(GPS[i], N(20)) )

Here the kernel function denote a gaussian (with the appropriate standard-deviation) so that the weight functions reach their maximum close the nodes.

Image source : https://commons.wikimedia.org/wiki/User:TUBS

Remark : As only the grid nodes matters the grid does not even have to be square-ish, it could for exemple be made of the region center coordinates as they are likely to reflect key locations.