Improve Your Machine Learning Model with Feature Engineering
By: Maissa Ouni
Tags: Modeling, Machine Learning, Data Science, Feature Engineering

Photo by Christopher Rusev on Unsplash

Most of the time raw data is a poor input, it contains valuable information but It’s concealed from designing good machine learning models until it’s been processed into a form more amenable to use, that’s where the power of feature engineering comes into play.

Feature engineering is fundamental to making machine learning models make accurate predictions but also it is a time-consuming task, we usually devote every part of our time and big effort to process the variables in our dataset before we can use them.

In this article, I will cover some of the most useful techniques that will help especially every beginner in Machine Learning to improve his model by choosing the “best” techniques.

I will be using three different datasets:

Titanic dataset

House price dataset

Boston datset

You can also check this GitHub I uploaded some useful codes

https://github.com/MaissaDataScience

So Let’s Start! Bring up a warm drink, open your Jupyter notebook & let’s CODE

Feature Engineering Techniques

· 1.Missing Data Imputation

· 2.Categorical Encoding

· 3.Variable Transformation

· 4.Discretization

· 5.Outlier Engineering

· 6.Feature Scaling

· 7.Mixed Variables Engineering

· 8.Date-Time Engineering

First of all, you need to import Pandas, Numpy libraries

import pandas

import numpy

1.Missing Data Imputation

Imputation is the act of replacing missing data with statistical estimates of the missing values. The goal of any imputation technique is to produce a complete dataset that can be used to train machine learning models.

data[['LotFrontage', 'MasVnrArea', 'GarageYrBlt']].isnull().mean() LotFrontage 0.177397

MasVnrArea 0.005479

GarageYrBlt 0.055479

dtype: float64 def impute_na(df, variable, mean_median):

return df[variable].fillna(mean_median)

Mean / Median imputation

Mean / median imputation consists of replacing all occurrences of missing values (NA) within a variable by the mean (if the variable has a Gaussian distribution) or median (if the variable has a skewed distribution). it can be calculated on continuous and discrete numerical variables

median = data['LotFrontage'].median()

data[:,'LotFrontage_median'] = impute_na(data, 'LotFrontage',median)

mean = data['MasVnrArea'].mean()

data[:,'MasVnrArea_median'] = impute_na(data, 'MasVnrArea', median)

Arbitrary value imputation

Arbitrary value imputation consists of replacing all occurrences of missing values (NA) within a variable by an arbitrary value. Typically used arbitrary values are 0, 999, -999 (or other combinations of 9s) or -1 (if the distribution is positive) or 1.

Distribution of “Age” and “Fare” of the Titanic dataset

data['Age_99'] = impute_na(data, 'age', 99)

data['fare'] = impute_na(X_train, 'fare', 0)

End of distribution imputation

Determining the value of the arbitrary value can be laborious and it is usually a manual job. We can automate this process by automatically selecting arbitrary values at the end of the variable distributions.

If the variable is normally distributed, we can use the mean plus or minus 3 times the standard deviation

If the variable is skewed, we can use the IQR proximity rule

# Because Age looks approximately Gaussian, I use the # mean and std to calculate the replacement value data.age.mean() + 3 *data.age.std()

data['Age_imputed'] = impute_na(data,'age',data.age.mean() + 3 * data.age.std())

Distribution of “LotFrange” of the House Prices dataset

#Calculate the IQR

IQR =data['LotFrontage'].quantile(0.75)-data['LotFrontage']

.quantile(0.25) # calculate the upper boundary

extreme_value =data['LotFrontage'].quantile(0.75) + 3 * IQRdata.loc[:,'LotFrontage_imputed'] = impute_na(data, 'LotFrontage', extreme_value)

Frequent category imputation | Mode imputation

Mode imputation consists of replacing all occurrences of missing values (NA) within a variable by the mode, which in other words refers to the most frequent value or most frequent category in practice, we only use this technique on categorical

# Let's find the most frequent category for BsmtQual data['BsmtQual'].mode() 0 TA dtype: object data['BsmtQual'].fillna('TA', inplace=True) 0 TA dtype: object

data['BsmtQual'].fillna('TA', inplace=True)

2.Categorical Encoding

One hot encoding

One hot encoding consists of encoding each categorical variable with different boolean variables (also called dummy variables) which take values 0 or 1, indicating if a category is present in an observation.

For example, for the categorical variable “Gender”, with labels ‘female’ and ‘male’, we can generate the boolean variable “female”, which takes 1 if the person is ‘female’ or 0 otherwise, or we can generate the variable “male”, which takes 1 if the person is ‘male’ and 0 otherwise.

#With sklearn

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(categories='auto',drop='first',sparse=False, handle_unknown='ignore')

tmp = encoder.transform(data.fillna('Missing')) #With get dummies

tmp = pd.get_dummies(data['sex'])

One hot Encoding of frequent categories

OHE of frequent or top categories is equivalent to grouping all the remaining categories under a new category, we can create dummy variables only for the most frequent categories

Integer Encoding

Integer encoding consists of replacing the categories by digits from 1 to n (or 0 to n-1, depending on the implementation), where n is the number of distinct categories of the variable.

#first let's create a dictionary with the mappings of categories to numbers

ordinal_mapping = { k: i for i, k in enumerate(X_train['Neighborhood'].unique(), 0) } # replace the labels with the integers

data['Neighborhood'] = data['Neighborhood'].map(ordinal_mapping)

Count or frequency encoding

In count encoding, we replace the categories by the count of the observations that show that category in the dataset. Similarly, we can replace the category by the frequency -or percentage- of observations in the dataset. That is, if 10 of our 100 observations show the color blue, we would replace blue by 10 if doing count encoding, or by 0.1 if replaced by the frequency.

# replace the labels with the counts

count_map = data['Neighborhood'].value_counts().to_dict()

data['Neighborhood'] =data['Neighborhood'].map(count_map) # replace the labels with the frequencies

frequency_map = (data['Exterior1st'].value_counts() / data) ).to_dict()

data['Exterior1st'] = data['Exterior1st'].map(frequency_map)

Target guided encoding

Ordered Integer Encoding

Mean Encoding

Probability ration Encoding

Weigh of Evidence Encoding

These methods allow us to capture information while pre-processing the labels of categorical variables, they create a monotonic relationship between the variable and the target.

we are going to cover one of them which is the Mean Encoding

Mean encoding implies replacing the category by the average target value for that category. For example, if we have the variable city, with categories London, Manchester, and Bristol, and we want to predict the default rate, if the default rate for London is 30% we replace London by 0.3 if the default rate for Manchester is 20% we replace Manchester by 0.2 and so on.

data.groupby(['cabin'])['survived'].mean()

ordered_labels =data.groupby(['cabin'])['survived'].mean().to_dict()

data['cabin'] = data['cabin'].map(ordered_labels)

3.Variable Transformation

Some machine learning models like linear and logistic regression assume that the variables are normally distributed. Often, variables are not normally distributed, but, transforming the variables to map their distribution to a Gaussian distribution may, and often does, boost the performance of the machine learning algorithm.

If a variable is not normally distributed, it is often possible to find a mathematical transformation to normalize its distribution.

Logarithmic transformation — np.log(X)

data['GrLivArea_log'] = np.log(data['GrLivArea'])

Reciprocal transformation — 1 / X

data['GrLivArea_reciprocal'] = 1 / (data['GrLivArea'])

Square root transformation — X**(1/2)

data['GrLivArea_sqr'] = data['GrLivArea']**(1/2)

Exponential

data['GrLivArea_exp'] = data['GrLivArea']**(1/1.5)

Box-Cox transformation

The Box-Cox transformation is defined as:

T(Y)=(Y exp(λ)−1)/λ if λ!=0, or log(Y) otherwise.

where Y is the response variable and λ is the transformation parameter. λ varies from -5 to 5. Briefly, for each λ (the transformation tests several λs), the correlation coefficient of the Probability Plot (Q-Q plot below, the correlation between ordered values and theoretical quantiles) is calculated (this optimization equation actually varies with the implementation). The value of λ corresponding to the maximum correlation on the plot is then the optimal choice for λ

import scipy.stats as stats

data['GrLivArea_boxcox'], param = stats.boxcox(data['GrLivArea'])

Yeo-Johnson transformation

Yeo-Johnson is the same as Box-Cox for the positive values of the variable, but it has different equations for the negative values of the variable as described Again, the function searches over a bunch of λ and selects the one that returns the best fit to a normal distribution.

data['Gr_yeojohnson'], param = stats.yeojohnson(data['GrLivArea'])

4.Discretization

The discretization is the process of transforming continuous variables into discrete variables by creating a set of contiguous intervals that span the range of the variable’s values., it is also called binning, where the bin is an alternative name for the interval.

There are several approaches to transform continuous variables into discrete ones. Discretization methods fall into 2 categories: supervised and unsupervised. Unsupervised methods do not use any information, other than the variable distribution, to create the contiguous bins in which the values will be placed. Supervised methods typically use target information in order to create bins or intervals.

Equal width discretization

Equal width discretization divides the scope of possible values into N bins of the same width. The width is determined by the range of values in the variable and the number of bins we wish to use to divide the variable:

width = (max value — min value) / N and N is the number of bins or intervals.

# let's capture the range of the variable age

age_range = X_train['age'].max() - X_train['age'].min()

# let's divide the range into 10 equal width bins

age_range / 10 # now let's capture the lower and upper boundaries

min_value = int(np.floor( X_train['age'].min()))

max_value = int(np.ceil( X_train['age'].max()))

# let's round the bin width

inter_value = int(np.round(age_range / 10)) # let's capture the interval limits, so we can pass them to the pandas cut

# function to generate the bins

intervals = [i for i in range(min_value, max_value+inter_value, inter_value)]

# let's make labels to label the different bins

labels = ['Bin_' + str(i) for i in range(1, len(intervals))] # create binned age / discretise age

# create one column with labels

data['Age_disc_labels'] = pd.cut(x=data['age'],bins=intervals, labels=labels, include_lowest=True)

# and one with bin boundaries

data['Age_disc'] = pd.cut(x=data['age'],bins=intervals include_lowest=True)

Discretization with k-means clustering

This discretization method consists in applying k-means clustering to the continuous variable.

Briefly, the algorithm works as follows:

1) Initialization: random creation of K centers

2) Each data point is associated with the closest center

3) Each center position is re-computed as the center of its associated points

Steps 2 and 3 are repeated until convergence is reached. The algorithm minimizes the pairwise squared deviations of points within the same cluster.

from sklearn.preprocessing import KBinsDiscretizer disc = KBinsDiscretizer(n_bins=5, encode='ordinal', strategy='kmeans') disc.fit(data[['age', 'fare']]) data_t = disc.transform(data[['age', 'fare']])

data_t = pd.DataFrame(data_t, columns = ['age', 'fare'])

Discretization with Decision Trees

Discretization with Decision Trees consists of using a decision tree to identify the optimal bins. When a decision tree makes a decision, it assigns an observation to one of n end leaves. Therefore, any decision tree will generate a discrete output, which values are the predictions at each of its n leaves.

5.Outlier Engineering

An outlier is a data point that is significantly different from the remaining data.

Statistics such as the mean and variance are very susceptible to outliers. In addition, some Machine Learning models are sensitive to outliers which may decrease their performance. Thus, depending on which algorithm we wish to train, we often remove outliers from our variables.

Check Outliers

#Visualise Outlier with Boxplot plt.subplot(1, 3, 3)

sns.boxplot(y=df[variable])

plt.title('Boxplot')

plt.show()

Boxplot of “RM” of Boston dataset

Trimming

Trimming, also known as truncation, involves removing the outliers from the dataset. We only need to decide on a metric to determine outliers.

# Let's calculate the boundaries outside which sit the outliers

# for skewed distributions distance passed as an argument, gives us the option to estimate 1.5 times or 3 times the IQR to calculate

the boundaries. def find_skewed_boundaries(df, variable, distance):

IQR = df[variable].quantile(0.75) - df[variable].quantile(0.25)

lower_boundary = df[variable].quantile(0.25) - (IQR * distance)

upper_boundary = df[variable].quantile(0.75) + (IQR * distance)return upper_boundary, lower_boundary # find limits for RM

RM_upper_limit, RM_lower_limit = find_skewed_boundaries(boston, 'RM', 1.5) # let's flag the outliers in the data set

outliers_RM = np.where(boston['RM'] > RM_upper_limit, True, np.where(boston['RM'] < RM_lower_limit, True, False)) # let's trimm the dataset

boston_trimmed = boston.loc[~(outliers_RM + outliers_LSTAT + outliers_CRIM), ]

Censoring or Capping

Censoring, or capping means capping the maximum and /or minimum of distribution at an arbitrary value. In other words, values bigger or smaller than the arbitrarily determined ones are censored. It can be done at both tails, or just one of the tails, depending on the variable and the user.

Capping IQR proximity rule

def find_skewed_boundaries(df, variable, distance): IQR = df[variable].quantile(0.75) - df[variable].quantile(0.25) lower_boundary = df[variable].quantile(0.25) - (IQR * distance)

upper_boundary = df[variable].quantile(0.75) + (IQR * distance)

return upper_boundary, lower_boundary # find limits for RM

RM_upper_limit, RM_lower_limit = find_skewed_boundaries(boston, 'RM', 1.5) # let's flag the outliers in the data set

outliers_RM = np.where(boston['RM'] > RM_upper_limit, True, np.where(boston['RM'] < RM_lower_limit, True, False))

# Now let's replace the outliers by the maximum and minimum limit boston['RM']= np.where(boston['RM'] > RM_upper_limit, RM_upper_limit, np.where(boston['RM'] < RM_lower_limit, RM_lower_limit, boston['RM']))

Capping Gaussian approximation

# calculate the boundaries outside which sit the outliers

# for a Gaussian distribution

def find_normal_boundaries(df, variable, distance):

upper_boundary = df[variable].mean() + distance * df[variable].std()

lower_boundary = df[variable].mean() - distance * df[variable].std()

return upper_boundary, lower_boundary

# find limits for RM

RM_upper_limit, RM_lower_limit = find_skewed_boundaries(boston, 'RM', 1.5)

RM_upper_limit, RM_lower_limit

# Now let's replace the outliers by the maximum and minimum limitboston['RM']= np.where(boston['RM'] > RM_upper_limit, RM_upper_limit,np.where(boston['RM'] < RM_lower_limit, RM_lower_limit, boston['RM']))

Capping Quantiles

# the boundaries are the quantiles

def find_boundaries(df, variable):

lower_boundary = df[variable].quantile(0.05)

upper_boundary = df[variable].quantile(0.95)

return upper_boundary, lower_boundary # find limits for RM

RM_upper_limit, RM_lower_limit = find_boundaries(boston, 'RM') # Now let's replace the outliers by the maximum and minimum limit boston['RM']= np.where(boston['RM'] > RM_upper_limit, RM_upper_limit, np.where(boston['RM'] < RM_lower_limit, RM_lower_limit, boston['RM']))

6.Feature Scaling

Most of the data contain features highly varying in magnitudes, units, and range most of the machine learning algorithms for regression use Euclidean distance between two data points in their computations, this is a problem, and The regression coefficient is directly influenced by the scale of the variable Feature scaling refers to the methods or techniques used to normalize the range of independent variables in our data, or in other words, the methods to set the feature value range within a similar scale

Standardisation

Standardisation involves centering the variable at zero and standardizing the variance to 1. The procedure involves subtracting the mean of each observation and then dividing by the standard deviation:

z = (x — x_mean) / std

from sklearn.preprocessing import StandardScaler scaler = StandardScaler()

scaler.fit(X_train)

X_train_scaled = scaler.transform(X_train)

X_test_scaled = scaler.transform(X_test)

Mean Normalisation

Mean normalisation involves centering the variable at zero, and re-scaling to the value range its minimum and maximum values are within the range of -1 to 1. The procedure involves subtracting the mean of each observation and then dividing by the difference between the minimum and maximum value:

x_scaled = (x — x_mean) / ( x_max — x_min)

from sklearn.preprocessing import StandardScaler , RobustScaler scaler_mean = StandardScaler(with_mean=True, with_std=False) # set up the robustscaler so that it does NOT remove the median

# but normalises by max()-min(), important for this to set up the

# quantile range to 0 and 100, which represent the min and max values

scaler_minmax = RobustScaler(with_centering=False,with_scaling=True, quantile_range=(0, 100)) scaler_mean.fit(X_train)

scaler_minmax.fit(X_train) X_train_scaled=scaler_minmax.transform(scaler_mean.transform(X_train)) X_test_scaled=scaler_minmax.transform(scaler_mean.transform(X_test))

Minimum and maximum scaling squeezes the values between 0 and 1 but the mean is not centered at zero and the standard deviation varies across variables. It subtracts the minimum value from all the observations, and then divides it by the value range:

X_scaled = (X — X.min )/ (X.max — X.min)

from sklearn.preprocessing import StandardScaler scaler_mean = StandardScaler(with_mean=True, with_std=False) # set up the robustscaler so that it does NOT remove the median

# but normalises by max()-min(), important for this to set up the

# quantile range to 0 and 100, which represent the min and max values

scaler_minmax = RobustScaler(with_centering=False,with_scaling=True, quantile_range=(0, 100)) scaler_mean.fit(X_train)

scaler_minmax.fit(X_train) X_train_scaled=scaler_minmax.transform(scaler_mean.transform(X_train)) X_test_scaled=scaler_minmax.transform(scaler_mean.transform(X_test))

Scaling to quantiles and median — RobustScaling

In this procedure, the median is removed from the observations and then they are scaled to the inter-quantile range (IQR). The IQR is the range between the 1st quartile (25th quantile) and the 3rd quartile (75th quantile).

X_scaled = X — X_median / ( X.quantile(0.75) — X.quantile(0.25) )

from sklearn.preprocessing import RobustScaler scaler = RobustScaler()

scaler.fit(X_train)

X_train_scaled = scaler.transform(X_train)

X_test_scaled = scaler.transform(X_test)

7.Mixed Variables Engineering

Mixed variables are those which values contain both numbers and labels. A mixed variable can contain numbers or labels in different observations or numbers and labels in each observation.

For example, the variable vehicle registration is an example of letters and numbers combined in each observation (e.g., NK11DGX)

The observations of the variable contain either numbers or strings

data['open_il_24m'].unique() array(['C', 'A', 'B', '0.0', '1.0', '2.0', '4.0', '3.0', '6.0', '5.0', '9.0', '7.0', '8.0', '13.0', '10.0', '19.0', '11.0', '12.0', '14.0', '15.0'], dtype=object) # extract numerical part data['open_il_24m_numerical'] = pd.to_numeric(data["open_il_24m"],errors='coerce',downcast='integer' ) # extract categorical part data['open_il_24m_categorical']=np.where(data['open_il_24m_numerical'].isnull(),data['open_il_24m'],np.nan)

The observations of the variable contain numbers and strings

data['cabin'].unique()

array(['B5', 'C22', 'E12', 'D7', 'A36', 'C101', nan, 'C62', 'B35', 'A23','B58', 'D15', 'C6', 'D35', 'C148', 'C97', 'B49', 'C99', 'C52', 'T','A31', 'C7', 'C103', 'D22', 'E33', 'A21', 'B10', 'B4', 'E40', 'B38', 'E24', 'B51', 'B96', 'C46', 'E31', 'E8', 'B61', 'B77', 'A9', 'C89', 'A14', 'E58', 'E49', 'E52', 'E45', 'B22', 'B26', 'C85', 'E17', 'B71', 'B20', 'A34', 'C86', 'A16', 'A20', 'A18', 'C54', 'C45', 'D20', 'A29', 'C95', 'E25', 'C111', 'C23', 'E36', 'D34',]) # let's extract the numerical and categorical part for cabin data['cabin_num'] = data['cabin'].str.extract('(\d+)') data['cabin_cat'] = data['cabin'].str[0] # captures the first letter

8.Date-Time Engineering

Engineering Dates

Date variables are a special type of categorical variable. By their own nature, date variables will contain a multitude of different labels, each one corresponding to a specific date and sometimes time. Date variables, when preprocessed properly can highly enrich a dataset. For example, from a date variable we can extract: Day, Month, Year, Week, Semester…

import datetime # let's parse the dates, currently cast as strings, into datetime format data['issue_dt'] = pd.to_datetime(data.issue_d) # Extracting week of year from date, varies from 1 to 52 data['issue_dt_week'] = data['issue_dt'].dt.week #Extracting month from date - 1 to 12 data['issue_dt_month'] = data['issue_dt'].dt.month

Engineering Time

Engineering time is to extract different ways of representing time from a timestamp, we can extract for example hour, minute, second, elapsed time…

import datetime df['hour'] = df['date'].dt.hour df['min'] = df['date'].dt.minute df['sec'] = df['date'].dt.second

Reference & Additional reading resources:

https://www.udemy.com/course/feature-engineering-for-machine-learning/

https://towardsdatascience.com/smarter-ways-to-encode-categorical-data-for-machine-learning-part-1-of-3-6dca2f71b159

https://towardsdatascience.com/6-different-ways-to-compensate-for-missing-values-data-imputation-with-examples-6022d9ca0779

https://medium.com/@Cambridge_Spark/tutorial-introduction-to-missing-data-imputation-4912b51c34eb

http://www.mtome.com/Publications/CiML/CiML-v3-book.pdf

https://blog.zopa.com/2017/07/20/tips-honing-logistic-regression-models/

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1431352

https://towardsdatascience.com/ways-to-detect-and-remove-the-outliers-404d16608dba

https://sebastianraschka.com/Articles/2014_about_feature_scaling.html

https://scikitlearn.org/stable/auto_examples/preprocessing/plot_all_scalin.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py