What is Feature Engineering?
By: Ahmad Sachal
Tags: Feature Engineering, Data Analysis, Python, Data Science, Machine Learning

In this article, we’ll look at what Feature Engineering is and discuss some techniques used for feature engineering.

Feature

In machine learning and data science, a feature is an individual measurable property or characteristic of a phenomenon, usually in the form of structured columns. For instance, we have a simple data table with students and their math exam marks as given below:

Name Maths

0 Chandler 71

1 Ross 77

2 Monica 91

The columns i.e., Name and Maths are features in the data.

What Is Feature Engineering?

Feature Engineering is the step in data science in which you extract features from raw data to facilitate data analysis or machine learning.

When the goal is data analysis or machine learning, you need data that does not contain outliers, missing values, or unneeded features. Moreover, you need data that can be analyzed. If you want to plot graphs, you need to trim and scale your data accordingly. In machine learning, you need to engineer the data to make it compatible/standardized so that you can design an efficient machine learning model.

Feature Engineering Techniques

We will briefly discuss various techniques that are commonly used for feature engineering.

Imputation — Dealing with Missing Values

When we are working with data, we get a lot of null or missing values. These missing values must always be taken seriously since they can affect the analysis quite a lot and can pretty much dampen the performance of our machine learning models. There are a few ways to cater to this problem.

Dropping Rows with Missing Values: We can drop the rows containing missing values. It is important, though, that we look and identify if the features we’re basing our analysis or model on have missing values or not, since we don’t want to drop significant data due to missing values of features that aren’t even going to be much significant for us. We can then drop the rows for those missing values. This isn’t the best method to deal with the problem because, in some cases, it could really reduce the data size and do more harm than good. So, you have the method of imputation to do a much better job for you here.

We have two types of imputation methods depending on the type of data.

Numerical Imputation — For missing values of numerical data, we can either change the values to 0 or set it to the mean value of the columns as:

data = data.fillna(0) #Replace NA value with 0 data = data.fillna(data.mean()) #Replace NA value with the mean value

Categorical Imputation — For missing values of categorical data, we can change the value to the most commonly occurring value or change it to “other”.

data['column'].fillna(data['column'].value_counts().idxmax(), inplace=True)

One-Hot Encoding

This is a feature engineering method to convert categorical data into a numeric form and is one of the most common ones used in machine learning. We cannot always work with categorical data directly for machine learning, in fact in most cases we can’t. We need the categorical data converted into the numeric form so that we could implement our algorithms efficiently.

In one-hot encoding, we convert our categorical data column into multiple columns of ones and zeros. For example, if we have a column named “colors” containing three values i.e., [‘red’, ‘green’, ‘blue’], after the encoding method, we’d get:

red, green, blue

1, 0, 0

0, 1, 0

0, 0, 1

Sklearn provides us a class for One-Hot Encoding and it can be imported and applied to the data with just one line. Documentation can be seen here.

Feature Scaling — Normalization and Standardization

In machine learning, we need the features to have comparable value ranges. For example, we can not compare values of height and weight since they have different value ranges and completely different units, but we can scale the variables in a way that would make it easier to compare them and use the data efficiently. Following are the two common methods for feature scaling:

Normalization

In normalization, we scale all the features into a fixed range from 0 to 1. The formula used for normalization is:

This method is also called min-max normalization.

Standardization

In standardization, values are centered around the mean value with unit variance. The formula is:

where μ is mean and σ is the standard deviation

Handle Outliers

Outliers are the values that are either too high or too low relative to the overall data. Such values affect the skewness of the data and also affect the model prediction. It is, therefore, very important to handle outliers. Some of the methods are as follows:

Remove the values: One way of dealing with outliers is removing them from the overall distribution, but this method is not always the most efficient one since important data could be lost doing this.

One way of dealing with outliers is removing them from the overall distribution, but this method is not always the most efficient one since important data could be lost doing this. Imputation: We could just treat the outlier values like the missing values and do imputations as discussed before.

We could just treat the outlier values like the missing values and do imputations as discussed before. Capping the max/min values: In this method, we can replace the max/min values with some arbitrary max/min values that would indicate that above or below that value, the variable will behave in a similar way.

Log Transform

In order to handle skewed data (to make it more normal), we use logarithmic transformation. This method has more than one advantage as this also reduces our outlier problem by reducing the effect an outlier creates.

The important thing to keep in mind is that this method cannot be implemented directly on negative-valued data since that’ll throw an error. So either a different approach should be used or 1 could be added to change the data values to positive i.e., log(x+1).

Feature Split

Splitting data logically is a good way to extract more useful information from the available data. The implementation of the split is subject to the available data and the information it can provide. For a simple example, let’s say we have data with full names of people and their ages as below:

Name Age

Mr. Rob Stark 25

Mr. Balian de Ibelin 30

Miss Mary Jane 23

Mrs. Caitylin Stark 50

In the above data, you can see that genders are not mentioned, but we can extract this information by titles provided in the names i.e., Mr, Mrs, etc.

You can string split the title part into a new column and categorize the available data to genders. This way you now can provide the model new information.

Apart from feature splitting, grouping methods are also used for feature extraction and get better results.

Conclusion

Whether we are to do analysis on data or want to make a good prediction model, we need to do feature engineering. It is the main step toward getting good results and usually takes a lot longer than the actual machine learning or analysis. It is, therefore, key to becoming well-versed in the techniques used for feature engineering. The article is not an exhaustive list of all the feature engineering techniques but is of the most common and useful ones. The main thing to focus on is that we have to extract/create/engineer the most significant and compatible features from raw data in accordance with the required task.