Automated Feature Engineering Tools
By: Rajneesh Jha
Tags: Feature Engineering, Meta Learning, Machine Learning, Automl, Features

But to our rescue comes some of the cool tools which automates the whole feature engineering process and creates a large pool of features in a very short span for both classification and regression tasks.

We have found following tools which automates the whole feature engineering process and creates large number of features for both relation and non-relational data. While some of them only performs feature engineering, we have some tools which also perform feature selection. Many a times these tools can generate a lot of features which may be correlated among themselves and not have any predictive power to improve the resultant metric, so it’s better to drop such features and thus making feature selection an integral part of feature engineering process.

Here is the list of some of the best tools available :-

1. FeatureTools

2. AutoFeat

3. TsFresh

4. Cognito

5. OneBM

6. ExploreKit

7. PyFeat

FeatureTools :-

One of the most popular Python library for automated feature engineering is FeatureTools, which generates a large feature set using “deep feature synthesis”. This library is targeted towards relational data, where features can be created through aggregations (e.g. given some customers (data table 1) and their associated loans (in table 2), a new feature could be the sum of each customer’s loans), or transformations (e.g. time since the last loan payment). DFS requires structured and relational data for creating new features.

There are 2 main components of FeatureTools :-

· Entity and Entity-Set : Entity can be thought of as a single data-frame while Entity-Set is combination of more than one data-frames.

· Primitives : These are basic operations like mean, mode, max etc. that can be applied to the data. It can either be a Transformation or Aggregation.

Advantages :-

1) Most popular and hence lot of resources are available.

2) We can specify the variable types.

3) Custom primitives can be created.

4) Expanding Features with respect to time can be created.

5) Best at handling relational database.

Limitations :-

1) Creates large number of features leading to curse of dimensionality.

2) For database which are not relational we will have to use normalization.

3) Does not have support for unstructured data.

4) Features extracted are basic statistical features which are aggregated independently of other columns of target variable.

AutoFeat :

AutoFeat is one of the python library which automates feature engineering and feature selection along with fitting a Linear Regression model. They generally fir Linear Regression model to make the process more explainable.

The strategy followed by AutoFeat is somewhat orthogonal to that of FeatureTools: It is not meant for relational data, found in many business application areas, but was rather built with scientific use cases in mind, where e.g. experimental measurements would instead be stored in a single table. For this reason, AutoFeat also makes it possible to specify the units of the input variables to prevent the creation of physically nonsensical features.

Here is the feature selection process used by AutoFeat :

Advantages :-

· Only open source framework for general purpose automated feature engineering which does not care about relational data.

· Also does feature selection to reduce the dimensionality problem.

· Does not create physically non-sensical features and hence useful in logistic data.

Limitations :-

· Not good at handling relational data.

· Only make simpler features like ratios, products and other basic transformations.

· Does not consider feature interaction for making new features.

· Only fit automated model for regression data not for classification problem.

TsFresh :

TsFresh, which stands for “Time Series Feature extraction based on scalable hypothesis tests”, is a Python package for time series analysis that contains feature extraction methods and a feature selection algorithm. Currently, it automatically extracts 64 features from time series data that describe both basic and complex characteristics of a time series (such as the number of peaks, average value, maximum value, time reversal symmetry statistic, etc.), and those features can be used to build regression or classification based machine learning models.

Advantages :

· Best open source python tool available for time series classification and regression.

· Can be easily integrated with FeatureTools.

Limitations :

· Can only be used for time-series data that too only good for supervised learning.

PyFeat :

PyFeat is a practical and easy to use toolkit implemented in Python for extracting various features from proteins, DNAs and RNAs.

It can only be used with genome data, hence making it not so useful for everyday classification and regression task but find its usefulness in pharma industries.

ExploreKit :

ExploreKit is based on the intuition that highly informative features often result from manipulations of elementary ones, they identify common operators to transform each feature individually or combine several of them together. it uses these operators to generate many candidate features, and chooses the subset to add based on the empirical performance of models trained with candidate features added.

Generation and Selection process is as follows:

Advantages :

· Uses meta learning to rank candidate features rather than running feature selection on all created features which can sometimes be very large.

Limitations :

· No open source implementation either in Python or R.

OneBM :

OneBM works directly with multiple raw tables in a database. It joins the tables incrementally, following different paths on the relational graph. It automatically identifies data types of the joint results, including simple data types (numerical or categorical) and complex data types (set of numbers, set of categories, sequences, time series and texts), and applies corresponding pre-defined feature engineering techniques on the given types. In doing so, new feature engineering techniques could be plugged in via an interface with its feature extractor modules to extract desired types of features in specific domain. it supports data scientists by automating the most popular feature engineering techniques on different structured and unstructured data.

Feature selection is used to remove irrelevant features extracted in the prior steps. First, duplicated features are removed. Second, if the training and test data have an implicit order defined by a column, e.g. timestamp, then drift features are detected by comparing the distribution between the value of features in the training and a validation set. If two distributions are different, the feature is identified as a drift feature which may cause over-fitting. Drift features are all removed from the feature set. Besides, we also employ Chi-square hypothesis testing to test whether there exists a dependency between a feature and the target variable. Features that are marginally independent from the target variable are removed.

Advantages :

· Works well with both relational as well as non-relational data.

· Generates simple as well as complex features as compared to FeatureTools.

· Tested in Kaggle competitions where it out performed state of the art models.

· Can be used to create feature for big data also.

Limitations :

· No open source implementation.

Cognito :

Cognito is a system that automates feature engineering but from a single database table. In each step, it recursively applies a set of predefined mathematical transformations on the table’s columns to obtain new features from the original table. In doing so, the number of features is exponential in the number of steps. Therefore, a feature selection strategy was proposed to remove redundant features. it improves prediction accuracy on UCI datasets. Since it does not support relational databases with multiple tables, in order to use Cognito, data scientists need to get as input one table produced from raw data of multiple tables.

Limitations :

· No open source implementation.

· Extra efforts needed with relational data.

We can use these techniques to come with large number of features without investing much time and focus more on other aspects of machine learning which is modelling and at scale deployment.