Chapter-6 How to learn feature engineering?
By: Ashish Patel
Tags: Feature Engineering, Sklearn, Ml Research Lab, Data Science, Machine Learning

1. What is characteristic engineering?

There is a saying that is widely circulated in the industry: data and characteristics determine the upper limit of machine learning, and models and algorithms just approach this upper limit. What is the characteristic project? As its name suggests, its essence is an engineering activity designed to maximize the extraction of features from raw data for use by algorithms and models. By summarizing and summarizing, people think that feature engineering includes the following aspects:

Feature processing is the core part of feature engineering. Sklearn provides a more complete feature processing method, including data preprocessing, feature selection, and dimensionality reduction. The first contact with sklearn is often attracted to its rich and convenient algorithm model library, but the feature handling library described here is also very powerful!

In this paper, the IRIS (Iris) dataset in sklearn is used to illustrate the feature processing functions. The IRIS dataset was compiled by Fisher in 1936 and contains four features (Sepal.Length, Sepal.Width, Petal.Length, Petal.Width), eigenvalues Both are positive floating point numbers in centimeters. The target value is the classification of Iris (Iris Setosa), Iris Versicolour (Iris Virginica), Iris Virginica (Virginia Iris). The code to import the IRIS data set is as follows:

2. Data preprocessing

Through feature extraction, we can get unprocessed features, and the features at this time may have the following problems:

Does not belong to the same dimension: that is, the specifications of the features are different and cannot be compared together. Non-dimensionalization can solve this problem.

Information redundancy: For some quantitative features, the effective information contained is interval division, such as academic achievement. If you only care about “pass” or not “pass”, you need to convert the quantitative test score into “1” and “0”. “” indicates passing and failing. Binarization can solve this problem.

Qualitative features cannot be used directly: some machine learning algorithms and models can only accept input from quantitative features, so qualitative features need to be converted to quantitative features. The easiest way is to specify a quantitative value for each qualitative value, but this method is too flexible and increases the work of tuning. The qualitative feature is usually converted into a quantitative feature by means of dummy coding : if there are N qualitative values, then this feature is extended to N features. When the original feature value is the i-th qualitative value, the i-th extended feature is assigned. Is 1, other extension features are assigned a value of 0. Compared with the directly specified method, the dumb coding method does not need to increase the work of the parameter adjustment. For the linear model, the use of the dumb coded feature can achieve a nonlinear effect.

There are missing values: missing values ​​need to be added.

Low information utilization: Different machine learning algorithms and models use different information in the data. As mentioned earlier, in linear models, the use of qualitative features dumb coding can achieve nonlinear effects. Similarly, polynomialization of quantitative variables, or other transformations, can achieve non-linear effects.

We use the preproccessing library in sklearn for data preprocessing to cover the solution to the above problem.

2.1 Dimensionless

Dimensionless converts data of different specifications to the same specification. Common non-dimensionalization methods are standardization and interval scaling. The premise of standardization is that the eigenvalues ​​follow a normal distribution, and after normalization, they are converted into a standard normal distribution. The interval scaling method utilizes the boundary value information to scale the range of features to a range of features, such as [0, 1].

2.1.1 Standardization

Standardization requires the calculation of the mean and standard deviation of the feature, expressed as:

The code that normalizes the data using the StandardScaler class of the preproccessing library is as follows:

2.1.2 Interval scaling method

There are many ideas for interval scaling. The common one is to use two max values ​​to scale. The formula is expressed as:

The code for interval scaling of data using the MinMaxScaler class of the preproccessing library is as follows:

2.1.3 The difference between standardization and normalization

In simple terms, standardization is to process data according to the columns of the feature matrix, which converts the feature values ​​of the samples to the same dimension by the method of z-score. Normalization is the processing of data according to the rows of the feature matrix. The purpose is that the sample vector has a uniform standard when the point multiplication operation or other kernel function calculates the similarity, that is, it is converted into a “unit vector”. The normalization formula with the rule l2 is as follows:

The code that normalizes the data using the Normalizer class of the preproccessing library is as follows:

2.2 Binary Quantitative Features

The core of quantitative feature binarization is to set a threshold. The value greater than the threshold is 1 and the value less than or equal to the threshold is 0. The formula is as follows:

The code for binarizing data using the Binarizer class of the preproccessing library is as follows:

2.3 For qualitative features dumb coding

Since the features of the IRIS data set are all quantitative features, their target values ​​are used for dummy coding (actually not required). The code for dumb encoding data using the OneHotEncoder class of the preproccessing library is as follows:

2.4 Missing value calculation

Since the IRIS data set has no missing values, a new sample is added to the data set, and all four features are assigned a value of NaN, indicating that the data is missing. The code for missing data calculations using the Imputer class of the preproccessing library is as follows:

2.5 Data Transformation

Common data transformations are polynomial-based, exponential-based, log-based functions. The four characteristics of the polynomial conversion formula with degree 2 are as follows:

The code for polynomial conversion of data using the PolynomialFeatures class of the preproccessing library is as follows:

Data transformation based on single argument function can be done in a unified way. The code for logarithmic function conversion of data using FunctionTransformer of preproccessing library is as follows:

3. Feature selection

When the data pre-processing is complete, we need to select meaningful algorithms and machine models for machine learning to train. In general, choose features from two perspectives:

Whether the feature diverge: If a feature does not diverge, for example, the variance is close to zero, that is, the sample has substantially no difference in this feature, this feature is not useful for distinguishing samples.

If a feature does not diverge, for example, the variance is close to zero, that is, the sample has substantially no difference in this feature, this feature is not useful for distinguishing samples. Correlation between features and goals: This is more obvious, and features that are highly correlated with the target should be preferred. In addition to the variance method, the other methods described in this paper are considered from the correlation.

According to the form of feature selection, the feature selection method can be divided into three types:

Filter: Filter method, which scores each feature according to divergence or correlation, sets the threshold or the number of thresholds to be selected, and selects features.

Filter method, which scores each feature according to divergence or correlation, sets the threshold or the number of thresholds to be selected, and selects features. Wrapper: A wrapper method that selects several features at a time, or excludes several features, based on an objective function (usually a predictive effect score).

A wrapper method that selects several features at a time, or excludes several features, based on an objective function (usually a predictive effect score). Embedded: The integration method, which first uses some machine learning algorithms and models to train, obtains the weight coefficients of each feature, and selects features according to the coefficients from large to small. Similar to the Filter method, but it is trained to determine the pros and cons of the feature.

We use the feature_selection library in sklearn for feature selection.

3.1 Filter

3.1.1 Variance Selection Method

Using the variance selection method, the variance of each feature is first calculated, and then the feature whose variance is greater than the threshold is selected according to the threshold. The code for selecting features using the Variance Threshold class of the feature_selection library is as follows:

3.1.2 Correlation coefficient method

Using the correlation coefficient method, the correlation coefficient of each feature to the target value and the P value of the correlation coefficient are first calculated. Use the SelectKBest class of the feature_selection library to combine the correlation coefficients to select the feature code as follows:

3.1.3 Chi-square test

The classical chi-square test is to test the correlation of qualitative independent variables to qualitative dependent variables. Suppose the independent variable has N kinds of values, and the dependent variable has M kinds of values. Consider the difference between the observed value and the expected value of the sample frequency whose independent variable is equal to i and the dependent variable is equal to j, and construct the statistic:

It is not difficult to find that the meaning of this statistic is simply the correlation of the independent variable to the dependent variable . Use the SelectKBest class of the feature_selection library in conjunction with the chi-square test to select the feature code as follows:

3.1.4 Mutual Information Method

Classical mutual information is also used to evaluate the correlation of qualitative independent variables to qualitative dependent variables. The mutual information calculation formula is as follows:

In order to process quantitative data, the maximum information coefficient method is proposed. The code using the SelectKBest class of the feature_selection library combined with the maximum information coefficient method to select features is as follows:

3.2 Wrapper

3.2.1 Recursive Feature Elimination

The recursive elimination feature method uses a base model to perform multiple rounds of training. After each round of training, the features of several weight coefficients are eliminated, and the next round of training is performed based on the new feature set. The code that uses the RFE class of the feature_selection library to select features is as follows:

3.3 Embedded

3.3.1 Penalty-based feature selection

Using the base model with penalty terms, in addition to filtering out the features, dimensionality reduction is also performed. Use the SelectFromModel class of the feature_selection library in conjunction with the logistic regression model with the L1 penalty to select the feature code as follows:

In fact, the principle of dimension reduction of the L1 penalty term is to retain one of the features that have equal relevance to the target value , so the unselected feature does not represent unimportant. Therefore, it can be optimized in combination with the L2 penalty term. The specific operation is as follows: if a feature has a weight of 1 in L1, a feature that has a small difference in weights in L2 and a weight of 0 in L1 constitutes a homogeneous set, and the features in the set are equally divided into L1. Weight, so you need to build a new logistic regression model:

Use the SelectFromModel class of the feature_selection library in conjunction with the logistic regression model with L1 and L2 penalty terms to select the feature code as follows:

3.3.2 Feature selection based on tree model

In the tree model, GBDT can also be used as a base model for feature selection. The code of the feature is selected by using the SelectFromModel class of the feature_selection library in conjunction with the GBDT model.

4. Dimensionality reduction

When the feature selection is completed, the model can be directly trained, but the feature matrix is ​​too large, resulting in a large amount of computation and a long training time. Therefore, it is also necessary to reduce the dimension of the feature matrix. Common dimensionality reduction methods In addition to the L1 penalty-based model mentioned above, there are principal component analysis (PCA) and linear discriminant analysis (LDA). Linear discriminant analysis itself is also a classification model. PCA and LDA have many similarities, the essence of which is to map the original sample to the lower-dimensional sample space, but the mapping target of PCA and LDA is different: PCA is to make the mapped sample have the largest divergence. LDA is designed to give the mapped sample the best classification performance . So PCA is an unsupervised dimensionality reduction method, and LDA is a supervised dimensionality reduction method.

4.1 Principal Component Analysis (PCA)

The code for selecting features using the PCA class of the decomposition library is as follows:

4.2 Linear Discriminant Analysis (LDA)

The code for selecting features using the LDA class of the lda library is as follows:

References: