Feature Engineering Steps in Machine Learning : Quick start guide : Basics
Tags: Tf Idf, Machine Learning, Standardization, Feature Engineering, Normalization

This story will help you quickly learn the basics of how you can create features out of the data for your machine learning algorithm.

Encoding nominal variables:

This are categorical variables without any order in the dataset. They can be converted from categorical to numeric features using one hot encoder library or get_dummies method from pandas.

Eg. Colors as category in the dataset.

Image courtesy: AWS

They can be transformed using one hot encoding technique.

Image courtesy: AWS

from sklearn.preprocessing import LabelEncoder,OneHotEncoder

import pandas as pd df=pd.DataFrame({“cat”:[‘blue’,’orange’,’green’]}) print(‘label encoding: ‘)

lb=LabelEncoder()

print(lb.fit_transform(df[‘cat’])) print(“*”*20)

print(‘one-hot encoding: ‘)

oe=OneHotEncoder()

print(oe.fit_transform(df[[‘cat’]]).toarray()) print(“*”*20)

print(‘one-hot encoding using dummies: ‘)

print(pd.get_dummies(df[[‘cat’]]))

2. Handling missing values:

Missing values in the data can either be dropped or can be imputed.

Dropping the missing values can lead to information loss or bias in the samples. Before dropping the missing values, carefully analyse about the implications.

import pandas as pd

df=pd.DataFrame({‘Col1’:[‘a’,’b’,’c’,’d’],’Col2':[1,None,2,None]}) print(df) df=df.dropna(how=’all’)

df=df.dropna(thresh=4)

df=df.dropna(subset=[‘Col2’])

Imputing the missing values also requires knowledge about the features and the implications.

Imputation can be done using distribution properties: mean, median and frequency of categories.

from sklearn.impute import SimpleImputer

import numpy as np

import pandas as pd df=pd.DataFrame({‘Col1’:[‘a’,’b’,’c’,’d’],’Col2':[1,None,2,None]}) print(df)

print(“After imputation : “+’*’*20) imputer=SimpleImputer(missing_values=np.nan,strategy=’mean’)

df2=pd.DataFrame(imputer.fit_transform(df[[‘Col2’]]))

df=pd.concat([df,df2],axis=1)

print(df)

There are other libraries like MICEImputer and fancyimpute too.

3. Feature extraction and reduction strategies:

Try to minimize the features as much as possible to reduce curse of dimensionality. Remove the features that are highly correlated with other features and use only one of them. If possible apply dimensionality reduction techniques to decide optimal number of techniques. If required derive one feature out of multiple features and drop the base features.

4. Scaling and Normalization:

Common techniques used are

Mean/variance standardization MinMax scaling Maxabs scaling Robust scaling Normalizer

Scaling : Its done for a particular or multiple columns within the dataset.

Normalization: Its done for a row of data within dataset independently.

Mean/Variance standardization:

Image Courtesy: AWS

from sklearn.preprocessing import StandardScaler

import numpy as np

import pandas as pd df=pd.DataFrame({‘Col1’:[3,4,9,1],’Col2':[1000,3000,2000,4000]})

print(df) print(‘After Scaling ‘+”*”*20)

scale=StandardScaler()

df2=pd.DataFrame(scale.fit_transform(df),columns=[‘Col1’,’Col2'])

print(df2)

Min/Max standardization:

Image Courtesy: AWS

from sklearn.preprocessing import MinMaxScaler

import numpy as np

import pandas as pd df=pd.DataFrame({‘Col1’:[3,4,9,1],’Col2':[1000,3000,2000,4000]})

print(df) print(‘After Scaling ‘+”*”*20)

scale=MinMaxScaler()

df2=pd.DataFrame(scale.fit_transform(df),columns=[‘Col1’,’Col2'])

print(df2)

Maxabs Standardization:

Image Courtesy: AWS

from sklearn.preprocessing import MaxAbsScaler

import numpy as np

import pandas as pd df=pd.DataFrame({‘Col1’:[3,4,9,1],’Col2':[1000,3000,2000,4000]})

print(df) print(‘After Scaling ‘+”*”*20)

scale=MaxAbsScaler()

df2=pd.DataFrame(scale.fit_transform(df),columns=[‘Col1’,’Col2'])

print(df2)

Robust Standardization:

Image Courtesy: AWS

from sklearn.preprocessing import RobustScaler

import numpy as np

import pandas as pd df=pd.DataFrame({‘Col1’:[3,4,9,1],’Col2':[1000,3000,2000,4000]})

print(df) print(‘After Scaling ‘+”*”*20)

scale=RobustScaler()

df2=pd.DataFrame(scale.fit_transform(df),columns=[‘Col1’,’Col2'])

print(df2)

Normalizer:

For each row re scaling can be performed based on L1, L2 and Max norm. The scaled values are scaled with standard deviation =1. It’s widely used in text analytics.

from sklearn.preprocessing import Normalizer

import numpy as np

import pandas as pd df=pd.DataFrame({‘Col1’:[3,4,9,1],’Col2':[1000,3000,2000,4000]})

print(df) print(‘After Scaling ‘+”*”*20)

scale=Normalizer()

df2=pd.DataFrame(scale.fit_transform(df),columns=[‘Col1’,’Col2'])

print(df2)

5. Feature Transformation:

Polynomial transformations can be used to transform features.

from sklearn.preprocessing import PolynomialFeatures

import numpy as np

import pandas as pd df=pd.DataFrame({‘a’:np.random.rand(5),’b’:np.random.rand(5)})

cube=PolynomialFeatures(degree=3)

cube_features=cube.fit_transform(df)

df_cube=pd.DataFrame(cube_features)

print(df_cube)

More the degree of the polynomial, more are the chances of overfitting.

6. Text Based Features:

Text based features are created from documents using simple bag of words model counting the frequency of words within each document.

CountVectorizer:

Only considers the importance or frequency of word within a document.

from sklearn.feature_extraction.text import CountVectorizer

# list of text documents consider each sentence as document

text = [“The is a demo sentence.”,’It is summer here.’]

# create the transform

vec = CountVectorizer()

# tokenize and build vocab

vec.fit(text)

print(vec.vocabulary_)

# encode document

vec = vec.fit_transform(text)

# summarize encoded vector

print(vec.shape)

print(vec.toarray())

TfidfVectorizer:

This technique assigns weighted values to each feature word based on its frequency within a document (TF or term frequency) as well as the frequency of the feature in other documents (IDF or inverse document frequency)

Weight is calculated as a multiplication of TF*IDF.

where IDF= log [ n / df(t) ] + 1 , n=no of documents, df(t)= in how many documents term t appears.

from sklearn.feature_extraction.text import TfidfVectorizer

# list of text documents consider each sentence as document

text = [“The is a demo sentence.”,’It is summer here.’]

# create the transform

vec = TfidfVectorizer()

# tokenize and build vocab

vec.fit(text)

print(vec.vocabulary_)

# encode document

vec = vec.fit_transform(text)

# summarize encoded vector

print(vec.shape)

print(vec.toarray())

HashingVectorizer:

This technique calculates frequency only but doesn't store vocabulary dictionary in memory. The advantage is less storage but disadvantage is inability to find important features using inverse transform. Also distinct tokens can get mapped to same feature index because of the same hash value generated hence causing collision.

from sklearn.feature_extraction.text import HashingVectorizer

# list of text documents consider each sentence as document

text = [“The is a demo sentence.”,’It is summer here.’]

# create the transform

vec = HashingVectorizer()

# tokenize and build vocab

vec.fit(text)

# encode document

vec = vec.fit_transform(text)

# summarize encoded vector

print(vec.shape)

print(vec.toarray())

So these were some of the common feature engineering techniques. I hope you enjoyed the post. Keep learning, see you in the next post.

A note from the Plain English team

Did you know that we have four publications and a YouTube channel? You can find all of this from our homepage at plainenglish.io — show some love by giving our publications a follow and subscribing to our YouTube channel!