Feature Engineering techniques in Python | by Defend Intelligence | Towards Data Science

Features engineering is a crucial part in each Machine Learning project. In this article we will go around some techniques to handle this task. Please do not hesitate to comment with new ideas, I will try to keep this article updated as much as possible.
When performing features engineering, in order to have a general model, it is always recommended to work on the whole DataFrame, if you have two 2 files juste merge them (train and test).
df = pd.concat([train[col],test[col]],axis=0)#The label column will be set as NULL for test rows# FEATURE ENGINEERING HEREtrain[col] = df[:len(train)]test[col] = df[len(train):]
Sometimes the type encoding of a column is not the best choice, as for example encoding in int32 a column containing only value from 0 to 10. One of the most popular function used a function to reduce the memory usage by converting the type of column to the best type as possible.
A common way to remove outliers is to use the Z-score.
If you are looking to remove each row where at least one column contains an outlier (defined with the Z-score) you can use the following code :
from scipy import statsdf[(np.abs(stats.zscore(df)) < 3).all(axis=1)]
Some tree based algorithm can handle NAN value but he will had a step between NAN et non-NAN value, that could be non sense sometime. A common trick is just to fill all nan value by a value lower than the lowest value in the column considered (for example -9999).
df[col].fillna(-9999, inplace=True)
You can treat categorical features with a label encoding to deal with them as a numeric. You can also decide to treat them as category. I recommend to try both and keep what improve you Cross-Validation by this line of code (after label encoding).
df[col] = df[col].astype('category')
Sometime string variable contain multiple information in one variable. For example FRANCE_Paris . You will need to split it with a regex or using a split method for example :
new = df["localisation"].str.split("_", n = 1, expand = True)df['country'] = new[0]df['city']=new[1]
Otherwise, two (string or numeric) columns can be combined into one column. For example a column with a department of france (75 , for Paris) and the district code (001) can become a zip code : 75001)
df['zipcode'] = df['departement_code'].astype(str)                +'_'                +df['disctrict_code'].astype(str)
One of the common feature engineering is to apply simple mathematical operation to create new feature. For example if we have the width and the height of a rectangle we can calculate the area.
df['area'] = df['width'] * df['height']
Create a column that create a column from the popular value_count method is a powerful technique for tree based algorithm, to define if a value is rare or common.
counts = df[col].value_counts.to_dict()df[col+'_counts'] = df[col].map(counts)
Dealing with Date and parse each element of a date is crucial in order to analyze event.
First things with we need to convert our Date column (often considered as a string column with pandas). One of the most important field is to know how to use the format parameters. I strongly recommend to save this site as bookmark ! :)
For exampel if we are looking to convert a date column with this following format : 30 Sep 2019 we will use this piece of code:
df['date'] =  pd.to_datetime(df[col], format='%d %b %Y')
Once your column is converted to datetime we may need to extract date components in news columns :
df['year'] =  df['date'].yeardf['month'] = df['date'].monthdf['day'] = df['date'].day
In order to continue to detect rare and common value, that is really imporant for Machine Learning prediction, we can decide to detect if a value is rare or common in a subgroup based on a static method. For example here we will like to know which Smartphone brand user do the longest call by calculating the mean of each subclass.
temp = df.groupby('smartphone_brand')['call_duration']       .agg(['mean'])       .rename({'mean':'call_duration_mean'},axis=1)df = pd.merge(df,temp,on='smartphone_brand',how=’left’)
With this method a ML algorithm will be able to tell which call have a non common value of call_duration regarding the smartphone brand.
Normalization could be sometime really useful.
In order to achieve a normalization of a column against itself:
df[col] = ( df[col]-df[col].mean() ) / df[col].std()
Or you can normalize one column against another column. For example if you create a Group Statistic (described above) indicating the mean value for call_duration each week. Then you can remove time dependence by
df[‘call_duration_remove_time’] = df[‘call_duration’] — df[‘call_duration_week_mean’] 
The new variable call_duration_remove no longer increases as we advance in time because we have normalized it against the affects of time.
Each column add time computing for your preprocessing but also for your model training. I strongly recommend to test a new feature and see how the features improve (or not...) you evaluation metrics. If it is not the case you should just remove the feature created / modified.