An Easier Way to Encode Categorical Features | by Rebecca Vickery | Towards Data Science

I have recently been working on a machine learning project which had several categorical features. Many of these features were high cardinality, or in other words, had a high number of unique values. The simplest method of handling categorical variables is usually to perform one-hot encoding, where each unique value is converted into a new column with 1 or a 0 denoting the presence or absence of this value. However, when the cardinality of a feature is high this method will often produce so many new features that the model performance decreases.
I started to write my own encoders to try alternative methods to encode some of the features starting with something called weight of evidence. In a binary classification problem weight of evidence uses the distribution of unique values in the feature in both the positive and negative class and creates a new feature relating to these values. Naturally, this took a while to encode and then get it to work in my existing scikit-learn pipeline.
Then I stumbled across this library called category_encoders which has, not only weight of evidence but pretty much every possible way to encode categorical features already written and ready to use. This means that I no longer need to write this custom encoder and I can now quickly evaluate a whole host of different encoders and select the best one. In this post, I want to share this library and give an example of how it can be used in a scikit-learn pipeline.
This library comprises a set of transformers which follow the scikit-learn style which means that as well as being used alone, they can also be used inside a scikit-learn pipeline. The transformers provide a wide variety of methods to transform categorical data including the very popular one-hot encoding. This library is particularly useful for handling high cardinality features where the one-hot encoding method is likely to result in poorer model performance.
Let’s look at an example of this in use. In the following example I am using the adults dataset which I have downloaded from the UCI machine learning repository. This data comprises a set of characteristics about each person and the target variable which denotes whether they earn under or over $50k per year.
The library can be installed via pip.
pip install category_encoders
Or Conda.
conda install -c conda-forge category_encoders
To start with here are the imports I am using.
import pandas as pdimport numpy as npfrom sklearn import preprocessingfrom sklearn.model_selection import train_test_splitfrom sklearn.pipeline import Pipelinefrom sklearn.ensemble import RandomForestClassifierfrom sklearn.impute import SimpleImputerfrom sklearn.preprocessing import StandardScalerfrom sklearn.compose import ColumnTransformerfrom sklearn.metrics import f1_scoreimport category_encoders as ce
Next, I am downloading the data and transforming it into a pandas data frame.
url_data = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data'column_names = ['age', 'workclass', 'fnlwgt', 'education', 'educational-num','marital-status',                'occupation', 'relationship', 'race', 'gender','capital-gain', 'capital-loss',                 'hours-per-week', 'native-country','income']adults_data = pd.read_csv(url_data, names=column_names)
I then create a variable for each feature type, categorical and numerical, for later use in the pipeline, and splitting the values into test and train data sets. One important thing to note is that although scikit-learn can handle non-numerical target variables, the category_encoders library cannot. So one additional step here is to use the LabelEncoder to transform the y labels.
numeric_features = adults_data.select_dtypes(include=['int64', 'float64']).columnscategorical_features = adults_data.select_dtypes(include=['object']).drop(['income'], axis=1).columnsX = adults_data.drop('income', axis=1)y = adults_data['income']le = preprocessing.LabelEncoder()label_encoder = le.fit(y)y = label_encoder.transform(y)X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
Next, I am running the below code which constructs a pipeline and loops through a list of category_encoders printing the score for each model. I have used a Random Forest model as a simple example.
encoder_list = [ce.backward_difference.BackwardDifferenceEncoder,                ce.basen.BaseNEncoder,               ce.binary.BinaryEncoder,                ce.cat_boost.CatBoostEncoder,                ce.hashing.HashingEncoder,                ce.helmert.HelmertEncoder,                ce.james_stein.JamesSteinEncoder,                ce.one_hot.OneHotEncoder,                ce.leave_one_out.LeaveOneOutEncoder,                ce.m_estimate.MEstimateEncoder,                ce.ordinal.OrdinalEncoder,                ce.polynomial.PolynomialEncoder,                ce.sum_coding.SumEncoder,                ce.target_encoder.TargetEncoder,                ce.woe.WOEEncoder                ]for encoder in encoder_list:        numeric_transformer = Pipeline(steps=[    ('imputer', SimpleImputer(strategy='median')),    ('scaler', StandardScaler())])    categorical_transformer = Pipeline(steps=[    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),    ('woe', encoder())])        preprocessor = ColumnTransformer(    transformers=[        ('num', numeric_transformer, numeric_features),        ('cat', categorical_transformer, categorical_features)])        pipe = Pipeline(steps=[('preprocessor', preprocessor),                      ('classifier', RandomForestClassifier(n_estimators=500))])        model = pipe.fit(X_train, y_train)        y_pred = model.predict(X_test)    print(encoder)    print(f1_score(y_test, y_pred, average='macro'))
The output is as follows. You can see from the below that for this model the ordinal encoder gives the best score and the leave one out encoder gives the lowest.
Handling categorical variables is just one aspect of tuning a machine learning model there are of course many other steps involved. The category_encoder library provides a way to very quickly evaluate a number of different methods for handling these features. It is definitely something I will be using often in future projects.
Thanks for reading!