Using Multilabelbinarizer from sklearn
By: Devashish Thakar
Tags: Python, Feature Engineering, Beginners Guide

U sing Multilabelbinarizer from sklearn

Working with textual data is perhaps the toughest part of data science. One of the challenges is to convert long lists or dictionaries into features within the dataset for further modelling and analysis. I illustrate a problem I faced and a possible solution that utilizes the Multilabebinarizer in sklearn

I take out a chunk of three observations from a massive dataset of automobile models and their features. For illustration, it looks like this:

# Import the data and see what's contained

data = pd.read_excel(‘dictionary_like_features.xlsx’)

data.head()

Zooming into the above snapshot to get a better sense of how the ‘features’ column is structured:

data[‘features’][1]

A few observations regarding the ‘features’ column:

i) Features are split between ‘basic’ and ‘top’

ii) ‘basic’ features have a value for each key (e.g., ‘Fuel Type’: ‘Gasoline’)

iii) ‘top’ features have a list of values for each key(e.g., ‘Safety’: [‘Brake Assist’, ‘Stability Control’])

This might seem like a very specific example, but the basic idea is to try to handle a nested textual data structure.

Next step, suppose I am interested in the ‘top’ features and want to create a unique column for each top feature encountered in the dataset. Here are a few pre-processing steps before applying Multilabelbinarizer:

# To evaluate all entries as seperate strings rather than one big #string

data[‘features’] = data.features.apply(literal_eval) # To create columns for all the features from the features column

data = data.join(pd.io.json.json_normalize(data.features)) # Fill NaNs with blanks since Multilabelbinarizer cannot handle that

data = data.replace(np.nan, ‘’, regex=True)

The basic and top features look something like this after pre-processing:

Basic features

Top features

Thus, all basic features have a column for each feature type and an associated value, while the top features have a list of values instead of a just one value. The last step is to seperate all the values in the top features into individual columns using Multilabelbinarizer. I define a function called ‘featurize’ to implement these steps:

def featurize(df):

‘’’

Initial specific columns that would be converted to binary features

‘’’

s1 = df[‘top.Seating’]

s2 = df[‘top.Safety’]

s3 = df[‘top.Convenience’]

s4 = df[‘top.Entertainment’]



‘’’

Initiate an instance of multilabel binarizer

‘’’

mlb = MultiLabelBinarizer()



‘’’

Apply the binarizer to the specific columns to create seperate dataframes

for each featurized column

‘’’

Seating = pd.DataFrame(mlb.fit_transform(s1),columns=mlb.classes_, index=df.index)

Safety = pd.DataFrame(mlb.fit_transform(s2),columns=mlb.classes_, index=df.index)

Convenience = pd.DataFrame(mlb.fit_transform(s3),columns=mlb.classes_, index=df.index)

Entertainment = pd.DataFrame(mlb.fit_transform(s4),columns=mlb.classes_, index=df.index)



‘’’

Extract the final dataframe by concatenating all the four above

‘’’

extract = pd.concat([Seating, Safety, Convenience, Entertainment], axis=1)



return(extract)

As can be seen below, each feature in the top features has a column with a binary value indicating whether that particular feature exists in the car model

# Apply and see how the top features look like

final = featurize(data)

final.head()