One Hot Encoding, Standardization, PCA: Data preparation for segmentation in python | by Indraneel Dutta Baruah | Towards Data Science

Data driven customer targeting or product bundling are critical for businesses to stay relevant against the intense competition they face. Consumers are now spoilt for choice and prefer personalized product offerings. With the coming of the fourth industrial revolution in the form of the immense growth of artificial intelligence and big data technologies, there is no better time to leverage segmentation models to perform such analysis. But before we do a deep dive into these models, we should be aware of what kind of data is needed for these models. This is the focus of my blog as we will be going through all the steps necessary for transforming our raw dataset to the format we need for training and testing our segmentation algorithms.
The Data
For this exercise, we will be working with clickstream data from an online store offering clothing for pregnant women. It has data from April 2008 to August 2008 and includes variables like product category, location of the photo on the webpage, country of origin of the IP address and product price in US dollars. The reason I chose this dataset is that clickstream data is becoming a very important source of providing fine-grained information about customer behaviour. It also provides us a dataset with typical challenges like high dimensionality, need for feature engineering, presence of categorical variables and different scales of fields.
We will try to prepare the data for product segmentation by performing the following steps:
Exploratory Data Analysis (EDA)Feature EngineeringOne Hot EncodingStandardisationPCA
Exploratory Data Analysis (EDA)
Feature Engineering
One Hot Encoding
Standardisation
PCA
Exploratory Data Analysis (EDA)
We will first try to read the dataset (using read_csv function) and look at the top 5 rows (using head function):
# Read dataset and look at top recordsimport pandas as pddf = pd.read_csv('e-shop clothing 2008.csv', delimiter=";")df.head(5)
We have the data at a daily level for multiple clothing models (field name: “page 2 (clothing model)”). Next, let us check the number of rows and columns and their types (using the info function)
#Check the number of rows and columns and their typesdf.info()
We have 165474 records and 14 fields. One thing to note is that a lot of fields are numeric but should ideally be strings. Let us convert the fields as string using as.type(str) function:
# Convert categorical variables to stringcat_vars = ['year', 'month', 'day', 'country', 'session ID',               'page 1 (main category)', 'page 2 (clothing model)',   'colour',               'location', 'model photography', 'price 2', 'page']df[cat_vars] = df[cat_vars].astype(str)df.info()
Let us check the properties of the numeric fields next:
# Check properties of numeric fieldsdf.describe()
As seen in figure 4, the product price (field name: ‘price’) is on a much larger scale than sequence of clicks during one session (field name: ‘order’). This means that we will have to standardize these fields to bring them to the same scale as distance based models like K-means are affected by the scale of the fields.
Feature Engineering
As previously mentioned, our dataset is at a daily level and we need to aggregate the data at a product level because we want to perform product segmentation. We create the following features while aggregating at the product level:
Most frequently occurring product colour, day of browsing, country, photo type (profile, en face), price type (higher or lower than category average), page number within website and location of the product’s photo on the page (using the mode function)Total number of unique session IDs (using the nununique function)Median, minimum and maximum of sequence of clicks during one session and product price (using the median, min and max function)
Most frequently occurring product colour, day of browsing, country, photo type (profile, en face), price type (higher or lower than category average), page number within website and location of the product’s photo on the page (using the mode function)
Total number of unique session IDs (using the nununique function)
Median, minimum and maximum of sequence of clicks during one session and product price (using the median, min and max function)
# Feature Engineeringfrom scipy.stats import mode df2 = df.groupby(['country','page 1 (main category)','page 2 (clothing model)']).agg(                                                                             median_no_of_clicks_per_session=('order', 'median'),                                                                             min_no_of_clicks_per_session=('order', 'max'),                                                                             max_no_of_clicks_per_session=('order', 'min'),                                                                             median_price=('price', 'median'),                                                                             min_price=('price', 'max'),                                                                             max_price=('price', 'min'),                                                                             total_number_of_sessions =('session ID', pd.Series.nunique),                                                                             most_frequent_day=('day', lambda x: mode(x)[0][0]),                                                                             most_frequent_colour=('colour', lambda x: mode(x)[0][0]),                                                                             most_frequent_location=('location', lambda x: mode(x)[0][0]),                                                                             most_frequent_photo_type=('model photography', lambda x: mode(x)[0][0]),                                                                             most_frequent_price_type =('price 2', lambda x: mode(x)[0][0]),                                                                             most_frequent_page_number =('page', lambda x: mode(x)[0][0])                                                                            )df2
One Hot Encoding
One hot encoding creates dummy variables which is a duplicate variable which represents one level of a categorical variable. Presence of a level is represented by 1 and absence is represented by 0. If the categorical variable is ordinal (i.e. categories of the variable have an order) then we can translate the variable to a numeric variable using the OrdinalEncoder function. In our case, the categorical variables don’t have any ordinality and hence, we use the get_dummies function to create the dummy variables.
# One hot encoding - to convert categorical data to continuouscat_vars = ['most_frequent_day',           'most_frequent_colour', 'most_frequent_location',           'most_frequent_photo_type', 'most_frequent_price_type',           'most_frequent_page_number']df2[cat_vars] = df2[cat_vars].astype(str)df3 = pd.get_dummies(df2)df3.head(5)
We can also use the OneHotEncoder function instead of get_dummies function if our nominal features are integers.
Standardization
As shown in figure 4, our numeric features have different scales. Scaling helps to compare independent features with different ranges or units after converting them to comparable values. There are two major scaling methods:
Normalization is the ideal choice when we know that the distribution of data does not follow a Gaussian distribution or for algorithms that do not assume any data distribution like K-Nearest Neighbors and Neural Networks. On the other hand, standardization can be used when data follows a Gaussian distribution. But these are not strict rules and ideally we can try both and select the option which gives the best cluster validation results. We will be standardizing our numeric fields in this example using the StandardScaler function.
# Standardizingfrom sklearn.preprocessing import StandardScalercon_vars = ['median_no_of_clicks_per_session', 'min_no_of_clicks_per_session',           'max_no_of_clicks_per_session', 'median_price', 'min_price',           'max_price', 'total_number_of_sessions']scaler = StandardScaler()df3[con_vars]=scaler.fit_transform(df3[con_vars])df3.head(5)
Principal Component Analysis (PCA)
Principal component analysis combines our current features in a specific way to create new features and then we can drop the “least important” while still retaining the most valuable parts of all of the original variables. This is a useful method when we have a lot of features to handle. It calculates the covariance matrix of all the features and then generates the eigenvectors and eigenvalues from the matrix. Then, the covariance matrix is multiplied by the eigenvectors to create principal components. These principal components are the new features based on our original features and their importance in terms of explaining the variability in the dataset is given by eigenvalues. We can keep the top ranked principal components which explain a minimum level of variance in our original dataset.
We can implement PCA analysis using the pca function from sklearn.decomposition module. I have set up a loop function to identify number of principal components that explain at least 85% of the variance in the dataset.
# PCAfrom sklearn.decomposition import PCA# Loop Function to identify number of principal components that explain at least 85% of the variancefor comp in range(3, df3.shape[1]):    pca = PCA(n_components= comp, random_state=42)    pca.fit(df3)    comp_check = pca.explained_variance_ratio_    final_comp = comp    if comp_check.sum() > 0.85:        break        Final_PCA = PCA(n_components= final_comp,random_state=42)Final_PCA.fit(df3)cluster_df=Final_PCA.transform(df3)num_comps = comp_check.shape[0]print("Using {} components, we can explain {}% of the variability in the original data.".format(final_comp,comp_check.sum()))
As seen in figure 8, 15 components are able to explain 85% of the variance in our dataset. We can now use these features in our unsupervised models like K means, DBSCAN, hierarchical clustering etc to segment our products.
Conclusion
In this post, we learnt about the steps needed to prepare data for segmentation analysis.
Specifically, we learned:
How we should perform exploratory data analysis by looking at the data, the field types and the properties of numeric fields.
Examples of what kind of features we can create from the raw categorical and continuous fields.
How to implement one hot encoding in python as well as ordinal encoding
Various types of scaling techniques and how to choose between them
What is PCA and how to use it in python for feature reduction
Do you have any questions or suggestions about this blog? Please feel free to drop in a note.
Finally, I encourage you to check out the article below for an in-depth explanation of different methods for selecting optimal number of clusters for segmentation:
Cheat sheet to implementing 7 methods for selecting optimal number of clusters in Python
If you are, like me, passionate about AI, Data Science, or Economics, please feel free to add me on LinkedIn.