Exploratory Data Analysis, Categorical Data — Part II | by Priyanka Banerjee | Towards Data Science

“Coming up with features is difficult, time-consuming, requires expert knowledge. ‘Applied machine learning’ is basically feature engineering.”
— Prof. Andrew Ng.
Data scientists spend close to 75% of their time in analyzing data and engineering features which are indeed a difficult and time-consuming processes. They require domain knowledge along with mathematical computations.
Exploratory Data Analysis and Feature Engineering are the processes of selecting, profiling and transforming data into features which can be fetched as inputs for machine learning models to make predictions. We should remember that good quality features always help in improving the overall model performance. Many times even though the machine learning task can be same in different scenarios but the features extracted after analyzing the data in each scenario will be different from the other.
The above data set has a feature “predicted_category” which detects the emotion of the person based on his statement which is present in the feature “cleaned_hm”. “Raw features” are obtained directly from the data set with no extra data manipulation or engineering whereas “Derived features” are obtained from feature engineering, where we extract features from existing data attributes.
nycdata = pd.read_csv(“train.csv”)nycdata.head(5)
nycdata.shape(50000, 8)
Now, after certain feature engineering, let's check :
nycdata.head(5)
nycdata.shape(50000, 12)
Hence, we have extracted 4 more features from our original raw data set through feature engineering.
In this article, I will deal with data analysis and a bit of feature engineering of Categorical Data.
Let us consider a simple example where the statements of people determine their emotion. Now, if I consider all to be happy in this data set, happiness is categorized into several groups — Affection, Achievement, Leisure, Bonding. Eg- Someone has got a promotion in his workplace hence, he is happy which denotes “Achievement” whereas someone else is happy because he had a warm conversation with his friends after a long long time which clearly shows “Bonding”, a mother is happy because of her child’s birthday which shows again “Affection”. We can clearly see here is no way of ordering these values of the attribute of “Happiness”.
But there are certain ordinal categorical data like shoe sizes or sizes of tee shirts or education level, etc. which follow a particular order.
I prefer to start always by loading all the required necessary dependencies.
import numpy as np #linear algebraimport pandas as pd #data processingimport os #operating system dependent modules of pythonimport matplotlib.pyplot as plt #visualizationimport seaborn as sns #visualization%matplotlib inlinefrom nltk.corpus import stopwordsfrom nltk.stem import PorterStemmerfrom sklearn.preprocessing import LabelEncoderimport refrom wordcloud import WordCloud, ImageColorGeneratorfrom sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
Loading the .csv file
at.head(5)
Let us get the information of our data set.
at.info()Output--<class 'pandas.core.frame.DataFrame'>RangeIndex: 60321 entries, 0 to 60320Data columns (total 5 columns):hmid                  60321 non-null int64reflection_period     60321 non-null objectcleaned_hm            60321 non-null objectnum_sentence          60321 non-null int64predicted_category    60321 non-null objectdtypes: int64(2), object(3)memory usage: 2.3+ MB
In case of a big data set having several features, I can get the name of all the features by the command :
dataset.columns
Details information of the data set is obtained by :
at.describe()
I can get a description of those features which have numeric data. In case I want a description of all the features, I will use :
at.describe(include='all')
This is a descriptive statistics to summarize the central tendency, dispersion and shape of a data set’s distribution.
Encoding Categorical Variable:
I need to predict the “happiness” based on the sentences. Initially I have mapped an index to the different labels of “happiness” :
from sklearn.preprocessing import LabelEncodercols = [‘predicted_category’]lbl = LabelEncoder()pred_lbl = lbl.fit_transform(at[cols])mappings = {index: label for index, label in enumerate(lbl.classes_)}mappings{0: 'achievement', 1: 'affection', 2: 'bonding', 3: 'enjoy_the_moment', 4: 'exercise', 5: 'leisure', 6: 'nature'}
Through the function “Label Encoder”, an object “lbl” has mapped a number to each value of the “predicted_category” feature.
Now, I need to apply a specific encoding scheme to our categorical variables. The reason is simple. If I directly feed the pred_lbl attribute as a feature in a machine learning model, it would consider it to be a continuous numeric feature thinking value 6(‘Nature’) is greater than 2(‘Bonding’) but that is meaningless because ‘Nature’ is certainly not bigger or smaller than ‘Bonding’, These are actually categories which cannot be compared directly. Hence I need an additional layer of encoding schemes where dummy features are needed to be created for each unique value.
dummy_features = pd.get_dummies(at['predicted_category'])at=pd.concat([at[['hmid','cleaned_hm']],dummy_features],axis=1)at.head(9)
Since our predicted_category is only dependent on the statements of the people, hence we have considered “cleaned_hm” along with “hmid”
Clean and pre-process raw text :
The feature “cleaned_hm” contains messy raw data which we need to clean.
stops = set(stopwords.words("english"))def cleanData(text, lowercase = False, remove_stops = False, stemming = False):    txt = str(text)#print(txt)    txt = re.sub(r'[^A-Za-z0-9\s]',r'',txt)#print(txt)    txt = re.sub(r'\n',r' ',txt)#print(txt)    #convert whole text to lower case & remove stopwords and stemmers    if lowercase:        txt = " ".join([w.lower() for w in txt.split()])        if remove_stops:        txt = " ".join([w for w in txt.split() if w not in stops])        if stemming:        st = PorterStemmer()        txt = " ".join([st.stem(w) for w in txt.split()])#print(txt)    return txt# clean descriptionat['cleaned_hm'] = at['cleaned_hm'].map(lambda x: cleanData(x, lowercase=True, remove_stops=True, stemming=True))
Unlike the raw data set, I have a much compact information now for my data to be fed for machine learning. The function “cleanData” has removed punctuation and words that are not required and has converted it into complete lower case.
Now, all my features are in numeric format except “cleaned_hm”. I have to get the words that have the most frequencies. The same can be achieved by using WordCloud.
#use word cloud to understand the word that has the most frequencytext = ' '.join(at['cleaned_hm'].tolist())text = text.lower()wordcloud = WordCloud(background_color="white", height=2700, width=3600).generate(text)plt.figure( figsize=(14,8) )plt.imshow(wordcloud.recolor(colormap=plt.get_cmap('Set2')), interpolation='bilinear')plt.axis("off")
Vectorization :
To make sense from the feature “cleaned_hm” which can be used in our machine learning models, we’ll need to convert each statement to a numeric representation, which we call vectorization.
Next time we will get into the details of various types of vectorization and how to work with them.