Black Friday: How much will a customer spend?
By: Super Albert
Tags: Data Science, Education, Artificial Intelligence, Programming, Projects

1. Exploratory Data Analysis (EDA)

We’ve made our first assumptions on the data and now we are ready to perform some basic data exploration and come up with some inference. Hence, the goal for this section is to take a glimpse on the data as well as any irregularities so that we can correct on the next section, Data Pre-Processing.

1.1. Univariate Analysis

To get an idea of the distribution of numerical variables, histograms are an excellent starting point. Let’s begin by generating one for Purchase , our target variable.

1.1.1. Distribution of the target variable: Purchase

plt.style.use(‘fivethirtyeight’)

plt.figure(figsize=(12,7))

sns.distplot(train.Purchase, bins = 25)

plt.xlabel(“Amount spent in Purchase”)

plt.ylabel(“Number of Buyers”)

plt.title(“Purchase amount Distribution”)

It seems like our target variable has an almost Gaussian distribution.

print (“Skew is:”, train.Purchase.skew())

print(“Kurtosis: %f” % train.Purchase.kurt())

1.1.2. Numerical Predictors

Now that we’ve analysed our target variable, let’s consider our predictors. Let’s start by seeing which of our features are numeric.

numeric_features = train.select_dtypes(include=[np.number])

numeric_features.dtypes

1.1.2.1. Distribution of the variable Occupation

As seen in the beginning, Occupation has at least 20 different values. Since we do not known to each occupation each number corresponds, is difficult to make any analysis. Furthermore, it seems we have no alternative but to use since there is no way to reduce this number as we did on Project Bigmart with Item_Type.

sns.countplot(train.Occupation)

1.1.2.2. Distribution of the variable Marital_Status

As expected there are more single people buying products on Black Friday than married people, but do they spend more?

1.1.2.3. Distribution of the variable Product_Category_1

From the distribution for products from category one, it is clear that three products stand out, number 1, 5 and 8. Unfortunately, we do not know which product each number represents.

1.1.2.4. Distribution of the variable Product_Category_2

1.1.2.5. Distribution of the variable Product_Category_3

1.1.2.6. Correlation between Numerical Predictors and Target variable

corr = numeric_features.corr() print (corr[‘Purchase’].sort_values(ascending=False)[:10], ‘

’)

print (corr[‘Purchase’].sort_values(ascending=False)[-10:])

There does not seem to be any predictor that would have a high impact on Purchase , since the highest correlation is give by Occupation with 0.0208. On the other hand, Product_Category_1 has a negative correlation with our target with the value -0.3437 which is somehow odd.

#correlation matrix

f, ax = plt.subplots(figsize=(20, 9))

sns.heatmap(corr, vmax=.8,annot_kws={'size': 20}, annot=True);

There seems to be no multicollinearity with our predictors which is a good thing, although there is some correlation among the product categories. Are category 2 and 3 necessary? Can we dispose them?

1.1.3. Categorical Predictors

For categorical variables, bar charts and frequency counts are the natural counterparts to histograms. Now is time to look at the variables that contain some insight on the assumptions previously made.

1.1.3.1. Distribution of the variable Gender

Most of the buyers are males, but who spends more on each purchase: man or woman?

sns.countplot(train.Gender)

1.3.2. Distribution of the variable Age

sns.countplot(train.Age)

As expected, most purchases are made by people between 18 to 45 years old.

1.1.3.3. Distribution of the variable City_Category

Supposing ‘A’ represents the biggest city whereas ‘C’ the smallest, it curious to see that the medium size cities ‘B’ had higher sales than the others. But do they also spent more?

sns.countplot(train.City_Category)

1.1.3.3. Distribution of the variable Stay_In_Current_City_Years

The tendency looks like the longest someone is living in that city the less prone they are to buy new things. Hence, if someone is new in town and needs a great number of new things for their house that they’ll take advantage of the low prices in Black Friday to purchase all the things needed.

sns.countplot(train.Stay_In_Current_City_Years)

1.2. Bivariate Analysis

Firstly we individually analysed some of the existent features, now it is time to understand the relationship between our target variable and predictors as well as the relationship among predictors.

1.2.1. Numerical Variables

1.2.1.1. Occupation and Purchase analysis

Although there are some occupations which have higher representations, it seems that the amount each user spends on average is more or less the same for all occupations. Of course, in the end, occupations with the highest representations will have the highest amounts of purchases.

Occupation_pivot = \

train.pivot_table(index='Occupation', values="Purchase", aggfunc=np.mean) Occupation_pivot.plot(kind='bar', color='blue',figsize=(12,7))

plt.xlabel("Occupation")

plt.ylabel("Purchase")

plt.title("Occupation and Purchase Analysis")

plt.xticks(rotation=0)

plt.show()

1.2.1.2. Marital_Status and Purchase analysis

Now, this is a interesting aspect. We had more single customers than married. However, on average an individual customer tends to spend the same amount independently if his/her is married or not. Again, if you had all the purchases the single group, since has a higher representation, will have the highest purchase values.

1.2.1.3. Product_category_1 and Purchase analysis

If you see the value spent on average for Product_Category_1 you see that although there were more products bought for categories 1,5,8 the average amount spent for those three is not the highest. It is interesting to see other categories appearing with high purchase values despite having low impact on sales number.

For examples, if instead of the average spent we look at the amount spent on purchase, as illustrated in the chart below, that distribution that we saw for this predictor previously appears here. For example, those three products have the highest sum of sales since their were three most sold products.

We can se the same behaviour for the other two categories.

1.2.2. Categorical Variables

1.2.2.1. Gender and Purchase analysis

On average the male gender spends more money on purchase contrary to female, and it is possible to also observe this trend by adding the total value of purchase. This last conclusion is more reasonable since the percentage of male buyers is higher than female buyers.

1.2.2.2. Age and Purchase analysis

Total amount spent in purchase is in accordance with the number of purchases made, distributed by age.

Nevertheless, regarding the average spend by group age we can see the amount spent is almost the same for everyone. Curiously, on average customer with more than 50 years old are the ones who spent the most.

1.2.2.3. City_Category and Purchase analysis

We saw previously that city type ‘B’ had the highest number of purchases registered. However, the city whose buyers spend the most is city type ‘C’.

1.2.2.4. Stay_In_Current_City_Years and Purchase analysis

Again, we see the same pattern seen before which show that on average people tend to spend the same amount on purchases regardeless of their group. People who are new in city are responsible for the higher number of purchase, however looking at it individually they tend to spend the same amount independently of how many years the have lived in their current city.

2. Data Pre-Processing

During our EDA we were able to take some conclusions regarding our first assumptions and the available data:

Age : should be treated as numerical. It presents age groups.

City_Category : We can convert this to numerical as well, with dummy variables. Should take a look at the frequency of the values.

Occupation : It seems like it has at least 16 different values, should see frequency and try to decrease this value.

Gender : There are possibly two gender, we can make this binary.

Product_ID : Should see if the string “P” means something and if there are other values.

Stay_In_Current_City_Years : We should deal with the ‘+’ symbol.

Product_Category_2 and Product_Category_3 : Have NaN values.

Usually, datasets for every challenge such as those presented in Analytics Vidhya or Kaggle come seperated as a train.csv and a test.csv . It is generally a good idea to combine both sets into one, in order to perform data cleaning and feature engineering and later divide them again. With this step we do not have to go through the trouble of repeting twice the same code, for both datasets. Let’ s combine them into a dataframe data with a source column specifying where each observation belongs.

# Join Train and Test Dataset

train[‘source’]=’train’

test[‘source’]=’test’ data = pd.concat([train,test], ignore_index = True, sort = False) print(train.shape, test.shape, data.shape)



2. 1. Looking for missing values

#Check the percentage of null values per variable

data.isnull().sum()/data.shape[0]*100

The only predictors having missing value are Product_Category_1 and Product_Category_2 . We can either try to impute the missing values or drop these predictors. We can text both approaches to see which returns the best results.

2.1.1. Numerical Values

2.1.1.1. Imputing the value Zero

data["Product_Category_2"]= \

data["Product_Category_2"].fillna(-2.0).astype("float") data.Product_Category_2.value_counts().sort_index()

data[“Product_Category_3”]= \

data[“Product_Category_3”].fillna(-2.0).astype(“float”)

data.Product_Category_3.value_counts().sort_index()

2.1.1.2. Removing Product_Category_1 group 19 and 20 from Train

#Get index of all columns with product_category_1 equal 19 or 20 from train

condition = data.index[(data.Product_Category_1.isin([19,20])) & (data.source == “train”)]

data = data.drop(condition)

2.1.2. Categorical Values

#Apply function len(unique()) to every data variable

data.apply(lambda x: len(x.unique()))

2.2.2.1. Frequency Analysis