Exploratory Data Analysis(EDA). As medium has some redering issues with…
By: Gaurav Sharma
Tags: Seaborn, Data Visualization, Data Science, Data Analysis, Exploratory Data Analysis

Analysis on Categorical Attributes

The analysis includes both descriptive stats and EDA.

I will first slice out the categorical columns from original data-frame and then do analysis on it, keeping the original data untouched. At the end I will incorporate needed changes in the original data-frame.

As origin and name consists of text data so it needs some pre-processing. We will remove all extra spaces from each string, otherwise the same string with different spacing will be treated as different categories which should not be the case.

I will create an artificial categorical attribute named mpg_level which categorizes mpg into low , medium and high . This is done for two reasons, first it will help a lot in EDA i.e., I can bifurcate plots on the basis of mpg and secondly this is easy to understand as compared to numbers.

I am dividing mpg into three regions as,

[min, 17) -> low

[17, 29) -> medium

[29, max) -> high

Also the choice of the range is analytical and can be anything till it seems to be reasonable.

Note: This is feature-engineering and mostly done in predictive modelling but it makes sense to introduce it here.

Let’s inspect the unique characters in origin , cylinders and model_year . I am leaving name because it is almost unique for each entry in this case, hence nothing interesting to inspect it.

Although descriptive stats for categorical attributes are not much informatic but still it’s worth looking once. Also pandas describe function is only for numeric data and in df_cat cylinders and model_year are the only numeric type.

df_cat.describe()

It seems that most of the values in cylinders are 4 and (min, max) is (3, 8).

Analysis of Distribution

Now we analyze the distribution for each categorical attribute and make some insights from the plots.

In case of categorical variables an ideal (or atleast loved) distribution is uniform or uniform-like , below is an uniform distribution.

Let’s plot the distribution for different categorical attributes in our data.

Let’s calculate the proportion of dominant classes in each category.

Insights

origin is highly imbalanced, usa alone consists of 62.5% of data whereas japan & europe are having similar proportion. We will see this dominance in future analysis. We will try to find the reason for this in our further analysis.

is highly imbalanced, alone consists of 62.5% of data whereas & are having similar proportion. We will see this dominance in future analysis. cylinders is highly imbalanced, 4 alone consists of 50.77% of data. Whereas 8 & 6 are nearly in same proportion but 3 & 5 collectively accounts for only 7 entries i.e., 1.8% of entire data. We will see this huge proportional imbalance in cylinders in future analysis.

is highly imbalanced, alone consists of 50.77% of data. Whereas & are nearly in same proportion but & collectively accounts for only 7 entries i.e., 1.8% of entire data. We will see this huge proportional imbalance in in future analysis. mpg_level is highly imbalanced, medium alone consists of 52.3% of data while low & high are in the same proportion. This dominance is due the fact of our thresholding while manufacturing this feature because the medium range is broader hence it consists of more data points. It won't be there in original mpg feature as it is continuous.

is highly imbalanced, alone consists of 52.3% of data while & are in the same proportion. This dominance is due the fact of our thresholding while manufacturing this feature because the medium range is broader hence it consists of more data points. It won't be there in original feature as it is continuous. model_year is considerably balanced which is good.

Now we analyze car's name .

Firstly, even though name is categorical but it has a lot categories and this even makes sense because product names generally varies a lot in any domain. So it’s not fruitful to do analysis on car names as these are names just like product id and seems to hold no important insights.

But one thing to be noticed here is that each car name starts with a company name, so maybe the case that there are very few companies in the data-set and it will be fruitful to extract them as separate feature and do analysis on that. So let’s do it.

I will create a new attribute named as car_company by extracting the first word from all names. I will also remove the car company from each car name because it is not needed now, and also rename column name to car_name .

Now, check for total unique values in car_company .

Great this is what we wished and indeed we get. Our idea that there will be few car companies involved in this data is indeed correct. Because the number of categories are less we can now do analysis on that. So we took a step in right direction just by our in-tuition.

Now, we analyze the distribution of car_company .

Insights

We found that car_name has a lot categories, close to total data points. So it's not fruitful to do analysis on that as it is unique for most of the points and also in most cases names are safe to be avoided as they doesn't have correlations with other.

has a lot categories, close to total data points. So it's not fruitful to do analysis on that as it is unique for most of the points and also in most cases names are safe to be avoided as they doesn't have correlations with other. We then create an artificial attribute named car_company by extracting company names from car names. We find that there are much few car companies as compared to car names (around 8 times less).

by extracting company names from car names. We find that there are much few car companies as compared to car names (around 8 times less). We then found that the distribution of car_company is not uniform and most of the proportion is covered by top 15 car companies. Whereas ford and chevrolet alone comprises of around 23% (almost a quarter).

Conclusion

Every categorical attribute except model_year is highly imbalanced and far from uniform distribution . In all cases most of the data is comprised of top few categories.

is highly imbalanced and far from . In all cases most of the data is comprised of top few categories. Although model_year is not perfectly uniform but we can think it as uniform-like distribution . This is a digestible assumption for two reasons, first we can clearly see in plot that indeed the distribution is uniform-like and also this is not the entire population but a sample of it so may be in large run it will converge to uniform which may be the true population distribution (Law of Large Number).

Now we will analyze how different features behaves on changing other features.

Insights

We can clearly see the impact of imbalanced categories in our bifurcated plots.

cylinders bifurcated by origin

Japan is the only origin with vehicles having 3 cylinders.

Europe is the only origin with vehicles having 5 cylinders.

USA is the only origin with vehicles having 8 cylinders.

All origins has 4 cylinder vehicles and in almost equal proportion, also because 4 is dominating in cylinders.

All origins has 6 cylinder vehicles but dominated by USA due the fact that it is dominating in origin.

mpg_level bifurcated by origin

Japan doesn’t have any vehicle with low mpg_level whereas europe has negligible vehicles with low mpg_level and almost all vehicles that has low mpg_level are from usa.

Japan has the most vehicles with high mpg_level.

USA has the most vehicles with medium mpg_level (again due to the fact that most vehicles belongs to USA).

mpg_level bifurcated by cylinders

Vehicles with low mpg_level has either 6 or 8 cylinders and most of them has 8 cylinders.

Almost all vehicles with high mpg_level has 4 cylinders and with very few (less than 5) has 5–6 cylinders.

Most vehicles with medium mpg_level has 4,6 & 8 cylinders. This is due to the fact that most of the vehicles has these number of cylinders as we inspected in cylinder distribution earlier.

Let’s analyze mpg_level with cylinders bifurcated by origin.

Insights

Japan doesn’t have any cylinder with low mpg_level and most of it’s vehicles have high mpg_level with mostly 4 cylinders(this we already inspected from earlier plots as well, nothing new).

Almost every vehicle in europe has 4 cylinders and most of them are medium or high.

USA has few vehicles with high mpg level as compared to other knowing the fact that most of the vehicles belongs to USA.

Note: Although these insights can be detected from earlier plots but it was bit hard and bifurcation helped us and we extracted even more meaning from data.

Conclusion

Japan is leading in case of mpg level with most of it’s vehicles having high mpg_level. It has more than twice vehicles with high mpg compared to other origins.

It seems like as number of cylinders increases mpg decreases in general.

Let’s analyze model_year,

Insights

As year progresses manufacturing of low mpg car decreases and eventually stopped after 79.

As year progresses manufacturing of high mpg car increases and infact after 79 their manufacturing was 1.5–2 times as compared to vehicles with medium mpg.

Throughout the years there is no significant change in manufacturing of vehicles with medium mpg and infact from 74–79 their manufacturing was more than the sum of remaining two.

From the scatter plot we can clearly see an upward linear trend i.e., as year progress mpg increases.

Insights

As the year progresses vehicles with more cylinders (8 & 6) decreases significantly.

As the year progresses vehicles with less cylinders increases.

One important thing to be noticed that throughout the years vehicles with 4 cylinders have significant proportion and infact in the 80’s most of the vehicles has 4 cylinders.

These results make sense because as year progresses technology advances vehicles with low mpg and more cylinders looses focus and vehicles with high mpg and less cylinders are the new stars.

Insights

In the starting year manufacturing in USA is dominated completely.

As the year progresses japan and europe started manufacturing more vehicles. Infact, in the year 80 both japan and europe manufactured more than USA. This may be due to something because throughout the years USA dominates and suddenly there is a considerable decrease in their manufacturing. This may have something to do in USA in year 80.

Initially europe manufactures more vehicle then japan but then japan exceeds it after 76.

Let’s analyze car_company,

As car_company contains a lot categories and most of them has very low proportion so we will analyze only the top 15 car companies.

We can see that top 15 car companies alone manufactures 83% of vehicles.

Insights

Top manufacturing companies have all mpg level vehicles but companies with less vehicles focuses more on high or medium mpg vehicle.

All top manufacturing companies are from usa and that is the reason why usa has most of the vehicles in data-set (this is one of our key findings). We now answered this question asked earlier by us.

and that is the reason why usa has most of the vehicles in data-set (this is one of our key findings). We now answered this question asked earlier by us. All top manufacturing companies focuses on vehicles with cylinders 4, 6 & 8 equally, but the companies with less manufacturing generally uses less cylinders in their vehicles.

We are done with the analysis of categorical attributes and found lots of interesting things and answered many unknown questions. Now we will incorporate the required changes of df_cat into df.

Every attribute except car_name is of interest and participated in our analysis. So we will not add car_name to our data-frame as it is of no interest. This is feature reduction and is an integral part of feature engineering .

Save these changes to a new file.