EDA from Scratch: Hands on Guide
By: Sanket Kangle
Tags: Artificial Intelligence, Machine Learning, Business Intelligence, Statistics, Exploratory Data Analysis

As seen from above graph, we do not have any missing values. alternate way to check missing values is as follows

df.isna().value_counts()

Image from author

False under each column name signifies there is no missing value in that column.

EDA

The feature count gives total number of rental at particular time. Let’s see how count is varied w.r.t. other few features.

Following command plots a scatter plot with season on x-axis and count on y-axis

df.plot.scatter(x='season', y='count')

Image from author

From this plot we understand that for Fall season, the people used rental bike the most and in spring the least.

Following command plots a scatter plot with holiday on x-axis and count on y-axis

df.plot.scatter(x="holiday", y="count")

Image from author

It is clear from this plot that on holiday, people use less renal bikes. The cause for this might be that a lot of them using bikes to go to office/work and on holiday they are not using it.

Following command plots a scatter plot with workingday on x-axis and count on y-axis

df.plot.scatter(x="workingday", y="count")

Image from author

What we suspected in last plot is confirmed in this one. on workingday people used more rental bikes.

Following command plots a scatter plot with weather on x-axis and count on y-axis

df.plot.scatter(x="weather", y='count')

Image from author

Recall weather column data fields:

weather

1: Clear, Few clouds, Partly cloudy, Partly cloudy

2: Mist + Cloudy, Mist + Broken clouds, Mist + Few clouds, Mist

3: Light Snow, Light Rain + Thunderstorm + Scattered clouds, Light Rain + Scattered clouds

4: Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog

As weather is getting clear to stormy, number of bike users decreased. Also the frequency for ‘4’ is low(only one data point) as such weather is also rare.

Following command plots a scatter plot with temperature on x-axis and count on y-axis

df.plot.scatter(x="temp", y="count")

Image from author

For extreme low temperature there are very few who used bikes and as temperature increases, the count also increases.

Following command plots a scatter plot with humidity on x-axis and count on y-axis

df.plot.scatter(x="humidity", y="count")

Image from author

for extremely low and high humidity less bikes are in use which makes sense as they are corresponding to adverse weather conditions.

Following command plots a scatter plot with windspeed on x-axis and count on y-axis

df.plot.scatter(x="windspeed", y="count")

As wind speed increased, the count decreases gradually which is anticipated as again with high wind speed associated with bad weather.

In conclusion till now, we understood two important points:

1. Many people rent bike for traveling to office and home

2. Bad weather conditions have adverse effect on bike rentals

Now, let’s focus on how the count varies during the day and how other features affect it. We have to first get hour of the day from date time column, and then work with it.

Below is a function that will return a bar plot of count based on hour for a given year

def plot_by_hour(data, year=None, agg='sum'):

dd = data #flag 1

if year : #flag 2

dd = dd[dd.datetime.dt.year == year]

dd.loc[:, ('hour')] = dd.datetime.dt.hour #flag 3

by_hour = dd.groupby(['hour', 'workingday'])['count'].agg(agg).unstack() #flag 4

return by_hour.plot(kind='bar', title="year = {0}".format(year),figsize=(15,5), width=0.9) #flag 5

We created a function called plot_by_hour which takes input one dataframe(data) and year.

#flag 1: we assign complete dataframe(both years 2011 & 2012) to dd

#flag 2: if year is provided as an argument to the function, then we slice the dataframe dd only for that year

#flag 3:creating a column called hour in dataframe dd and populating it with hour from datetime column

in dataframe dd and populating it with hour from column #flag 4: From dataframe dd, we are grouping the data by hour and workingday columns and then getting its count column and assigning it to a variable called by_hour

and columns and then getting its column and assigning it to a variable called by_hour #flag 5: Returning the bar plot

Now, let’s plot the graph for individual years and combined

plot_by_hour(df, year=2011)

plot_by_hour(df, year=2012)

plot_by_hour(df)

Image from author

Image from author

Image from author

From above plots, it is clear that working professionals do use rental bikes for commuting to office and home, as there are consistence spikes at rush hour(office starting and ending hours) on working day which are not there on holidays. Also, in night time, the use of rental bike is minimal

The use of rental bikes had increases in 2012 than in 2011, let’s focus month-wise and hour-wise increase of rental bikes in 2012 than in 2011. Following is function to do the same

def plot_by_year(data,col, title):

dd = data.copy() #flag 1

dd['year'] = data.datetime.dt.year #flag 2

dd['month'] = data.datetime.dt.month #flag 3

dd['hour'] = data.datetime.dt.hour #flag 4

by_year = dd.groupby([col, 'year'])["count"].agg('sum').unstack() #flag 5

return by_year.plot(kind='bar', figsize=(15,5), width=0.9, title=title) #flag 6

We created a function plot_by_year which takes input dataframe(data), name of column(col) and title for plot(title)

#flag 1 : creating copy of dataframe so as to not disturb original dataset

#flat 2 : creating column named year in dataframe dd and populating with year extracted from datetime column of data

in dataframe dd and populating with year extracted from column of data #flag 3 :creating column named month in dataframe dd and populating with month extracted from datetime column of data

in dataframe dd and populating with month extracted from column of data #flag 4 : creating column named hour in dataframe dd and populating with hour extracted from datetime column of data

in dataframe dd and populating with hour extracted from column of data #flag 5 : grouping dataframe dd on columns col(given as argument to function), year and aggregating their sum and storing it in variable by_year

and aggregating their sum and storing it in variable by_year #flag 6 : returning the plot

plot_by_year(df,'month', "Rent bikes per month in year 2011 and 2012")

Image from author

From above graph, it is clear that, in every month, 2012 has more bikes rented than 2011. Let’s see, does by hour also 2012 is leading or not

plot_by_year(df, 'hour', 'Rent bikes per hour in year 2011 and 2012')

Image from author

Yes! hourly also 2012 leads. This means in 2012, the rental got more popular i.e. marketing strategy in 2011 was good that is why count increased in 2012.

Let’s count the percentage growth

count_2011 = df[df.year==2011]['count'].agg('sum')

count_2012 = df[df.year==2012]['count'].agg('sum')

percentage_growth = (count_2012-count_2011)*100/count_2011

Hence the percentage growth come around 66.69%

Conclusion: (in addition to last conclusions)

3. In office starting and ending hours, the demand for rental bikes is more as many working professional use it as a way of commute

4. In night time, the demand is very low

5. Marketing strategy used in 2011 must be efficient, as count has increased in 2012 by margin of 66.69%

You can get complete code for this exercise here.