Statistical Approach for Linear Regression
By: Kevin Akbari
Tags: Boombikes, Statsmodels, Linear Regression, Machine Learning, Statistics

From our analysis in step 2 above, we can drop the following columns intuitively:

instant : We can drop this column since it cleary does not help in bike demand prediction. dteday : This column is redundant since we already have yr and mnth columns and date is not showing any specific patter for bike counts. casual : We can drop this column since we are going to use cnt as our terget variable and we know that casual+registered=cnt. registered : We can drop this column since we are going to use cnt as our terget variable and we know that casual+registered=cnt.

#dropping columns based on above df=df.drop(['instant','dteday','casual','registered'],axis=1) #check the remaining columns in df

df.head()

We need to convert season , mnth , holiday , weekday , workingday & weathersit categorical columns to dummy variables.

, , , , & categorical columns to dummy variables. Regarding yr column, it is better to convert this column into the no. of years completed since BoomBikes has been operating in this particular area as the popularity of BookBikes is increasing year by year. So we can not use this column for future years in the same marke place and when we use this model into new market place for first 2 years, we assume that popularity of BoomBikes will increase at the same yearly pace.

column, it is better to convert this column into the no. of years completed since BoomBikes has been operating in this particular area as the popularity of BookBikes is increasing year by year. So we can not use this column for future years in the same marke place and when we use this model into new market place for first 2 years, we assume that popularity of BoomBikes will increase at the same yearly pace. We would use all the P columns to represent a categorical column with P different categories, if no. of categories are more than two in categorical feature. Let statistical technique to decide which column to drop in those variables.

#get new column named 'yr_since_start' as just discussed

df['yr_since_start']=df.yr.apply(lambda x:0 if x=='2018' else 1) #sanitiy check for yr_since_start column

df.yr_since_start.value_counts() 1 365

0 365

Name: yr_since_start, dtype: int64 season_dummies=pd.get_dummies(df.season,drop_first=False) #more than 2 cagories

mnth_dummies=pd.get_dummies(df.mnth,drop_first=False) #more than 2 categories

holiday_dummies=pd.get_dummies(df.holiday,drop_first=True) #exactly 2 categories

weekday_dummies=pd.get_dummies(df.weekday,drop_first=False) #more than 2 categories

workingday_dummies=pd.get_dummies(df.workingday,drop_first=True) #exactly 2 categories

weathersit_dummies=pd.get_dummies(df.weathersit,drop_first=False) #more than 2 categories #concat the df with dummy variables

df=pd.concat([df, season_dummies, mnth_dummies, holiday_dummies,

weekday_dummies, workingday_dummies, weathersit_dummies], axis = 1) #drop the categorical columns

df=df.drop(['season','yr','mnth','holiday','weekday','workingday','weathersit'],axis=1) df.info() <class 'pandas.core.frame.DataFrame'>

RangeIndex: 730 entries, 0 to 729

Data columns (total 34 columns):

# Column Non-Null Count Dtype

--- ------ -------------- -----

0 temp 730 non-null float64

1 atemp 730 non-null float64

2 hum 730 non-null float64

3 windspeed 730 non-null float64

4 cnt 730 non-null int64

5 yr_since_start 730 non-null int64

6 fall 730 non-null uint8

7 spring 730 non-null uint8

8 summer 730 non-null uint8

9 winter 730 non-null uint8

10 Apr 730 non-null uint8

11 Aug 730 non-null uint8

12 Dec 730 non-null uint8

13 Feb 730 non-null uint8

14 Jan 730 non-null uint8

15 Jul 730 non-null uint8

16 Jun 730 non-null uint8

17 Mar 730 non-null uint8

18 May 730 non-null uint8

19 Nov 730 non-null uint8

20 Oct 730 non-null uint8

21 Sep 730 non-null uint8

22 Non-Holiday 730 non-null uint8

23 Fri 730 non-null uint8

24 Mon 730 non-null uint8

25 Sat 730 non-null uint8

26 Sun 730 non-null uint8

27 Thu 730 non-null uint8

28 Tue 730 non-null uint8

29 Wed 730 non-null uint8

30 Workingday 730 non-null uint8

31 Bad 730 non-null uint8

32 Good 730 non-null uint8

33 Medium 730 non-null uint8

dtypes: float64(4), int64(2), uint8(28)

memory usage: 54.3 KB

Now we have successfully converted all the categorical columns to dummy variables. As a result, we have all the columns in our df as numeric type.

Step 4: Splitting the Data into Training and Testing Sets

Now let’s split the data into train-test for regression model.

np.random.seed(0) # We specify this so that the train and test data set always have the same rows, respectively

df_train, df_test = train_test_split(df, train_size = 0.7, test_size = 0.3, random_state = 100) #splitting into 70%-30%

Rescaling the Features

We have some columns in 1s, some in 10s and some in 1000s. Hence it is important for us to scale them so that linear regression model converge faster and coefficient of features becomes comparable and non-dominant. Let’s use Min-Max scaling for the same.

Ideally we should be using two seperate scaler object to fit using X_train and y_train since in the new market place we would only have X_new and we would have to predict y_new. At that time also we would have to transform X_new using our old scaler and scaler object does not take one column less due to absence of y_new, if both are fit together. Neverthless, for now let’s just fit both X_train and y_train together.

scaler = MinMaxScaler() # Apply scaler() to all the columns except the 'yes-no' and 'dummy' variables

num_vars = ['temp', 'atemp', 'hum', 'windspeed','cnt','yr_since_start'] df_train[num_vars] = scaler.fit_transform(df_train[num_vars])

Note just above that we have scaled yr_since_start column since we would get this column data as no. of years since BoomBikes start operating.

Dividing the train data into X and Y sets for the model building

y_train = df_train.pop('cnt')

X_train = df_train

Step 5: Building a linear model

Our dataset has more than 15 independent variables and 1 dependent variable. Let’s use RFE first to get the top 15 significate features and then do manual feature filtering with the help of p-values and VIF values of predictor variables.

RFE: It is termed as Recursive feature elimination provided by SciKitLearn package.

# Running RFE with the output number of the variable set to 15

lm = LinearRegression()

rfe = RFE(lm, 15)

rfe = rfe.fit(X_train, y_train) list(zip(X_train.columns,rfe.support_,rfe.ranking_)) [('temp', True, 1),

('atemp', False, 6),

('hum', True, 1),

('windspeed', True, 1),

('yr_since_start', True, 1),

('fall', True, 1),

('spring', False, 12),

('summer', True, 1),

('winter', True, 1),

('Apr', False, 16),

('Aug', False, 8),

('Dec', False, 4),

('Feb', False, 5),

('Jan', False, 2),

('Jul', True, 1),

('Jun', False, 17),

('Mar', False, 15),

('May', False, 7),

('Nov', False, 3),

('Oct', False, 14),

('Sep', True, 1),

('Non-Holiday', True, 1),

('Fri', False, 13),

('Mon', False, 10),

('Sat', True, 1),

('Sun', False, 9),

('Thu', False, 18),

('Tue', False, 11),

('Wed', False, 19),

('Workingday', True, 1),

('Bad', True, 1),

('Good', True, 1),

('Medium', True, 1)] col = X_train.columns[rfe.support_]

print('Columns whom RFE supports are: ', X_train.columns[rfe.support_])

print('Columns whom RFE doesnt support are: ', X_train.columns[~rfe.support_]) Columns whom RFE supports are: Index(['temp', 'hum', 'windspeed', 'yr_since_start', 'fall', 'summer',

'winter', 'Jul', 'Sep', 'Non-Holiday', 'Sat', 'Workingday', 'Bad',

'Good', 'Medium'],

dtype='object')

Columns whom RFE doesnt support are: Index(['atemp', 'spring', 'Apr', 'Aug', 'Dec', 'Feb', 'Jan', 'Jun', 'Mar',

'May', 'Nov', 'Oct', 'Fri', 'Mon', 'Sun', 'Thu', 'Tue', 'Wed'],

dtype='object')

We are left with above 15 columns whom RFE supports. Let’s build model using statsmodel, for the detailed statistical analysis and manual feature elimination.

# function to find VIF of different features def find_vif(X):

vif = pd.DataFrame()

vif['Features'] = X.columns

vif['VIF'] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]

vif['VIF'] = round(vif['VIF'], 2)

vif = vif.sort_values(by = "VIF", ascending = False)

return vif

LR Model 1:

# Creating X_test dataframe with RFE selected 15 variables

X_train_rfe = X_train[col]

X_train_rfe = sm.add_constant(X_train_rfe)

lm_1 = sm.OLS(y_train,X_train_rfe).fit() # Running the linear model print(lm_1.summary()) OLS Regression Results

==============================================================================

Dep. Variable: cnt R-squared: 0.847

Model: OLS Adj. R-squared: 0.843

Method: Least Squares F-statistic: 196.3

Date: Sat, 28 Nov 2020 Prob (F-statistic): 1.13e-191

Time: 13:00:41 Log-Likelihood: 517.87

No. Observations: 510 AIC: -1006.

Df Residuals: 495 BIC: -942.2

Df Model: 14

Covariance Type: nonrobust

==================================================================================

coef std err t P>|t| [0.025 0.975]

----------------------------------------------------------------------------------

const 0.0984 0.032 3.043 0.002 0.035 0.162

temp 0.5278 0.033 15.897 0.000 0.463 0.593

hum -0.1595 0.037 -4.268 0.000 -0.233 -0.086

windspeed -0.1806 0.025 -7.110 0.000 -0.231 -0.131

yr_since_start 0.2297 0.008 28.660 0.000 0.214 0.245

fall 0.0477 0.012 3.820 0.000 0.023 0.072

summer -0.0526 0.015 -3.553 0.000 -0.082 -0.024

winter -0.1080 0.015 -7.367 0.000 -0.137 -0.079

Jul -0.0549 0.018 -3.035 0.003 -0.090 -0.019

Sep 0.0818 0.016 4.956 0.000 0.049 0.114

Non-Holiday 0.0594 0.027 2.206 0.028 0.006 0.112

Sat 0.0532 0.015 3.661 0.000 0.025 0.082

Workingday 0.0432 0.012 3.744 0.000 0.021 0.066

Bad -0.1121 0.023 -4.910 0.000 -0.157 -0.067

Good 0.1342 0.011 12.299 0.000 0.113 0.156

Medium 0.0763 0.013 5.800 0.000 0.050 0.102

==============================================================================

Omnibus: 64.879 Durbin-Watson: 2.065

Prob(Omnibus): 0.000 Jarque-Bera (JB): 158.454

Skew: -0.661 Prob(JB): 3.91e-35

Kurtosis: 5.390 Cond. No. 1.80e+15

============================================================================== Warnings:

[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

[2] The smallest eigenvalue is 6.7e-28. This might indicate that there are

strong multicollinearity problems or that the design matrix is singular.

Looking at the p-values, all of them are less than 5%. Let’s checkout the VIF as well.

#find VIF of different features of X_train_rfe

find_vif(X_train_rfe)

VIF of Bad , Good and Medium are infinite due to obvious reason that any two of the three can 100% explain the third one. Let's start by dropping Medium as it seems the least impactful, based on our univariate analysis and it's coefficient has least magnitude among the three.

LR Model 2:

X_train_lm2=X_train_rfe.drop('Medium',axis=1) X_train_lm2 = sm.add_constant(X_train_lm2) lm_2 = sm.OLS(y_train,X_train_lm2).fit() # Running the linear model print(lm_2.summary()) OLS Regression Results

==============================================================================

Dep. Variable: cnt R-squared: 0.847

Model: OLS Adj. R-squared: 0.843

Method: Least Squares F-statistic: 196.3

Date: Sat, 28 Nov 2020 Prob (F-statistic): 1.13e-191

Time: 13:00:41 Log-Likelihood: 517.87

No. Observations: 510 AIC: -1006.

Df Residuals: 495 BIC: -942.2

Df Model: 14

Covariance Type: nonrobust

==================================================================================

coef std err t P>|t| [0.025 0.975]

----------------------------------------------------------------------------------

const 0.1747 0.043 4.079 0.000 0.091 0.259

temp 0.5278 0.033 15.897 0.000 0.463 0.593

hum -0.1595 0.037 -4.268 0.000 -0.233 -0.086

windspeed -0.1806 0.025 -7.110 0.000 -0.231 -0.131

yr_since_start 0.2297 0.008 28.660 0.000 0.214 0.245

fall 0.0477 0.012 3.820 0.000 0.023 0.072

summer -0.0526 0.015 -3.553 0.000 -0.082 -0.024

winter -0.1080 0.015 -7.367 0.000 -0.137 -0.079

Jul -0.0549 0.018 -3.035 0.003 -0.090 -0.019

Sep 0.0818 0.016 4.956 0.000 0.049 0.114

Non-Holiday 0.0594 0.027 2.206 0.028 0.006 0.112

Sat 0.0532 0.015 3.661 0.000 0.025 0.082

Workingday 0.0432 0.012 3.744 0.000 0.021 0.066

Bad -0.1885 0.025 -7.638 0.000 -0.237 -0.140

Good 0.0578 0.010 5.559 0.000 0.037 0.078

==============================================================================

Omnibus: 64.879 Durbin-Watson: 2.065

Prob(Omnibus): 0.000 Jarque-Bera (JB): 158.454

Skew: -0.661 Prob(JB): 3.91e-35

Kurtosis: 5.390 Cond. No. 28.0

============================================================================== Warnings:

[1] Standard Errors assume that the covariance matrix of the errors is correctly specified. #check VIF values

find_vif(X_train_lm2)

All the p-values of variables seems less than 5% and all the VIF values are less than 5 now. Let’s check the coefficient of diferent predictors and check their strength.

lm_2.params.apply(lambda x:-x if x<0 else x).sort_values(ascending=False) #magnitude of different coefficients temp 0.527819

yr_since_start 0.229738

Bad 0.188485

windspeed 0.180601

const 0.174711

hum 0.159467

winter 0.108047

Sep 0.081773

Non-Holiday 0.059434

Good 0.057847

Jul 0.054881

Sat 0.053203

summer 0.052613

fall 0.047681

Workingday 0.043216

dtype: float64

From above, let’s remove Non-Holiday being higest p-value for our futher model.

LR Model 3:

X_train_lm3=X_train_lm2.drop('Non-Holiday',axis=1) X_train_lm3 = sm.add_constant(X_train_lm3) lm_3 = sm.OLS(y_train,X_train_lm3).fit() # Running the linear model print(lm_3.summary()) OLS Regression Results

==============================================================================

Dep. Variable: cnt R-squared: 0.846

Model: OLS Adj. R-squared: 0.842

Method: Least Squares F-statistic: 209.4

Date: Sat, 28 Nov 2020 Prob (F-statistic): 8.51e-192

Time: 13:00:42 Log-Likelihood: 515.37

No. Observations: 510 AIC: -1003.

Df Residuals: 496 BIC: -943.5

Df Model: 13

Covariance Type: nonrobust

==================================================================================

coef std err t P>|t| [0.025 0.975]

----------------------------------------------------------------------------------

const 0.2252 0.036 6.198 0.000 0.154 0.297

temp 0.5279 0.033 15.838 0.000 0.462 0.593

hum -0.1578 0.037 -4.209 0.000 -0.232 -0.084

windspeed -0.1812 0.025 -7.106 0.000 -0.231 -0.131

yr_since_start 0.2300 0.008 28.580 0.000 0.214 0.246

fall 0.0464 0.013 3.705 0.000 0.022 0.071

summer -0.0530 0.015 -3.562 0.000 -0.082 -0.024

winter -0.1095 0.015 -7.442 0.000 -0.138 -0.081

Jul -0.0537 0.018 -2.962 0.003 -0.089 -0.018

Sep 0.0797 0.017 4.819 0.000 0.047 0.112

Sat 0.0622 0.014 4.438 0.000 0.035 0.090

Workingday 0.0521 0.011 4.790 0.000 0.031 0.073

Bad -0.1880 0.025 -7.587 0.000 -0.237 -0.139

Good 0.0576 0.010 5.512 0.000 0.037 0.078

==============================================================================

Omnibus: 71.749 Durbin-Watson: 2.047

Prob(Omnibus): 0.000 Jarque-Bera (JB): 186.211

Skew: -0.707 Prob(JB): 3.67e-41

Kurtosis: 5.601 Cond. No. 23.1

============================================================================== Warnings:

[1] Standard Errors assume that the covariance matrix of the errors is correctly specified. find_vif(X_train_lm3)

All the p-values of variables seems less than 5% and all the VIF values are less than 5 now. Let’s check the coefficient of diferent predictors and check their strength.

lm_3.params.apply(lambda x:-x if x<0 else x).sort_values(ascending=False) #magnitude of different coefficients temp 0.527923

yr_since_start 0.229966

const 0.225209

Bad 0.187960

windspeed 0.181186

hum 0.157829

winter 0.109466

Sep 0.079694

Sat 0.062161

Good 0.057573

Jul 0.053732

summer 0.052962

Workingday 0.052059

fall 0.046375

dtype: float64

We want to make our model more simpler. Let’s remove Jul being highest p-value.

LR Model 4:

X_train_lm4=X_train_lm3.drop('Jul',axis=1)

X_train_lm4 = sm.add_constant(X_train_lm4) lm_4 = sm.OLS(y_train,X_train_lm4).fit() # Running the linear model print(lm_4.summary()) OLS Regression Results

==============================================================================

Dep. Variable: cnt R-squared: 0.843

Model: OLS Adj. R-squared: 0.839

Method: Least Squares F-statistic: 222.7

Date: Sat, 28 Nov 2020 Prob (F-statistic): 4.23e-191

Time: 13:00:43 Log-Likelihood: 510.90

No. Observations: 510 AIC: -995.8

Df Residuals: 497 BIC: -940.8

Df Model: 12

Covariance Type: nonrobust

==================================================================================

coef std err t P>|t| [0.025 0.975]

----------------------------------------------------------------------------------

const 0.2281 0.037 6.231 0.000 0.156 0.300

temp 0.5127 0.033 15.447 0.000 0.447 0.578

hum -0.1508 0.038 -3.999 0.000 -0.225 -0.077

windspeed -0.1799 0.026 -7.001 0.000 -0.230 -0.129

yr_since_start 0.2307 0.008 28.463 0.000 0.215 0.247

fall 0.0432 0.013 3.438 0.001 0.019 0.068

summer -0.0687 0.014 -4.912 0.000 -0.096 -0.041

winter -0.1135 0.015 -7.688 0.000 -0.142 -0.084

Sep 0.0940 0.016 5.900 0.000 0.063 0.125

Sat 0.0627 0.014 4.442 0.000 0.035 0.090

Workingday 0.0532 0.011 4.863 0.000 0.032 0.075

Bad -0.1912 0.025 -7.666 0.000 -0.240 -0.142

Good 0.0573 0.011 5.445 0.000 0.037 0.078

==============================================================================

Omnibus: 78.295 Durbin-Watson: 2.084

Prob(Omnibus): 0.000 Jarque-Bera (JB): 198.761

Skew: -0.777 Prob(JB): 6.91e-44

Kurtosis: 5.634 Cond. No. 23.1

============================================================================== Warnings:

[1] Standard Errors assume that the covariance matrix of the errors is correctly specified. find_vif(X_train_lm4)

All the p-values of variables are less than 5% and all the VIF values are less than 5. We still have 12 predictors, but we need a more simpler model to avoid any overfitting.

lm_4.params.apply(lambda x:-x if x<0 else x).sort_values(ascending=False) #magnitude of different coefficients temp 0.512675

yr_since_start 0.230700

const 0.228086

Bad 0.191206

windspeed 0.179875

hum 0.150823

winter 0.113489

Sep 0.094023

summer 0.068717

Sat 0.062701

Good 0.057310

Workingday 0.053223

fall 0.043207

dtype: float64

We can remove fall being highest p-value. Besides winter and summer are stronger indicator of shared bike count as per our univariate analysis.

LR Model 5:

X_train_lm5=X_train_lm4.drop('fall',axis=1)

X_train_lm5=sm.add_constant(X_train_lm5) lm_5 = sm.OLS(y_train,X_train_lm5).fit() # Running the linear model

print(lm_5.summary()) OLS Regression Results

==============================================================================

Dep. Variable: cnt R-squared: 0.839

Model: OLS Adj. R-squared: 0.836

Method: Least Squares F-statistic: 236.7

Date: Sat, 28 Nov 2020 Prob (F-statistic): 9.05e-190

Time: 13:00:43 Log-Likelihood: 504.91

No. Observations: 510 AIC: -985.8

Df Residuals: 498 BIC: -935.0

Df Model: 11

Covariance Type: nonrobust

==================================================================================

coef std err t P>|t| [0.025 0.975]

----------------------------------------------------------------------------------

const 0.2601 0.036 7.269 0.000 0.190 0.330

temp 0.4688 0.031 15.137 0.000 0.408 0.530

hum -0.1313 0.038 -3.484 0.001 -0.205 -0.057

windspeed -0.1883 0.026 -7.282 0.000 -0.239 -0.137

yr_since_start 0.2320 0.008 28.341 0.000 0.216 0.248

summer -0.0790 0.014 -5.722 0.000 -0.106 -0.052

winter -0.1430 0.012 -11.780 0.000 -0.167 -0.119

Sep 0.0963 0.016 5.983 0.000 0.065 0.128

Sat 0.0620 0.014 4.343 0.000 0.034 0.090

Workingday 0.0512 0.011 4.638 0.000 0.030 0.073

Bad -0.1851 0.025 -7.362 0.000 -0.235 -0.136

Good 0.0615 0.011 5.815 0.000 0.041 0.082

==============================================================================

Omnibus: 67.176 Durbin-Watson: 2.097

Prob(Omnibus): 0.000 Jarque-Bera (JB): 151.402

Skew: -0.710 Prob(JB): 1.33e-33

Kurtosis: 5.260 Cond. No. 22.8

============================================================================== Warnings:

[1] Standard Errors assume that the covariance matrix of the errors is correctly specified. find_vif(X_train_lm5)

All the p-values of variables are less than 5% and all the VIF values are less than 5. We have 11 predictors now, but should again try and drop feature to avoid any overfitting and get the simplest possible model.

lm_5.params.apply(lambda x:-x if x<0 else x).sort_values(ascending=False) #magnitude of different coefficients temp 0.468845

const 0.260113

yr_since_start 0.231953

windspeed 0.188260

Bad 0.185128

winter 0.142998

hum 0.131287

Sep 0.096289

summer 0.079033

Sat 0.061961

Good 0.061464

Workingday 0.051245

dtype: float64

Let’s remove hum column due to its low magnitude of t-value.

LR Model 6:

X_train_lm6=X_train_lm5.drop('hum',axis=1)

X_train_lm6=sm.add_constant(X_train_lm6) lm_6 = sm.OLS(y_train,X_train_lm6).fit() # Running the linear model

print(lm_6.summary()) OLS Regression Results

==============================================================================

Dep. Variable: cnt R-squared: 0.836

Model: OLS Adj. R-squared: 0.832

Method: Least Squares F-statistic: 253.5

Date: Sat, 28 Nov 2020 Prob (F-statistic): 2.18e-188

Time: 13:00:44 Log-Likelihood: 498.77

No. Observations: 510 AIC: -975.5

Df Residuals: 499 BIC: -929.0

Df Model: 10

Covariance Type: nonrobust

==================================================================================

coef std err t P>|t| [0.025 0.975]

----------------------------------------------------------------------------------

const 0.1592 0.021 7.499 0.000 0.117 0.201

temp 0.4466 0.031 14.574 0.000 0.386 0.507

windspeed -0.1604 0.025 -6.453 0.000 -0.209 -0.112

yr_since_start 0.2350 0.008 28.558 0.000 0.219 0.251

summer -0.0710 0.014 -5.158 0.000 -0.098 -0.044

winter -0.1412 0.012 -11.515 0.000 -0.165 -0.117

Sep 0.0898 0.016 5.558 0.000 0.058 0.122

Sat 0.0658 0.014 4.576 0.000 0.038 0.094

Workingday 0.0538 0.011 4.822 0.000 0.032 0.076

Bad -0.2015 0.025 -8.064 0.000 -0.251 -0.152

Good 0.0826 0.009 9.427 0.000 0.065 0.100

==============================================================================

Omnibus: 67.335 Durbin-Watson: 2.090

Prob(Omnibus): 0.000 Jarque-Bera (JB): 152.730

Skew: -0.709 Prob(JB): 6.84e-34

Kurtosis: 5.275 Cond. No. 15.1

============================================================================== Warnings:

[1] Standard Errors assume that the covariance matrix of the errors is correctly specified. find_vif(X_train_lm6)

lm_6.params.apply(lambda x:-x if x<0 else x).sort_values(ascending=False) #magnitude of different coefficients temp 0.446558

yr_since_start 0.234983

Bad 0.201450

windspeed 0.160400

const 0.159155

winter 0.141200

Sep 0.089836

Good 0.082562

summer 0.071029

Sat 0.065809

Workingday 0.053751

dtype: float64

All the p-values of variables are less than 5% and all the VIF values are less than 5. We have 10 predictors now, which seem reasonably simple. We have reasonably good values of R-squared & Adj. R-squared compared to previous models with high no. of features. Further, we have Prob (F-statistic) value statistically significant as compared to restricted model. So let’s go ahead with the analysis of this last built model.

Step 6: Residual Analysis of the train data

So, now to check if the error terms are also normally distributed which is one of the assumption of regression model, let us plot the histogram of the error terms on training model and see what it looks like.

y_train_pred_lm6 = lm_6.predict(X_train_lm6) #predicting y_train using final model # Plot the histogram of the error terms sns.distplot((y_train - y_train_pred_lm6), bins = 40)

plt.title('Error Terms') # Plot heading

plt.xlabel('Errors') # X-label

plt.ylabel('Count') # Y-label

plt.show()

# plot qqplot with statsmodels

sm.qqplot((y_train - y_train_pred_lm6),fit=True,line='45')

plt.show()

From the distribution of error plot, we note that error terms are approximately normally distributed with mean zero. Q-Q plot of error terms almost passes through straight line. This also indicates that error terms are normally distributed.

sns.scatterplot(x=y_train, y=(y_train - y_train_pred_lm6))

plt.title('Error Terms') # Plot heading

plt.xlabel('Scaled actual shared bike counts') # X-label

plt.ylabel('Residue') # Y-label

plt.show()

From the above we do not find any pattern for the different residual values of count on training set. Hence error terms are deemed independent .

. Here variance of error term seem almost constant.

Hence we can hereby declare that all the assumptions of Linear Regression is satisfied by our model no. 6 (lm_6). So let’s go ahead with prediction part.

Step 7: Making Predictions Using the Final Model

Now that we have fitted the model and checked the normality of error terms, it’s time to go ahead and make predictions using the final model.

Applying the scaling on the test sets

df_test[num_vars] = scaler.transform(df_test[num_vars]) #using the already fit scaler object

Dividing into X_test and y_test

y_test = df_test.pop('cnt')

X_test = df_test # Adding constant to test dataframe

X_test_lm6 = sm.add_constant(X_test) #check the variables in X_train_lm6

X_train_lm6.columns Index(['const', 'temp', 'windspeed', 'yr_since_start', 'summer', 'winter',

'Sep', 'Sat', 'Workingday', 'Bad', 'Good'],

dtype='object') # Creating X_test_lm6 dataframe by keeping variables as per X_train_lm6 X_test_lm6 = X_test_lm6[['const', 'temp', 'windspeed', 'yr_since_start', 'summer', 'winter',

'Sep', 'Sat', 'Workingday', 'Bad', 'Good']] # Making predictions using the final model y_pred_test_lm6 = lm_6.predict(X_test_lm6)

Step 8: Model Evaluation

Let’s now plot the graph for actual versus predicted values on test set.

# Plotting y_test and y_pred to understand the spread fig = plt.figure()

plt.scatter(y_test, y_pred_test_lm6)

plt.plot(y_test,y_test,'r') #plotting y_test-y_test line

fig.suptitle('y_test vs y_pred') # Plot heading

plt.xlabel('y_test') # X-label

plt.ylabel('y_pred') # Y-label

plt.show()

n0=X_train_lm6.shape[0] #no. of samples

p0=X_train_lm6.shape[1]-1 #no. of independent predictors r2_train=r2_score(y_train, y_train_pred_lm6)

print('R square value of train data on final model is: ',r2_train)

print('R square adjusted value of train data on final model is: ',1-((1-r2_train)*(n0-1)/(n0-p0-1))) R square value of train data on final model is: 0.8355154261193748

R square adjusted value of train data on final model is: 0.8322191420736709 n=X_test_lm6.shape[0] #no. of samples

p=X_test_lm6.shape[1]-1 #no. of independent predictors r2_test=r2_score(y_test, y_pred_test_lm6)

print('R square value of test data on final model is: ',r2_test)

print('R square adjusted value of test data on final model is: ',1-((1-r2_test)*(n-1)/(n-p-1))) R square value of test data on final model is: 0.8038003773854856

R square adjusted value of test data on final model is: 0.7943677032213263

From the above, we do not see much difference between R square values on train and test data sets.

#check the different parameters of features

lm_6.params.sort_values() Bad -0.201450

windspeed -0.160400

winter -0.141200

summer -0.071029

Workingday 0.053751

Sat 0.065809

Good 0.082562

Sep 0.089836

const 0.159155

yr_since_start 0.234983

temp 0.446558

dtype: float64

Predictors to watch out for:

Now since we have scaled all the variables within the range of 0 and 1 before training the model, we can directly compare the magnitudes of different coefficient of variables and based on the same, we can figure out the most significant variables i.e. higher the magnitude of coefficient, more important is the corresponding variable.

Temperature: Temperature is clearly the most impactful variable. The sign is positive hence higher the temperature, more people tend to use bikes. yr_since_start: This is second most significant variable. It has positive impact on bike sharing business. The same indicates BoomBikes’ business is hiking on yearly basis. Bad: This is third most impactful variable. It also has negative impact on bike sharing. Hence BoomBikes may reduce bike availability on days with Heavy Rain + Ice Pallets + Thunderstorm + Mist, Snow + Fog.

Conclusion:

Now that was a long run! If you are still with me till this point then congratulations, you have thoroughly explored statistical side of Linear Regression model. In fact the same technique can be used for different linear models as well.

Till next time!