Exploratory Data Analysis of Kaggle datasets. | by Gokul S Kumar | Towards Data Science

Exploratory Data Analysis or EDA refers to the process of knowing more about the data in hand and preparing it for modeling. To be frank, EDA and feature engineering is an art where you get to play around with the data and try to get insights from it before the process of prediction. Most people understand machine learning to be only about models and algorithms. But for these models to perform well with high accuracy, EDA is essential. EDA provides a lot of crucial information that is very easy to miss, information that helps that analysis in the long run.
There are no hard-bound rules on how to perform EDA. Everyone dealing with the data has to find their own way of performing EDA and understand the data accordingly. In this article, I will be explaining my step by step approach of doing EDA on the Home price dataset from Kaggle. The basic steps involved would be:
Importing the dataset.Getting basic insights.Analyzing the different features and dividing them into numerical and categorical.Dealing with missing values.Dealing with correlated features.
Importing the dataset.
Getting basic insights.
Analyzing the different features and dividing them into numerical and categorical.
Dealing with missing values.
Dealing with correlated features.
In the end, I will be implementing a modern approach to EDA using a library named ‘Sweetviz’, which I think, might help us save a lot of time and effort.
I won't be displaying the code in the article instead, I have given the link to the entire code here. Please go through it to keep up with the explanation.
As mentioned above, I will be using the home prices dataset from Kaggle, the link to which is given here.
We will be loading the train and the test dataset to a Pandas dataframe separately.
The dataframe containing the train and test data would like. We will try to get some basic insights about the entire data.
The detailed description of the features is given along with the dataset. Brief info is obtained.
<class 'pandas.core.frame.DataFrame'>Int64Index: 1460 entries, 1 to 1460Data columns (total 80 columns): #   Column         Non-Null Count  Dtype  ---  ------         --------------  -----   0   MSSubClass     1460 non-null   int64   1   MSZoning       1460 non-null   object  2   LotFrontage    1201 non-null   float64 3   LotArea        1460 non-null   int64   4   Street         1460 non-null   object  5   Alley          91 non-null     object  6   LotShape       1460 non-null   object  7   LandContour    1460 non-null   object  8   Utilities      1460 non-null   object  9   LotConfig      1460 non-null   object  10  LandSlope      1460 non-null   object  11  Neighborhood   1460 non-null   object  12  Condition1     1460 non-null   object  13  Condition2     1460 non-null   object  14  BldgType       1460 non-null   object  15  HouseStyle     1460 non-null   object  16  OverallQual    1460 non-null   int64   17  OverallCond    1460 non-null   int64   18  YearBuilt      1460 non-null   int64   19  YearRemodAdd   1460 non-null   int64   20  RoofStyle      1460 non-null   object  21  RoofMatl       1460 non-null   object  22  Exterior1st    1460 non-null   object  23  Exterior2nd    1460 non-null   object  24  MasVnrType     1452 non-null   object  25  MasVnrArea     1452 non-null   float64 26  ExterQual      1460 non-null   object  27  ExterCond      1460 non-null   object  28  Foundation     1460 non-null   object  29  BsmtQual       1423 non-null   object  30  BsmtCond       1423 non-null   object  31  BsmtExposure   1422 non-null   object  32  BsmtFinType1   1423 non-null   object  33  BsmtFinSF1     1460 non-null   int64   34  BsmtFinType2   1422 non-null   object  35  BsmtFinSF2     1460 non-null   int64   36  BsmtUnfSF      1460 non-null   int64   37  TotalBsmtSF    1460 non-null   int64   38  Heating        1460 non-null   object  39  HeatingQC      1460 non-null   object  40  CentralAir     1460 non-null   object  41  Electrical     1459 non-null   object  42  1stFlrSF       1460 non-null   int64   43  2ndFlrSF       1460 non-null   int64   44  LowQualFinSF   1460 non-null   int64   45  GrLivArea      1460 non-null   int64   46  BsmtFullBath   1460 non-null   int64   47  BsmtHalfBath   1460 non-null   int64   48  FullBath       1460 non-null   int64   49  HalfBath       1460 non-null   int64   50  BedroomAbvGr   1460 non-null   int64   51  KitchenAbvGr   1460 non-null   int64   52  KitchenQual    1460 non-null   object  53  TotRmsAbvGrd   1460 non-null   int64   54  Functional     1460 non-null   object  55  Fireplaces     1460 non-null   int64   56  FireplaceQu    770 non-null    object  57  GarageType     1379 non-null   object  58  GarageYrBlt    1379 non-null   float64 59  GarageFinish   1379 non-null   object  60  GarageCars     1460 non-null   int64   61  GarageArea     1460 non-null   int64   62  GarageQual     1379 non-null   object  63  GarageCond     1379 non-null   object  64  PavedDrive     1460 non-null   object  65  WoodDeckSF     1460 non-null   int64   66  OpenPorchSF    1460 non-null   int64   67  EnclosedPorch  1460 non-null   int64   68  3SsnPorch      1460 non-null   int64   69  ScreenPorch    1460 non-null   int64   70  PoolArea       1460 non-null   int64   71  PoolQC         7 non-null      object  72  Fence          281 non-null    object  73  MiscFeature    54 non-null     object  74  MiscVal        1460 non-null   int64   75  MoSold         1460 non-null   int64   76  YrSold         1460 non-null   int64   77  SaleType       1460 non-null   object  78  SaleCondition  1460 non-null   object  79  SalePrice      1460 non-null   int64  dtypes: float64(3), int64(34), object(43)memory usage: 923.9+ KB<class 'pandas.core.frame.DataFrame'>Int64Index: 1459 entries, 1461 to 2919Data columns (total 79 columns): #   Column         Non-Null Count  Dtype  ---  ------         --------------  -----   0   MSSubClass     1459 non-null   int64   1   MSZoning       1455 non-null   object  2   LotFrontage    1232 non-null   float64 3   LotArea        1459 non-null   int64   4   Street         1459 non-null   object  5   Alley          107 non-null    object  6   LotShape       1459 non-null   object  7   LandContour    1459 non-null   object  8   Utilities      1457 non-null   object  9   LotConfig      1459 non-null   object  10  LandSlope      1459 non-null   object  11  Neighborhood   1459 non-null   object  12  Condition1     1459 non-null   object  13  Condition2     1459 non-null   object  14  BldgType       1459 non-null   object  15  HouseStyle     1459 non-null   object  16  OverallQual    1459 non-null   int64   17  OverallCond    1459 non-null   int64   18  YearBuilt      1459 non-null   int64   19  YearRemodAdd   1459 non-null   int64   20  RoofStyle      1459 non-null   object  21  RoofMatl       1459 non-null   object  22  Exterior1st    1458 non-null   object  23  Exterior2nd    1458 non-null   object  24  MasVnrType     1443 non-null   object  25  MasVnrArea     1444 non-null   float64 26  ExterQual      1459 non-null   object  27  ExterCond      1459 non-null   object  28  Foundation     1459 non-null   object  29  BsmtQual       1415 non-null   object  30  BsmtCond       1414 non-null   object  31  BsmtExposure   1415 non-null   object  32  BsmtFinType1   1417 non-null   object  33  BsmtFinSF1     1458 non-null   float64 34  BsmtFinType2   1417 non-null   object  35  BsmtFinSF2     1458 non-null   float64 36  BsmtUnfSF      1458 non-null   float64 37  TotalBsmtSF    1458 non-null   float64 38  Heating        1459 non-null   object  39  HeatingQC      1459 non-null   object  40  CentralAir     1459 non-null   object  41  Electrical     1459 non-null   object  42  1stFlrSF       1459 non-null   int64   43  2ndFlrSF       1459 non-null   int64   44  LowQualFinSF   1459 non-null   int64   45  GrLivArea      1459 non-null   int64   46  BsmtFullBath   1457 non-null   float64 47  BsmtHalfBath   1457 non-null   float64 48  FullBath       1459 non-null   int64   49  HalfBath       1459 non-null   int64   50  BedroomAbvGr   1459 non-null   int64   51  KitchenAbvGr   1459 non-null   int64   52  KitchenQual    1458 non-null   object  53  TotRmsAbvGrd   1459 non-null   int64   54  Functional     1457 non-null   object  55  Fireplaces     1459 non-null   int64   56  FireplaceQu    729 non-null    object  57  GarageType     1383 non-null   object  58  GarageYrBlt    1381 non-null   float64 59  GarageFinish   1381 non-null   object  60  GarageCars     1458 non-null   float64 61  GarageArea     1458 non-null   float64 62  GarageQual     1381 non-null   object  63  GarageCond     1381 non-null   object  64  PavedDrive     1459 non-null   object  65  WoodDeckSF     1459 non-null   int64   66  OpenPorchSF    1459 non-null   int64   67  EnclosedPorch  1459 non-null   int64   68  3SsnPorch      1459 non-null   int64   69  ScreenPorch    1459 non-null   int64   70  PoolArea       1459 non-null   int64   71  PoolQC         3 non-null      object  72  Fence          290 non-null    object  73  MiscFeature    51 non-null     object  74  MiscVal        1459 non-null   int64   75  MoSold         1459 non-null   int64   76  YrSold         1459 non-null   int64   77  SaleType       1458 non-null   object  78  SaleCondition  1459 non-null   object dtypes: float64(11), int64(25), object(43)memory usage: 911.9+ KB(1460, 80)(1459, 79)
We can see that the train dataset consists of 80 features in total including the target variable SalePrice and 1460 training examples. The test or prediction dataset consists of 79 features (SalePrice is to be predicted) and 1459 data-points.
Any data set will contain certain missing values in its features, be it numerical features or categorical features. This happens due to many reasons such as unavailability of data, wrong entry of data, etc. We need to check for the missing values in our train dataset. The following visualization can help us in doing it.
We can see that the features ‘PoolQC’ and some others have around 90% of their data missing. We will deal with them later.
There are mainly two types of columns, numerical and categorical. Let’s divide our features in train_df into each of those.
The numerical features are:
Index(['MSSubClass', 'LotFrontage', 'LotArea', 'OverallQual', 'OverallCond',       'YearBuilt', 'YearRemodAdd', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2',       'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF', 'LowQualFinSF',       'GrLivArea', 'BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath',       'BedroomAbvGr', 'KitchenAbvGr', 'TotRmsAbvGrd', 'Fireplaces',       'GarageYrBlt', 'GarageCars', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF',       'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal',       'MoSold', 'YrSold'],      dtype='object')
The categorical features are:
Index(['MSZoning', 'Street', 'Alley', 'LotShape', 'LandContour', 'Utilities',       'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'Condition2',       'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st',       'Exterior2nd', 'MasVnrType', 'ExterQual', 'ExterCond', 'Foundation',       'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1', 'BsmtFinType2',       'Heating', 'HeatingQC', 'CentralAir', 'Electrical', 'KitchenQual',       'Functional', 'FireplaceQu', 'GarageType', 'GarageFinish', 'GarageQual',       'GarageCond', 'PavedDrive', 'PoolQC', 'Fence', 'MiscFeature',       'SaleType', 'SaleCondition'],      dtype='object')
Let’s see how the data in numerical features are distributed. The distplot gives us the univariate distribution plot of each variable as shown below. These plots also help us in knowing about the skewness of the features and outliers in them, if any.
We can’t decide upon the outliers on distribution plots alone. Box plots help us in knowing more about outliers in features. The dots outside the blue box depicts the data-points that pose as outliers.
We could also plot the features along with the target variable to do bivariate analysis. A scatterplot would do the trick.
All these plots can be used to detect outliers and to know better about the distribution of the features and their relationship with the target variable.
By closely observing the above plots, it can be concluded that the following features have outliers in them:
LotFrontageLotAreaMasVnrAreaBsmtFinSF1BsmtFinSF2TotalBsmtSF1stFlrSFLowQualFinSFGrLivAreaWoodDeckSFEnclosedPorch3SsnPorchScreenPorchPoolAreaMiscval
LotFrontage
LotArea
MasVnrArea
BsmtFinSF1
BsmtFinSF2
TotalBsmtSF
1stFlrSF
LowQualFinSF
GrLivArea
WoodDeckSF
EnclosedPorch
3SsnPorch
ScreenPorch
PoolArea
Miscval
Let’s take a closer look at those features by plotting their regression plots.
We can now confirm that the features are having outliers. The next step will be to remove them. We can get the limits to remove them(from respective features) from the boxplots and regression plots.
There might be features in the data-set that won’t be contributing much to the target variable. We need to remove those to get better accuracy. Multiple features having a high correlation with each other may cause over-fitting. Let’s find out the highly correlated features(correlation>0.8). This can be done by a heatmap containing the correlation values of the features.
From the plot, we can see that the following features are highly correlated with each other:
1stFlrSF & TotalBsmtSFTotRmsAbvGrd & GrdLivAreaGarageYrBlt & YearBuiltGarageArea & GarageCars
1stFlrSF & TotalBsmtSF
TotRmsAbvGrd & GrdLivArea
GarageYrBlt & YearBuilt
GarageArea & GarageCars
Removing any one feature in each of these four sets would be sufficient. We can decide on which one to remove by looking into its contribution to the target variable SalePrice. This can be done by checking its correlation with SalePrice.
SalePrice        1.000000OverallQual      0.801493GrLivArea        0.718996GarageCars       0.647613TotalBsmtSF      0.644541GarageArea       0.6344461stFlrSF         0.622300FullBath         0.562614YearBuilt        0.552223YearRemodAdd     0.528484TotRmsAbvGrd     0.524650GarageYrBlt      0.508125MasVnrArea       0.493222Fireplaces       0.464076BsmtFinSF1       0.392553LotFrontage      0.360988LotArea          0.332449WoodDeckSF       0.329587OpenPorchSF      0.3228312ndFlrSF         0.296424HalfBath         0.289755BsmtUnfSF        0.230026BsmtFullBath     0.220147BedroomAbvGr     0.161515ScreenPorch      0.069728MoSold           0.0510673SsnPorch        0.038210BsmtHalfBath    -0.014339YrSold          -0.023857BsmtFinSF2      -0.038292MiscVal         -0.048533LowQualFinSF    -0.064357OverallCond     -0.081193MSSubClass      -0.085814EnclosedPorch   -0.134095KitchenAbvGr    -0.139927PoolArea              NaNName: SalePrice, dtype: float64
We will be dropping the features(among the pairs shown above)having less correlation with the target variable.
We had checked already visualized the number of missing values in the features earlier. We need to remove the features having missing values above 90%.
PoolQC         1.000000MiscFeature    0.966387Alley          0.936975Fence          0.809524FireplaceQu    0.479692dtype: float64
As we can see the features PoolQC, MiscFeature, Alley is having more than 90% missing values. So we should drop those features.
We could plot the scatterplots once again to see if any more useless features.
As we can see the feature PoolArea is pretty much unimportant as it is reporting pool area for all the training examples as zero, therefore we can drop it as well.
Now we have to deal with the numerical features having missing values.
LotFrontage    0.175070MasVnrArea     0.005602SalePrice      0.000000YrSold         0.000000LotArea        0.000000dtype: float64
We can see that the features LotFrontage and MasVnrArea having missing features. We can replace them with their mean value using the inbuilt reputation functions from sklearn.
After replacing, we can check once again for any missing values in numeric features:
SalePrice        0.0YrSold           0.0LotFrontage      0.0LotArea          0.0OverallQual      0.0OverallCond      0.0YearBuilt        0.0YearRemodAdd     0.0MasVnrArea       0.0BsmtFinSF1       0.0BsmtFinSF2       0.0BsmtUnfSF        0.0TotalBsmtSF      0.02ndFlrSF         0.0LowQualFinSF     0.0GrLivArea        0.0BsmtFullBath     0.0BsmtHalfBath     0.0FullBath         0.0HalfBath         0.0BedroomAbvGr     0.0KitchenAbvGr     0.0Fireplaces       0.0GarageCars       0.0WoodDeckSF       0.0OpenPorchSF      0.0EnclosedPorch    0.03SsnPorch        0.0ScreenPorch      0.0MiscVal          0.0MoSold           0.0MSSubClass       0.0dtype: float64
As we can see, we have handled all the missing values in the numeric features in the training data. Checking for testing data
LotFrontage      0.155586MasVnrArea       0.010281BsmtFullBath     0.001371BsmtHalfBath     0.001371GarageCars       0.000685BsmtFinSF1       0.000685BsmtFinSF2       0.000685BsmtUnfSF        0.000685TotalBsmtSF      0.000685LowQualFinSF     0.000000LotArea          0.000000OverallQual      0.000000OverallCond      0.000000YearBuilt        0.000000YearRemodAdd     0.0000002ndFlrSF         0.000000YrSold           0.000000GrLivArea        0.000000MoSold           0.000000FullBath         0.000000HalfBath         0.000000BedroomAbvGr     0.000000KitchenAbvGr     0.000000Fireplaces       0.000000WoodDeckSF       0.000000OpenPorchSF      0.000000EnclosedPorch    0.0000003SsnPorch        0.000000ScreenPorch      0.000000MiscVal          0.000000MSSubClass       0.000000dtype: float64
Around nine features are having missing values. We can use the same method used in the training set to replace them with their respective mean values.
Checking again for any missing values after replacing:
YrSold           0.0GrLivArea        0.0LotFrontage      0.0LotArea          0.0OverallQual      0.0OverallCond      0.0YearBuilt        0.0YearRemodAdd     0.0MasVnrArea       0.0BsmtFinSF1       0.0BsmtFinSF2       0.0BsmtUnfSF        0.0TotalBsmtSF      0.02ndFlrSF         0.0LowQualFinSF     0.0BsmtFullBath     0.0MoSold           0.0BsmtHalfBath     0.0FullBath         0.0HalfBath         0.0BedroomAbvGr     0.0KitchenAbvGr     0.0Fireplaces       0.0GarageCars       0.0WoodDeckSF       0.0OpenPorchSF      0.0EnclosedPorch    0.03SsnPorch        0.0ScreenPorch      0.0MiscVal          0.0MSSubClass       0.0dtype: float64
So, we have finished dealing with all the missing values in the numerical features in both the train and test dataset.
Let’s now look into the distribution of categorical features.
We can see that some of the features are totally skewed. We can verify by checking the frequency of the top occurring category of the features.
Utilities        1427Street           1423Condition2       1415RoofMatl         1407Heating          1397LandSlope        1356CentralAir       1333Functional       1329PavedDrive       1309Electrical       1302GarageCond       1295LandContour      1289GarageQual       1282BsmtCond         1279ExterCond        1256SaleType         1240Condition1       1238BsmtFinType2     1233BldgType         1190SaleCondition    1174RoofStyle        1126MSZoning         1121LotConfig        1035BsmtExposure      940LotShape          916ExterQual         893MasVnrType        849GarageType        846KitchenQual       726HeatingQC         721HouseStyle        713BsmtQual          638Foundation        634GarageFinish      599Exterior1st       508Exterior2nd       497BsmtFinType1      426FireplaceQu       366Neighborhood      221Fence             153Name: freq, dtype: object
As shown above, the features ‘Utilities’, ‘Street’, ‘Condition2’, ‘RoofMatl’, ‘Heating’ are highly skewed (since they are having a single entry around 1400 times out of 1428 examples). It is better we remove those features.
Now we are going to fill the missing values in the categorical variables. We will be filling in a new category ‘None’ for missing values in the categorical features.
Checking once again for missing values in the train dataset after replacing.
SaleCondition    0.0SaleType         0.0Foundation       0.0ExterCond        0.0ExterQual        0.0MasVnrType       0.0Exterior2nd      0.0Exterior1st      0.0RoofStyle        0.0HouseStyle       0.0BldgType         0.0Condition1       0.0Neighborhood     0.0LandSlope        0.0LotConfig        0.0LandContour      0.0LotShape         0.0BsmtQual         0.0BsmtCond         0.0BsmtExposure     0.0FireplaceQu      0.0Fence            0.0PavedDrive       0.0GarageCond       0.0GarageQual       0.0GarageFinish     0.0GarageType       0.0Functional       0.0BsmtFinType1     0.0KitchenQual      0.0Electrical       0.0CentralAir       0.0HeatingQC        0.0Heating          0.0BsmtFinType2     0.0MSZoning         0.0dtype: float64
Now checking the test dataset’s categorical features for missing values.
Fence            0.801234FireplaceQu      0.500343GarageCond       0.053461GarageQual       0.053461GarageFinish     0.053461GarageType       0.052090BsmtCond         0.030843BsmtQual         0.030158BsmtExposure     0.030158BsmtFinType2     0.028787BsmtFinType1     0.028787MasVnrType       0.010966MSZoning         0.002742Functional       0.001371Exterior2nd      0.000685KitchenQual      0.000685SaleType         0.000685Exterior1st      0.000685HouseStyle       0.000000LotShape         0.000000LandContour      0.000000LotConfig        0.000000LandSlope        0.000000Neighborhood     0.000000Condition1       0.000000BldgType         0.000000HeatingQC        0.000000RoofStyle        0.000000Heating          0.000000PavedDrive       0.000000ExterQual        0.000000ExterCond        0.000000Foundation       0.000000Electrical       0.000000CentralAir       0.000000SaleCondition    0.000000dtype: float64
We will again replace all the missing values with a new category ‘None’ as done in the training dataset.
Now again checking for missing values in test_df after replacing.
SaleCondition    0.0SaleType         0.0Foundation       0.0ExterCond        0.0ExterQual        0.0MasVnrType       0.0Exterior2nd      0.0Exterior1st      0.0RoofStyle        0.0HouseStyle       0.0BldgType         0.0Condition1       0.0Neighborhood     0.0LandSlope        0.0LotConfig        0.0LandContour      0.0LotShape         0.0BsmtQual         0.0BsmtCond         0.0BsmtExposure     0.0FireplaceQu      0.0Fence            0.0PavedDrive       0.0GarageCond       0.0GarageQual       0.0GarageFinish     0.0GarageType       0.0Functional       0.0BsmtFinType1     0.0KitchenQual      0.0Electrical       0.0CentralAir       0.0HeatingQC        0.0Heating          0.0BsmtFinType2     0.0MSZoning         0.0dtype: float64
We have replaced all the missing values in all the features in the train and test dataset. This concludes our EDA and data cleaning.
As we saw in the above sections, performing EDA is a tiring task, as it takes up a lot of time and effort to visualize the data, perform different types of analysis, and come to a conclusion about the data. All this work can be made simpler by using a new and extremely useful library called Sweetviz. It takes a Pandas dataframe and creates a self-contained HTML report. And all this can be done in just 2 lines of code!
Here is a link to the HTML file containing the report of both the train and test dataset. We can get all types of visualizations such as distribution plots and correlation plots. We are also provided with numeric details such as the number of missing values, frequently occurring entries, skew, kurtosis, etc. This library is a better version of Pandas profiling and much cooler visualizations.
In this article, we have have seen the different steps involved in the process of EDA. We saw how to visualize the data in various plots for performing different types of analysis. We learnt how to detect outliers using these plots and how to remove them. We dealt with missing values by using methods suitable for the type of feature. Then we saw how all the analysis part could be done in a few lines of code using Sweetviz.
www.kaggle.com
The public notebooks submitted by creators in the Kaggle competitions contains a lot of information regarding dealing with different types of problems encountered during EDA. If you have time, I would recommend to go through multiple notebooks and fork down the methods to implement them in your own notebook. I learnt a lot about EDA and modelling regarding the housing prices dataset from the public notebooks in Kaggle.
towardsdatascience.com
This article in TDS directed me towards using Sweetviz for EDA. Kudos to Francois Bertrand and other contributors! Even though the visualizations and inferences are self-explanatory, please refer the article for details.