Exploratory Data Analysis(EDA) In Python
Tags: Kaggle, Python, Plotly, Seaborn, Exploratory Data Analysis

Introduction :

Photo by Luke Chesser on Unsplash

Exploratory Data Analysis is one of the best practices used in data science today.Basically it is essentially a type of storytelling for statisticians.We can say EDA as “A first look at the data”. Exploratory Data Analysis or (EDA) is understanding the data sets by summarizing their main characteristics often plotting them visually.The main goal of exploratory data analysis is to obtain confidence in your data to a point where you’re ready to engage a machine learning algorithm.This step is very important especially when we arrive at modelling the data in order to apply Machine Learning.Exploratory Data Analysis is nothing but the complete film of the data. When we watch the characters in film, we get to understand the relations between them. How to link the characters to the story of film so that it will never go off track is very important to imagine for a writer. That curious imagination is required for the data engineer to explore the data-set. In a hurry to get to the machine learning stage, some data scientists either entirely skip the exploratory process or do a very perfunctory job. This is a mistake with many implications, including generating inaccurate models, generating accurate models but on the wrong data, not creating the right types of variables in data preparation, and using resources inefficiently because of realizing only after generating models that perhaps the data is skewed, or has outliers, or has too many missing values, or finding that some values are inconsistent.So, one of the most important components to any data science experiment that doesn’t get as much importance as it should is Exploratory Data Analysis (EDA). Hence it is a critical step in analyzing the data from an experiment.

Benefits

It is used to understand and summarize the content of the data set to ensure that the features which we feed to our machine learning algorithms are refined and we get valid, correctly interpreted results. Gives some insight about the data set and some understanding the underlying structure. Helps us to extract important parameters and relationships that hold between them. Helps us to test underlying assumptions.

Difference between Data Analysis and Exploratory Data Analysis

Exploratory data analysis is a complement to inferential statistics, which tends to be fairly rigid with rules and formulas. At an advanced level, EDA involves looking at and describing the data set from different angles and then summarizing it.

Data Analysis is the statistics and probability to figure out trends in the data set. It is used to show historical data by using some analytics tools. It helps in drilling down the information, to transform metrics, facts, and figures into initiatives for improvement.

Data set

So here we are going to use the HuBMAP dataset and the entire code can be found in the following link:

https://www.kaggle.com/vineeth1999/hubmap-eda-pytorch-efficientunet-offline-training

Since this is an image competition we will also use some of the image functions and also some encoding functions (Run Length Encoding) to know about the dataset and also the competition more.

1) Importing the required Libraries

pandas : used to read .csv file and store it as data-frame.

numpy : used for mathematical calculations

matplotlib, seaborn, plotly : used for plotting colorful graphs

coloroma : used for colorful texts while printing

cv2, PIL: used for loading and working with images

tiff file: used for loading tiff files and converting into .jpg or .JPEG files

2) Listing the files in the dataset

We can see that there are 5 files in the dataset. So basically, train.csv contains the unique IDs for each image, as well as an RLE-encoded representation of the mask for the objects in the image and HuBMAP-20-dataset_information.csv contains additional information (including anonymized patient data) about each image.

Run-Length-Encoding (RLE): RLE is a form of lossless data compression in which runs of data (sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run. This is most useful on data that contains many such runs. Consider, for example, simple graphic images such as icons, line drawings, Conway’s Game of Life, and animations. It is not useful with files that don’t have many runs as it could greatly increase the file size.

Run Length Encoding

Run Length Encoding in python

This algorithm is very easy to implement and does not require much CPU horsepower. RLE compression is only efficient with files that contain lots of repetitive data. These can be text files if they contain lots of spaces for indenting but line-art images that contain large white or black areas are far more suitable. Computer-generated color images (e.g. architectural drawings) can also give fair compression ratios. And this is used to compress Tiff and Pdf files.

Run Length Encoding Sample for images

Run Length Encoding for an image in action

3) Reading the csv files

loading csv and displaying train csv file as data-frame

displaying hubmapdataset.csv as a data-frame

.csv files can be read and converted into data-frames using read_csv() function of pandas library and each data is separated by the delimiter “,” in the given set.

.head(n) function returns the first ’n’ observations from the dataset.

.tail(n) function returns the last ’n’ observations from the dataset.

.sample(n) function returns random ’n’ observations from the dataset.

.shape function returns the number of rows*number of columns from the dataset.

.columns returns the column names as a list for the given dataset.

.value_counts() returns the count of each value in a given column

4) Basic Data Exploration

To find what all columns it contains, of what types and if they contain any value in it or not, with the help of .info() function.

General Information

By observing the above data we can conclude that train.csv contains 2 object co 8 non-null columns and hubmap20dataset.csv contains 7 object columns, 6 int64 columns and 2 float64 columns with 13 non null columns.(Non-null means no missing values).

Missing value count

From the above pic we can see that there are 3 missing values in the hubmap20dataset. The function “isna().sum()” of pandas is used to return the missing value count of each column in the data-frame

5) Detailed Data Exploration