THE DATA ANALYSIS PROCESS
By: Bhavna Kumari
Tags: Data Analysis, Exploratory Data Analysis, Process, Data Science, Machine Learning

Hello Everyone! So, today in this blog we are going to learn about the data analysis process. we will be using an example of the “TITANIC dataset” for more understanding of the process. If you all want to see my full analysis you can see using this link.

DATA ANALYSIS PROCESS

At first, What is Data Analysis?

According to Wikipedia, Data analysis is a process of inspecting, cleansing, transforming and modeling data with the goal of discovering useful information, informing conclusions and supporting decision-making.

STEPS OF THE DATA ANALYSIS PROCESS:

There are basically 5 main steps of the Data Analysis Process :

Asking the right question from the data. Data Wrangling/ Data Munging / Data Preprocessing in simple terms called as Cleaning of Data. Exploratory Data Analysis in simple terms called exploring the data. Drawing Conclusions. Communicating Results /Story Telling.

STEP 1: ASKING THE RIGHT QUESTION

ASKING THE RIGHT QUESTION

The first step is to ask the right question from the data, to identify the objective of our analysis and also to draw a conclusion of the analysis.

Here in this Titanic dataset, our objective is to find out the factors which are responsible for a person’s chance of survival on the titanic.

STEP 2: DATA PREPROCESSING:

DATA PREPROCESSING

It is also known as Data Wrangling/Data Munging is the process of gathering, assessing, and cleaning of “raw” data into a form suitable for analysis. It has three subparts:

1.Gathering Data: After identifying the objective behind our analysis, the next step is collecting the necessary data required by us to draw appropriate conclusions. There are various methods by which we can collect data:-

API or Web Scraping: If the data needed is available on a particular website, then we can use the websites API (if available) or Web Scraping techniques to collect, and store data in our local storage. Data collected from the API is stored in a JSON format, and further, it is converted from JSON to the “.csv” format.

Databases: If the data we need is in the company database, then we can use SQL libraries to extract data from the database.

Sites: like kraggle.com, etc has data sets in appropriate formats to be downloaded by the users.

Here in this blog, we will use the Titanic dataset for our analysis:

At first, we have to import all the libraries and the “train.csv” which we will need for our analysis.

GATHERING DATA

2. Accessing Data: After the data is gathered and stored in a variable in the python file, we have to take a high-level overview of the data or in easy words getting more familiarised with the dataset.

ACCESSING DATA

The above output shows the name of columns, along with several non-null values and the data type of each column. From the above output, it is clear that Age, Cabin, and Embarked columns have missing values, which we need to deal with in the Data Cleaning stage.

INFORMATION ABOUT THE COLUMNS OF THE DATA:

PassengerId— Unique ID of each passenger.

Survived — Binary Feature (consisting of only 0 or 1 values) which indicates whether that passenger survived or not. ( 0 = NO, 1 = YES).

— Binary Feature (consisting of only 0 or 1 values) which indicates whether that passenger survived or not. ( 0 = NO, 1 = YES). Pclass — Indicates the socioeconomic status of the passenger. ( 1 = Upper Class, 2 = Middle Class, 3 = Lower Class ).

— Indicates the socioeconomic status of the passenger. ( 1 = Upper Class, 2 = Middle Class, 3 = Lower Class ). Name — Passenger name.

— Passenger name. Sex — Male or Female.

— Male or Female. Age — Age of the passenger.

— Age of the passenger. SibSp — Number of siblings/ spouses of the passenger aboard the titanic.

— Number of siblings/ spouses of the passenger aboard the titanic. Parch — Number of parents/ children of the passenger aboard the titanic.

— Number of parents/ children of the passenger aboard the titanic. Ticket — Ticket number of each passenger.

— Ticket number of each passenger. Fare — Fare paid by the passenger.

— Fare paid by the passenger. Cabin — shows the cabin number allotted to each passenger.

— shows the cabin number allotted to each passenger. Embarked — Shows the port of embarkation of the passenger. ( C = Cherbourg, Q = Queens town, S = Southampton ).

3.Data CLeaning Stage :According to Wikipedia,Data cleansing or data cleaning is the process of detecting and correcting (or removing) corrupt or inaccurate records from a recordset, table, or database and refers to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or coarse data.

FACT: Around 70% of the time of data analysis is spent on data cleaning only.

Here in this “TITANIC ”dataset, we have three columns( Age, Cabin, and Embarked columns) that have missing data so we will first fill those missing rows.

DATA CLEANING

AGE: Age column has a mean of 29.69 and a standard deviation of 14.52. That means it’s not possible to simply fill the missing values as the mean value as the standard deviation is very high. So we need to generate a list of random numbers equal to the length of missing values, between (mean-standard deviation) and (mean+standard deviation). So that we can fill up the missing values in the Data Frame with that of those in the list.

CODE FOR FILLING THE MISSING THE VALUES

OUTPUT

So, we have filled all the missing values of the Age column with the random values from(mean-std) to (mean+std).

STEP 3: EXPLORATORY DATA ANALYSIS(EDA) :

EXPLORATORY DATA ANALYSIS(EDA)

After the data is collected, cleaned, and processed, it is ready for Analysis. In this process you can manipulate your data, to find the exact information you need, or you might need to collect more data. During this phase, you can use data analysis tools and software which will help you to understand, interpret, and derive conclusions based on the requirements.

As the “TITANIC” Dataset is now cleaned we will perform some EDA on it.

At first, we will drop those columns which we don't need like PassengerId, Name, Ticket in our titanic data set and fill all the missing values in the data.

DROPPING UNWANTED COLUMNS

FILLING MISSING VALUES

TOTAL NO OF PEOPLE ON TITANIC ACCORDING TO CLASS:

TOTAL NO OF PEOPLE ON TITANIC ACCORDING TO CLASS

2. TOTAL NO OF PEOPLE WHO SURVIVED:

TOTAL NO OF PEOPLE WHO SURVIVED

3. FIND OUT WHETHER THE CLASS OF A PERSON HELPED HIM IN SURVIVING?

As you can see the passenger class contributes to the passenger survival

4. FIND OUT THE SURVIVAL RATE PER CLASS:

As you can see the lower class survival rate is very high.

FOR FULL ANALYSIS PLEASE SEE MY JUPYTER NOTEBOOK USING THIS link.

STEP 4: DRAWING CONCLUSION:

After the analysis is done, the next part is to find out the conclusion from the dataset.

There are 3 key questions which should be asked by us after the analysis: —

Did the analysis answer my original question?

Was there any limitation in my analysis which would affect my conclusions?

Was the analysis sufficient enough to help decision making?

From the analysis of the titanic data set, we were able to find out the major factors which contributed to a person’s chance of survival.

CONCLUSION

So, the conclusion from the “TITANIC” dataset is that:

Males had a higher chance of survival if they belonged to the upper class., Had an age between 0 to 4 years old, or 18 to 50 years old, and had 1 to 3 relatives traveling onboard the titanic.

Females had a higher chance of survival irrespective of their class, if they had an age between 0 to 4 years old, or 15 to 50 years old, and had 0 to 4 relatives traveling onboard the titanic.

STEP 5: COMMUNICATING RESULTS / STORYTELLING:

COMMUNICATING RESULTS

After asking questions, cleaning, exploring and drawing the conclusion from the data is done. The time for communicating the results of your analysis is arrived to the concerned people or your team. It is a kind of storytelling about what exactly you have done in your analysis through presentations, blogs or designing reports. Great soft skills are a plus in this step since your findings need to be communicated in a proper way to other people.

INTERESTING FACT: THE STEPS OF THE DATA ANALYSIS PROCESS IS NOT LINEAR YOU CAN PERFORM ANY STEP ANY TIME.

STEPS OF DATA ANALYSIS

Here is an example,Suppose you have done your analysis, drawn conclusions, then suddenly you find the possibility of representing a feature in a better way, or to construct a new feature out of other features present in the data set; thus, you would go back to step 3, perform feature engineering, and again perform the EDA with the new features added.

Thus, it is not always possible to follow these steps linearly.

CONCLUSION:

Following data collection, the data needs to be critically analyzed. For any research, data analysis is very important as it provides an explanation of various concepts, theories, frameworks and methods used. It eventually helps in arriving at conclusions and proving the hypothesis. I hope you all liked it so please give it a clap.