Understanding Data Analysis Step-By-Step | by Himanshu Sharma | Towards Data Science

In today’s world, over 2.5 quintillion bytes of data are created every single day. Being a Data Science Engineer/Data Scientist/ML Engineer or whatever you might call the art of data analysis is the first thing that you should master.
It is a process of inspecting, cleansing, transforming, and modeling data so that we can derive some useful information from the data and use it for future predictions.
Data analysis tools make it easier for users to process and manipulate data, analyze the relationships and correlations between data sets, and it also helps to identify patterns and trends for interpretation.
Here, we will be using python to study data analysis in-depth and study all the important aspects of data analysis. Here we will be using a car design dataset which you can download here.
Starting with importing the important libraries and reading the dataset
#importing the important librariesimport numpy as npimport pandas as pdimport matplotlib.pyplot as pltimport seaborn as sns#reading the dataset and creating a dataframedf = pd.DataFrame(pd.read_csv('car_design.csv'))df.head()
As we can see here that the data is loaded in the data frame named ‘df’, so let us start with the data wrangling part like knowing the dimensions of the data, how many rows and columns do data have.
# Dimention of datasetprint("Dataset is of ", df.ndim, " dimension.")# Rows and column of datasetprint("Dataset has ", df.shape[0], " rows.","\nDataset has ", df.shape[1], " columns.")#Knowing the data Types of the columnsprint("Data Types :")print(df.dtypes)
All the above-mentioned steps will help us understand the data and its attributes.
While we go through the data we see that some of the columns contain “?” instead of correct data which is considered as an anomaly in data. We need to replace it with ‘NaN’ to perform further operations.
df = df.replace("?", np.nan)df.head()
Now we will replace the missing data i.e. NaN with the column of that mean, this operation can only be performed on the columns with data type float. So we will convert the selected columns, convert them to float, and replace the missing data with the mean.
# Replacing the missing data with respective column meanfor i in ['normalized-losses', 'bore', 'stroke', 'horsepower', 'peak-rpm', 'price']:    df[i] = df[i].fillna(round(df[i].dropna().astype('float64').mean(),2))
As the data is now cleaned and without any anomalies, let us move forward and look at the statistical description of the data. For this, we will use the describe function. Describe function computes the summary of statistics pertaining to the Data Frame columns.
df.describe()
Now let us move forward to the next step.
In this, we will try to visualize different columns according to the problem statement and find out the relation between different attributes if any. As we are doing it for the general purpose we will be visualizing columns of our choice.
# Plotting histograms of the following numerical attributes --      # "engine-size", "peak-rpm","horsepower","price"hist_col_names = ["engine-size", "peak-rpm","horsepower","price"]for i in hist_col_names:    df[i] = df[i].astype(float) # It changes data typeplt.figure(figsize=(15,15))cnt=1for i in hist_col_names:    plt.subplot(4,2,cnt)    sns.distplot(df[i], hist=True, color = 'darkblue', kde_kws={'linewidth': 4})    plt.title(i)    cnt=cnt+1plt.tight_layout()plt.show()
With a little knowledge of statistics, we can clearly understand what thee distribution plots are trying to say. After this, we can also find the correlation of the different numerical attributes and visualize them using the Heatmap.
A heatmap is a two-dimensional graphical representation of data where the individual values that are contained in a matrix are represented as colors.
# Creating the correaltion matrixcorr = df.corr()# Plotting the correlation matrix on a heatmapfig, ax = plt.subplots(figsize=(12,9))sns.heatmap(corr, annot=True)ax.set_xticklabels(ax.get_xticklabels(), rotation=45);plt.show()
Another type of Data Visualization plot which is really insightful is known as Box-Plot, it is a method for graphically depicting groups of numerical data through their quartiles. Box plots may also have lines extending from the boxes indicating variability outside the upper and lower quartiles.
Let us plot a box-plot between ‘drive-wheels’ and ‘price’ attribute of the dataset. This will help us find out the price range for different drive-wheels car.
plt.figure(figsize=(8,8))sns.boxplot(x="drive-wheels", y="price", data=df,linewidth=1.5, palette ='Set2')
Now let us move to the final step that is drawing some insights from the data. Fo this let us imagine that our Problem Statement as “We need to find out the price range for different types of cars launched by a new company?”
In order to solve the problem statement let us first find out the average price of the cars according to Drive-Wheels.
df_group=df[['drive-wheels', 'body-style', 'price']]drive_wheel_avg=df_group.groupby(['drive-wheels'], as_index=False).mean()drive_wheel_avg
Now we will display the average of drive-wheels according to the body-style of the different cars.
group_avg=df_group.groupby(['drive-wheels','body-style'] , as_index=False).mean()group_avg
The final step is to convert the data obtained in the above step to a pivot table for better visualization, precise view, and understanding.
pivot_table=group_avg.pivot(index='drive-wheels', columns='body-style')pivot_table=pivot_table.fillna('Not Applicable')pivot_table
This table can help us find the price segment in which we can launch the car according to the data of other companies that are already in the market. Here we can clearly see that this inference is only based on limited attributes, if we want to get more precise inference we need to consider more attributes.
In this article, I have explained to you the basic steps that you should follow whenever you are working on a dataset. There is no hard and fast rule to follow these steps but they help you to get better insights.
Please find the dataset and the completed python file in my github where I have considered more attributes that you can go through for better understanding.
towardsdatascience.com
towardsdatascience.com
Thanks for reading! If you want to get in touch with me, feel free to reach me on hmix13@gmail.com or my LinkedIn Profile. You can also view the code and data I have used here in my Github.