How to use Pandas to analyze numeric data? | by WY Fok | Towards Data Science

Python already becomes a popular programming language in the data science area. Pandas, one of many popular libraries in data science, provides lots of great functions that help us transform, analyze and interpret data. I am sure that there are already too many tutorials and materials to teach you how to use Pandas. However, in this article, I am not solely teaching you how to use Pandas. Instead, I would like to use an example to demonstrate how to interpret your data correctly and smoothly, with the help of Pandas.
Panda VS Pandas
Prepare a python environmentDownload dataset from KaggleImport pandas
Prepare a python environment
Download dataset from Kaggle
Import pandas
Prepare a python environment
Prepare a python environment
Since we use Python to analyze data, of course first we need to install Python. For a beginner, I strongly recommend you to install Anaconda, since Anaconda will already pre-install a list of libraries that are very useful for data science. And you don’t need to concern about compiler or IDE.
Here is the link for downloading Anaconda Distribution which is free
Go to download
Jupyter Notebook is a great tool to do data analysis as you can see the result part by part so that you can know if the result is what you predict and you can also know if you make any error immediately.
After launching Jupyter Notebook, a webpage will pop out and you will see a page like this. You can select and create a path for storing your files. On the right side, there is a “New” button. Click this to create a folder or Python file.
There are too many useful functions and shortcuts provided by Jupyter Notebook. I am not here to describe and explain in detail. Also, I hope you already have some basic knowledge of coding Python since this is not an introduction to Python. Either way, if you have any questions feel free to comment and I will resolve your problem.
2. Download dataset from Kaggle
The dataset that will be used in this article is from Kaggle. The dataset includes suicide rates from 1985 to 2016 across different countries with their socio-economic information.
Download link
3. Import pandas
Last preparation, import pandas
Normally we will give an abbreviation for each library. You can skip this but then each time when you want to use any function you have to type in the whole name. So why not give it a short abbreviation.
Although this article focuses on using pandas, we still need some helps from numpy so we will also import numpy.
The first step is to import the dataset that we download from Kaggle. Although you can import the data from any path, my suggestion is that put all data and codes together so that you don’t need to type the exact path but just the file name. The file name downloaded is “master.csv”. So we know the dataset is in a csv file type. Other common file types include excel file and json.
The first pandas function we use by pandas is read_csv. A dataframe called data is created by:
data= pd.read_csv('master.csv')
We can use this to import a csv file to python and store it as a dataframe. Dataframe is like an excel table.
Normally pandas automatically interprets the dataset and identifies all necessary parameters in order to import the dataset properly. However, if you find out the output is not as expected as you desire, you can manually change parameters. Below are some common parameters which you can modify whey you use read_csv :
sep: Delimiter used to separate values across variables. Default is using comma ‘,’ . You can change the sep similar to below
sep: Delimiter used to separate values across variables. Default is using comma ‘,’ . You can change the sep similar to below
data = pd.read_csv('master.csv', sep = ';')
2. header: Normally a csv file has the first row showing the names of all variables. However, if there is no such name row in the csv file, it is better to tell pandas that there is no header by
header = None
Now you already import the dataset. As said before, you can see the result directly after running your code. So let’s see the dataset
data
I hope you will get the same result as mine. Now as you scrolling down, you can see a number but not all records. If you don’s want to show so many records, you can show some top or bottom records by typing
data.head(5) # showing top 5 records from the dataframe# ordata.tail(5) # showing bottom 5 records from the dataframe
Knowing the dataset is necessary before carrying any analysis as first this will give you a brief understanding and second this will prevent you from misinterpreting when you analyze.
The first question you may ask after importing is how many numbers of records in the dataset. Luckily you can get the answer with a line of code.
data.shape
The first number (index 0, python starts from zero) shows the numbers of row s and the second number (index 1) shows the numbers of columns. So the dataset has 27,820 rows and 12 columns.
The second you may ask is what these 12 columns are. There are plenty of methods to shows column names. But my favorite is to use
data.info()
info() doesn’t only provide you with all column names in the dataframe but it also provides you with the type of data stored in this dataframe. Also, you can know if there is a missing value in the dataframe. As shown above, the number column shows how many non-null counts in that variable. All show 27,820 except HDI for the year. So at least we know there are missing values in the dataset that we import.
Now you want to know what data is inside this dataframe. For example, how many countries are there in those almost 28k records? You want to get a list of all countries. You can get values of each column by
data['country']
But this will return you all values without deduplication. If you want to get a unique country, my favorite method is to use set function. Set function will only return a list of unique values without duplication.
So how many countries in total?
There are in total of 101 countries.
Then you want to know the number of suicides for each year. In Excel, you can perform this by pivot table. There is also a similar function for pandas called pivot_table. Imagine you are now using an excel pivot table to get the number of suicides. You will drag year to row label and drag suicide_no to Values
If you are familiar with the excel pivot table, then remember the same format as pandas pivot_table.
data.pivot_table(index='year',values='suicides_no',aggfunc='sum')
Oh, you want to break down into sex also?
data.pivot_table(index='year',columns='sex',values='suicides_no',aggfunc='max')
Here ‘index’ represents the variable to be placed in row label; ‘columns’ represents columns; ‘values’ represents the number in the pivot table. Just like excel, you can calculate sum, mean, max or min.
data.pivot_table(index='year',columns='sex',values='suicides_no',aggfunc=['sum','mean','max','min'])
You can also perform multi-index on row or column.
data.pivot_table(index=['country','year'],columns='sex',values='suicides_no',aggfunc=['sum'])
By using a pivot table, we can know a general breakdown of the dataset in terms of different variables. You can also spot any abnormalities in the dataset which may help you further analysis. Just like below:
data.pivot_table(index=['year'],columns='sex',values='suicides_no',aggfunc=['sum'])
What happened in 2016? Why was there a sudden drop?
Now you want to check the dataset and see what happens for 2016. In pandas, you need to specify which row and column you want to return. This can be done by providing index numbers or names or criteria. Now the criteria is that year has to be 2016. You can get the result by
data[data['year']==2016]
How to interpret this line of code?
There are two parts. The inner part “ data[‘year’]==2016 ” is a condition that needs to fulfill. Here means the year column in the data must be 2016. Be cautious that there are two equal signs, representing a comparison. Similarly, you can have a condition with larger, smaller, or not equal
data['year'] >= 2016 # larger than or equal to data['year']!=2016 # not equal to
More conditions, no problem. But make sure that you specify clearly if you want to fulfill all or either one condition.
# year must be larger than 2010 and smaller than 2015(data['year'] > 2010) & (data['year']<2015)# year can be larger than 2010 or smaller than 2000(data['year'] > 2010) | (data['year']<2000)
The outer part “ data[...] ” means to return all rows that fulfill the criteria in the inner part. Back to the example, we want to return all rows with year equal to 2016.
Seems no problem. But if we count the number of countries in 2016 and 2015, the problem arises.
Now we want to get the column country from the dataset which the year must be 2016. The easy way is to extend the previous line of code with
data[data['year']==2016]['country']
And then use set and len functions again to calculate the number of countries in 2016.
len(set(data[data['year']==2016]['country']))
Do the same thing for 2015. Here we come
The data in 2016 is clearly incomplete and so we should not compare 2016 with other years. So we should better remove all records for 2016.
As said, we can apply a condition to filter records from 2016 as below:
data2= data[data['year']!=2016]
Now we create another dataframe called ‘data2’ for further analysis.
Finding which variables are related to the dataset is crucial as this can help you discover which variables can cause the change of the results. For example, in the dataset there are some categorical variables:
country, year, sex, and generation
Pick a simple one, sex, as for example. You can make a comparison of the number of total suicides for each sex. If there is a large difference, it is safe that sex is a factor causing the difference in term of suicide
Of course, we can use pivot_table function to do so. But I am here to introduce another function, groupby. If you have experience in using SQL, I am sure you are familiar with the group by.
data2.groupby('sex').agg({'suicides_no':'sum'})
In term of SQL, the above statement is equivalent to:
select sex, sum(suicides_no) from data2 group by sex
Python will first group all records based on the values in ‘sex’ columns. After that, python will calculate the total number of suicides for each group of ‘sex’.
Male suicides are much higher than female suicides. At this moment, we can quite sure that sex is a factor affecting the number of suicides.
Brillant thing about groupby is that you can again split the dataset into more levels. Remember when you include more than one level, you need to use a square bracket [ ] to provide a list to pandas.
data2.groupby(['sex','generation']).agg({'suicides_no':'sum'})
Also, just like SQL, you can perform multiple aggregations here in groupby function.
data2.groupby(['sex']).agg({'suicides_no':['sum','max']})
or
data2.groupby(['sex']).agg({'suicides_no':'sum', 'population':'mean'})
Now we know sex is a factor. Then how about age?
data2.groupby(['age']).agg({'suicides_no':'sum'})
So the most popular age range for suicide is 35–54. Right?
Not quite. Not like sex which is around half-half, the population for each age group is not even. This is a common misinterpretation when compare with different categories when they have different sizes of population or base.
As a result, in order to have a better comparison, we should better compare suicide numbers with each total population. To do this in pandas, first create another dataframe to store suicide_no and population at each age level. Then calculate a division to get the average number of suicide for each age group.
# create a dataframe called data_age for age groupby data_age = data2.groupby([‘age’]).agg({‘suicides_no’:’sum’,’population’:’sum’})# calculate average of suicide number for each age group data_age[‘suicides_no’]/data_age[‘population’]
The first part of the code is similar as before. The second part is to perform a division. The good thing for python is that you can perform a calculation as a whole column instead of each by each cell like excel. So we divide the suicide_no column from population column.
Now the result is different. The suicidal situation gets worse as the age increases.
At this moment, you may say the ranking is not clear and you want to order the result in terms of the average suicide number. Then it is time to introduce sort_values function.
(data_age['suicides_no']/data_age['population']).sort_values()
Here because there is only one column, we do not need to specify which column to sort by. However, you have to specify which column if there is more than one column in the dataframe. Also, the default order is ascending and you can change it by adding ascending = False inside the bracket. Just like below:
(data_age['suicides_no']/data_age['population']).sort_values(ascending=False)
Now you can easily see the increasing trend by age.
We have already studied the impacts of sex and age. Now we move on to the country. We can find which country is more popular with suicide.
The first step is to create a table including numbers of suicides and population by country level. This can be done by using groupby function.
data_country = data2.groupby(['country']).agg({'suicides_no':'sum','population':'sum'})
Here we don’t care if all countries have the same number of records as we are calculating the average number over the period.
Then we calculate the average suicide by dividing suicides_no from population.
data_country['average_suicide'] = data_country['suicides_no']/data_country['population']
Here we create a new column called ‘average_suicide’ which will store the division result. Later use sort_values function to get the results in descending order. Instead of showing all countries, we can just show top N countries by using the head function
data_country.sort_values(by='average_suicide',ascending=False).head(10)
From the result, we can see most countries are in Eastern Europe. Then you can say the geolocation is also an indicator of affecting suicide.
That’s all for today. I hope you will not only learn how to use Python but also how to handle your dataset to draw better and more accurate conclusions.
Welcome to give comment so I know how to improve and write a better blog. If you like, give a clap and share it to people who are also in need like you. See you next time.