UCL Data Science Society: Pandas. Workshop 6: What is Pandas, Pandas... | by Philip Wilkinson | Towards Data Science

This year, as Head of Science for the UCL Data Science Society, the society is presenting a series of 20 workshops covering topics such as introduction to Python, a Data Scientists toolkit and Machine learning methods, throughout the academic year. For each of these the aim is to create a series of small blogposts that will outline the main points with links to the full workshop for anyone who wishes to follow along. All of these can be found in our GitHub repository, and will be updated throughout the year with new workshops and challenges.
The sixth workshop in the series is an introduction to the pandas python package in which we introduce you to Pandas, Pandas Series, Pandas DataFrame, accessing data and Pandas operations. While some of the highlights will be shared here, the full workshop including a problem sheet to test yourself can be found here.
If you have missed any of our previous workshops you can find the last three at the following links:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Pandas is an open source Python package that is widely used for data science/analysis and machine learning tasks, building on top of the Numpy package that we introduced last week. It is one of the most popular data wrangling packages for data science workflows and integrates well with many of the other libraries used during this process. It is useful because it makes tasks like:
Data cleaning
Data filling
Data normalisation
Merges and joins
Data visualisation
Data inspection
so much simpler than they would otherwise be. This functionality helps with its aim of becoming:
...the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language.
as described in the documentation.
Pandas series are the fundamental structure that make up the building blocks of pandas dataframes. While they are similar to Numpy arrays in their structure, the main difference is that we can specify a non-numerical index for a pandas series that is always linked, allowing us to access the location of the information based on that value. However, in most other respects, such as when applying numerical operations, series can behave much like arrays or lists.
To generate a series we need to use the pd.series(x) notation where x is a list or an array which is to be converted into the series. For this, we can use the index argument to set index = y where y is a list of the indices we want to use for the respective series values. This means that we can use both numerical and string notation (i.e. min, median, max rather than 1,2,3) if we so desired. An example of this is:
# Generating Pandas seriesseries = pd.Series([10,20,30,40])print(series)#out:0    101    202    303    40dtype: int64
or:
# Generating Pandas seriesseries = pd.Series([10,20,30,40], index=["a","b","c","d"])print(series)#out:a    10b    20c    30d    40dtype: int64
These series can be combined to make up a Pandas DataFrame, which is the main data structure that is associated with Pandas. Essentially, his is a table of data stored as a variable similarly to how you may store a dictionary of lists or multiple values in an array as a variable. Each column in a DataFrame is represented by a Pandas series meaning that we could extract each column seperately if we wished and the index would remain the same.
We can create a DataFrame in two ways, either by converting data already stored in Python into a Dataframe or by reading in an extrnal file. In the first instance, we can create a dataframe from a variety of structures, such as from lists or a list of lists or from zipped lists. In our case however we will use a dictionary of lists as follows using the `pd.DataFrame()` function:
#create a dictionary of listsdata_dict = {"Countries": ["England", "Scotland", "Wales", "Northern Ireland"],            "Capitals": ["London", "Edinburgh", "Cardiff", "Dublin"],            "Population (millions)": [55.98,  5.454, 3.136, 1.786]}#convert this to a dataframedata_df = pd.DataFrame(data_dict)#display the dataframedata_df
Or we can extract the data from an existing datasource such as a csv (comma separated variables) file using pd.read_csv as follows:
# Defines df as a data frame without adding an additional index column df = pd.read_csv("StarColours.csv")# Displays the data frame dfdf
Where a csv is a common data structure to use in conjunction with pandas, where each column is separated by a column as the name suggested. Of course, other data structures can also be used to be read in such as text files or excel files and various other parameters can be specified to allow these to be read in such as the delimiter (how columns are separated), the encoding (the sequence of characters), and compression (how the file is compressed). This can be used in conjunction with parameters such as use_cols where you can specify the columns to be read in or nrows which specifies how many rows you want to be read in.
Just as with a list, an array or dictionary we can also take a look at specific values, or ranges of values, from with the data structure. In the case of a DataFrame we can look at slices just as we would with a list or an array, but we can use the column names or the indexes to access this information.
In the case of the star colours dataset, we can select a specific column in a way similar to what we would for a dictionary value. The key here would be the column name as a string as follows:
# Returns the colour columndf["Color"]#alterntaively you can use dot notation#df.Color#but only if there are no spaces in the title#out:0               Blue1               Blue2               Blue3      Blue to White4    White to Yellow5      Orange to Red6                RedName: Color, dtype: object
We can take this even further by selecting a specific item, or items from the column, as we would in a dictionary of lists, by firstly accessing the column and then accessing the indices as follows:
# Selects seventh item in colour columndf["Color"][6]#out:'Red'
Or for a range:
# Selects 4th to 6th items in colour columndf["Color"][3:6]#out:3      Blue to White4    White to Yellow5      Orange to RedName: Color, dtype: object
We can further specify several columns by passing a list of column names as follows:
# Returns the specified columnsdf[["Color", "Main Characteristics", "Examples"]][3:6]
Alternatively we can access information from a DataFrame where a specific condition is met by using df.loc . This is where we can begin to use conditional statements to access information using the notation df.loc[df["columname"] > x] which will return all rows with a value greater than x for the column labeled columname . This can then be used in a conjunction with other conditionals as to either access or change information. For example:
# Returns all rows with values greater than #one for the average mass columndf.loc[df["Average Mass (The Sun = 1)"] > 1]
The benefit of using the DataFrame data structure is the variety of operations that can be performed on the data including the ability to perform mathematical operations on columns and across them, the creation of new columns to add back to the dataset and a host of inbuilt functionality as well.
For example, as we would with Numpy arrays, we can perform mathematical operations for each element across the column. For example we could add the mass average and the mass radius columns together:
# Adding the average mass and radius columns(df["Average Mass (The Sun = 1)"] + df["Average Radius (The Sun = 1)"])#out:0    75.01    25.02     5.73     3.04     2.25     1.76     0.7dtype: float64
Or we could calculate the average density of the stars based on average mass and average radius, and assign that back to a new column in the dataset:
# Defines a new average density column accordinglydf["Average Density"] = ((df["Average Mass (The Sun = 1)"]                          *3)                         /(4*                           np.pi*                           (df["Average Radius (The Sun = 1)"]                            **3)))# Displays the data framedf
Other than this, we have have multiple in built functions which we can use to look at the data or change it. For example, we can use df.head() to check the first five rows of the dataset, or we can use df.tail() to access the bottom five rows of the dataset:
# Returns first five rows from the data framedf.head()
We can also access all unique items from a column (i.e. no duplicates) as an array:
# Returns unique items from the colours columndf["Color"].unique()#out:array(['Blue', 'Blue to White', 'White to Yellow', 'Orange to Red', 'Red'],      dtype=object)
Or we can return a number of statistical paramaters for all columns with numerical values as:
# Returns unique items from the colours rowdf.describe()
Where we can also use inbuilt Numpy functions to access other statistical values individually such as using df.mean() to access the mean of the columns, df.median() to access the median of the columns, df.min() to access the minimum of the columns or df.max() to find the maximum of the columns. For example:
# Returns the mean of each numerical columndf.mean()Average Mass (The Sun = 1)      12.157143Average Radius (The Sun = 1)     4.028571Average Density                  0.261248dtype: float64
There is of course many more inbuilt functionalities built into the structure, including setting the index, converting the columns to arrays, lists and dictionaries, dropping a row or a column based on a condition, merging dataframes, dealing with NaN values and finally pushing the data to a csv file.
While this functionality is not covered here, it can be found in the workshop file here. Or, if you think you know all there is about pandas DataFrames, then why not try out problem sheet?
Dataset of stars comes from: https://www.enchantedlearning.com/subjects/astronomy/stars/startypes.shtml
The full workshop notebook, along with further examples and challenges, can be found HERE. If you want any further information on our society feel free to follow us on our socials:
Facebook: https://www.facebook.com/ucldata
Instagram: https://www.instagram.com/ucl.datasci/
LinkedIn: https://www.linkedin.com/company/ucldata/
And if you want to keep up date with stores from the UCL Data Science Society and other amazing authors, feel free to sign up to medium using my referral code below.