Pandas DataFrame: A lightweight Intro | by Deepak K Gupta (DAKSH) | Towards Data Science

Pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.
The Data Structures provided by Pandas are of two distinct types
Pandas DataFrame &Pandas Series
Pandas DataFrame &
Pandas Series
We’ll look in Pandas DataFrame in this post.
The general trend in learning Pandas (including the official documentation) is to get into Pandas Series first followed by Pandas DataFrame. However, after using Pandas for considerable duration, I’m convinced that we should start with Pandas DataFrame. The concept and rationale behind Pandas Series becomes clear and easier to understand once we’re comfortable with Pandas DataFrame.
Here is how the Technical Definition looks like
Pandas DataFrame is a 2-D labeled data structure with columns of potentially different type.
And here is how you should understand it
Pandas DataFrame is nothing but an in-memory representation of an excel sheet via Python programming language
So, Pandas DataFrame is similar to excel sheet and looks like this
Just like excel, Pandas DataFrame provides various functionalities to analyze, change, and extract valuable information from the given dataset.
In the real world, a Panda DataFrame will be created by loading the datasets from persistent storage, including but not limited to excel, csv and MySQL database.
However, to help you understand it better, I’ll be using Python Data Structures (Dictionary and list) over here.
As depicted in excel sheet above, if we consider column names as “Keys” and list of items under that column as “Values”, we can easily use a python dictionary to represent the same as
my_dict = {      'name' : ["a", "b", "c", "d", "e","f", "g"],     'age' : [20,27, 35, 55, 18, 21, 35],     'designation': ["VP", "CEO", "CFO", "VP", "VP", "CEO", "MD"]}
We can create a Pandas DataFrame out of this dictionary as
import Pandas as pddf = pd.DataFrame(my_dict)
The resultant DataFrame shall look similar to what we’ve seen in the excel sheet above as
There are chances that the Columns are not in sequence as defined in the dictionary because python implements dictionary as hash and doesn’t guarantee to preserve the sequence.
Since, we haven’t provided any Row Index values to the DataFrame, it automatically generates a sequence (0...6) as row index.
To provide our own row index, we need to pass index parameter in the DataFrame(...) function as
df = pd.DataFrame(my_dict, index=[1,2,3,4,5,6,7])
The index need not be numerical all the time, we can pass strings also as index. For example
df = pd.DataFrame(     my_dict,      index=["First", "Second", "Third", "Fourth", "Fifth", "Sixth", "Seventh"])
As you might have guessed that Index are homogeneous in nature which means we can also use NumPy arrays as Index.
np_arr = np.array([10,20,30,40,50,60,70])df = pd.DataFrame(my_dict, index=np_arr)
Unlike python lists or dictionaries and just like NumPy, a column of the DataFrame will always be of same type.
We can check the data type of a column either using dictionary like syntax or by adding the column name using DataFrame .
df['age'].dtype    # Dict Like Syntaxdf.age.dtype       # DataFrame.ColumnNamedf.name.dtype      # DataFrame.ColumnName
If we want to check the data types of all columns inside the DataFrame, we’ll use the dtypes function of the DataFrame as
df.dtypes      
It will display the type of all columns as
At any point of time, Pandas DataFrame will contain hundreds (if not thousands) of rows of data. We can only view them selectively at any point of time.
To selectively view the rows, we can use head(...) and tail(...) functions, which by default give first or last five rows (if no input is provided), otherwise shows specific number of rows from top or bottom
Here is how it displays the contents
df.head()   # Displays 1st Five Rows
df.tail()    # Displays last Five Rows
df.head(2)   # Displays 1st two Rows
df.tail(7)     # Displays last 7 Rows
This was all about data, however, what if we want to see Row Indexes and Columns name. Pandas DataFrame provides specific functions for viewing them
df.index     # For Row Indexesdf.columns   # For Columns
The Pandas DataFrame provides various column helper functions which is extremely useful for extracting valuable information from the column. Some of these are
unique → Provide unique elements from a column by removing duplicates. For example
df.designation.unique()
mean → Provided the mean value of all the items in the column. For example
df.age.mean()
Most of the time, the given datasets already contains a row index. In those cases, we don’t need Pandas DataFrame to generate a separate row index.
Not only its just a redundant information but also takes unnecessary amount of memory.
Pandas DataFrame allows setting any existing column or set of columns as Row Index. Here is how can use the columns of the Pandas DataFrame created earlier(using dictionary my_dict )
df = pd.DataFrame(my_dict)df.set_index("name")
df.set_index("age")
And we can set multiple columns as index by passing a list inside set_index(...) as
df.set_index(["name","age"])
Any data analytics activity requires data cleanup and its quite possible that we come to a conclusion to exclude some columns from the datasets that needs to be analyzed.
This not only saves memory but also help to analyze the data which is of interest
We’ll use the same dictionary for loading Pandas DataFrame, but this time we will specify the columns which will be part of DataFrame
df = pd.DataFrame(my_dict, columns=["name", "age"])
Pandas DataFrame provides multiple ways of deleting the rows and columns. There is no functional penalty for choosing one over another. You may use whatever syntax you’re comfortable with.
Using Dictionary Syntax → To remove a Column, we’ll use del as
del df['name']
Using Drop function → Allows us to delete Columns as well as Rows
Whether we’re going to delete Rows or Columns is decided by the second argument in the drop function.
The second argument “1” in function drop(...) denotes deletion of the “Column”, whereas “0” means deletion of the “Row
# Delete Column "age"df.drop('age',1)# Delete the Row with Index "3"df.drop(3,0)
We can also delete multiple Rows and Columns by passing the list in drop(...) function
# Delete Columns "name" & "age"df.drop(['name','age'],1)# Delete Rows with index "2","3", & "4"df.drop([2,3,4],0)
Note: The Row index are not index numbers, but the row(s) containing that value
What if we get a dataset without any columns?
Well, Pandas DataFrame don’t mind that and generates the DataFrame by implicitly adding Row Index as well as Column headers for us.
For example, if we create a DataFrame from the list below
my_list = [[1,2,3,4],           [5,6,7,8],           [9,10,11,12],           [13,14,15,16],           [17,18,19,20]]df = pd.DataFrame(my_list)
It will look like this
If we don’t want Pandas DataFrame to auto generate the Row Indexes and Column names, we can pass those in DataFrame function as
df = pd.DataFrame(my_list, index = ["1->", "2->", "3->", "4->", "5->"], columns = ["A", "B", "C", "D"])
And here is how it will look like
It should be noted that we can also create a Pandas DataFrame from NumPy arrays as
np_arr = np.array([[1,2,3,4],                   [5,6,7,8],                   [9,10,11,12],                   [13,15,16,16],                   [17,18,19,20]])df = pd.DataFrame(np_arr)
Just like Excel Sheets, Pandas DataFrame is at ease with doing mathematical operations on complete DataFrame.
It’s possible to do these operations in a Single Column and this is where Pandas Series come into picture.
We’ll look into Pandas Series in next post, let’s first see how mathematical operations work on complete DataFrame
Multiplications → We can multiply DataFrame with a scalar value of with another DataFrame
df * df
df * 10
Addition / Subtraction → Similar to Multiplication above, a DataFrame can be added / subtracted with a scalar value or with another DataFrame
df + 100
BitWise Operation → It’s possible to do BitWise operations ( AND(&), OR (|)) etc to complete DataFrame
df & 0
A series in nothing but a single column of a Pandas DataFrame.
Whenever, we have more than one series to work with, we have DataFrames.
Alternatively
A Pandas DataFrame is a collection of Pandas Series
I’ll talk about Panda Series in next post(s)
Thanks for Reading ...!!!!