Python Pandas: A Complete Guide for Beginners
By: Rahul Raj
Tags: Machine Learning, Pandas, Python, AI, Data Science

Pandas is a python library used in data manipulation ( create, delete, and update the data).

It is one of the most commonly used libraries for data analysis in python. Pandas offer data structures and operations for manipulating numerical and time-series data. Pandas is the go-to library when it comes to Python

[pandas] is derived from the term “panel data”, an econometrics term for data sets that include observations over multiple time periods for the same individuals. — Wikipedia

Pandas Use Cases

Calculate statistics and answer questions about the data (Mean, Median, Mode)

Check correlation between Two datasets

See the distribution of values of a dataset

Clean the data by removing missing values or filtering rows and columns by some criteria

Visualize the data with help from Matplotlib. Plot bars, lines, histograms, bubbles, and more.

Store the cleaned, transformed data back into a CSV, another file.

Rise of Popularity of Pandas:https://theatlas.com/charts/rJ9sZ5syf

Pandas Setup

Pandas is an easy package to install. Open up your terminal program (for Mac users) or command line (for PC users) and install it using either of the following commands:

conda install pandas

OR

pip install pandas

import pandas as pd

Reading Data

In pandas, we have a function name as read_csv which is used to read .csv files.Similarly to import a JSON file we use read_json .

import pandas as pd

df = pd.read_csv('data.csv')

#data.csv is the relative filepath for data df = pd.read_json(‘data.json’)

DataFrame operations

DataFrames possess hundreds of methods and other operations that are crucial to any analysis. As a beginner, you should know the operations that perform simple transformations of your data and those that provide fundamental statistical analysis.

df = pd.read_csv("Data.csv", index_col="Heading")

Viewing your data

The first thing to do when opening a new dataset is to print out a few rows to keep as a visual reference. We accomplish this with .head() :

df.head()

.head() outputs the first five rows of your DataFrame by default, but we could also pass a number as well: df.head(10) would output the top ten rows, for example.

To see the last five rows use .tail() . tail() also accepts a number, and in this case, we print the bottom five rows.:

df.tail(5)

Get the info of your data

.info() should be one of the very first commands you run after loading your data:

df.info()

OUT:

<class 'pandas.core.frame.DataFrame'>

RangeIndex: 100 entries, 0 to 99

Data columns (total 2 columns):

0 100 non-null float64

1 100 non-null float64

dtypes: float64(2)

memory usage: 1.6 KB

.info() provides the essential details about your dataset, such as the number of rows and columns, the number of non-null values, what type of data is in each column, and how much memory your DataFrame is using.

Calling .info() will quickly point out that the column you thought was all integers are actually strung objects.

Another fast and useful attribute is .shape , which outputs just a tuple of (rows, columns):

df.shape (100, 2)

Note that .shape has no parentheses and is a simple tuple of format (rows, columns). So we have 1000 rows and 2 columns in our movies DataFrame.

Column cleanup

Many times datasets will have verbose column names with symbols, upper and lowercase words, spaces, and typos. To make selecting data by column name easier we can spend a little time cleaning up their names.

Here’s how to print the column names of our dataset:

X= -2 * np.random.rand(100,2)

Y = 1 + 2 * np.random.rand(100,2)

df=pd.DataFrame(X,Y)

df.columns={

"RANDOM X","RANDOM Y"} print(df.columns) Index(['RANDOM Y', 'RANDOM X'], dtype='object')

Missing values

When exploring data, you’ll most likely encounter missing or null values, which are essentially placeholders for non-existent values. Most commonly you’ll see Python’s None or NumPy's np.nan , each of which is handled differently in some situations.

df.isnull().sum() RANDOM Y 0

RANDOM X 0

dtype: int64

.isnull() just by itself isn't very useful, and is usually used in conjunction with other methods, like sum() .

df.dropna(axis=1)

You can drop columns with null values by setting axis=0 or rows by setting axis=1

Understanding your variables

Using describe() on an entire DataFrame we can get a summary of the distribution of continuous variables:

df.describe()

Output

Relationships between variables

By using the correlation method .corr() we can generate the relationship between each continuous variable:

df.corr()

Corr between X and Y

Extracting Data

By column

You already saw how to extract a column using square brackets like this:

X_col = df[‘RandomX’]

type(X_col)

pandas.core.series.Series

By Rows

For rows, we have two options:

.loc - loc ates by name

- ates by name .iloc - locates by numerical index

Conditional Selection

df[df[‘RANDOM X’] >= -10].head(3)

X≥-10

Plotting

Another great thing about pandas is that it integrates with Matplotlib, so you get the ability to plot directly off DataFrames and Series. To get started we need to import Matplotlib ( pip install matplotlib ):

import matplotlib.pyplot as plt



plt.rcParams.update({‘font.size’: 20, ‘figure.figsize’: (10, 8)}) # set font and plot size to be larger

df.plot(kind='scatter', x='RANDOM X', y='RANDOM Y', title='X vs Y')

Scatter Plot

df[‘RANDOM X’].plot(kind=’hist’, title=’X’)

Histogram of X

Using a Boxplot we can visualize this data:

df[‘RANDOM X’].plot(kind=’box’)

That’s It!

Exploring, cleaning, transforming, and visualization data with pandas in Python is an essential skill in data science.

Just cleaning wrangling data is 50% of your job as a Data Scientist. Thanks for reading!

Found this article useful? Follow me (Rahula Raj) on Medium and check out my most popular articles below! Please 👏 this article to share it!