Statistical Functions of Pandas
By: Soner Yıldırım
Tags: Machine Learning, Artificial Intelligence, Data Analysis, Data Science, Pandas

Statistical Functions of Pandas

Understand your data well.

Photo by Stephen Dawson on Unsplash

Pandas is a widely-used Python library for data manipulation and analysis. It provides numerous methods and functions that expedite data analysis and preprocessing steps. On top of that, pandas also provides statistical functions which can be used to further understand the data. In this post, we will start with simple descriptive statistics and then introduce a few more complex statistical functions of pandas. As always, we import numpy and pandas:

import pandas as pd

import numpy as np

Basic Descriptive Statistics

Let’s start with the very basic descriptive statistics. We create a pandas series and calculate mean, median, standard deviation, and variance:

s = pd.Series([1,2,4,5,8,9,13,16,22,23,43,50,51]) print("Mean of s is {}".format(s.mean()))

print("Median of s is {}".format(s.median()))

print("Standar deviation of s is {}".format(s.std()))

print("Variance of s is {}".format(s.var()))

Mean is the average of the values in series.

is the average of the values in series. Median is the value in the middle when values are sorted in ascending or descending order.

is the value in the middle when values are sorted in ascending or descending order. Standard deviation is a measure of how spread out the values are. To be more specific, it is the square root of variance.

is a measure of how spread out the values are. To be more specific, it is the square root of variance. Variance: A measure of the variation among values. It is calculated by adding up squared differences of each value and the mean and then dividing the sum by the number of samples.

Covariance

We can also calculate the covariance of two series which is the joint variability of two random variables. So, covariance describes how similarly two random variables deviate from their mean. The random variable we use here are pandas series.

s1 = pd.Series(np.random.randint(20, size=10))

s2 = pd.Series(np.random.randint(20, size=10)) s1.cov(s2)

-5.6111

We can plot these series using a scatter plot and try to understand the meaning of the variance of -5.61. I will also mark the point that indicates the mean of both series. Let’s first calculate the means:

Then plot the values in s1 and s2 as well as the point indicating the mean:

plt.figure(figsize=(8,5))

plt.xlabel('s1')

plt.ylabel('s2')

plt.title('Covariance of s1 and s2')

plt.grid(True)

plt.scatter(s1, s2)

plt.scatter(8.5, 9.3) #mean of s1 and s2

The red lines show the means of series. The mean of s1 is the vertical line (x=8.5) and the mean of s2 is the horizontol line (y=9.3). Deviation from the mean is the difference between the values and the mean. Covariance is proportional to the product of deviation of s1 and s2 values. Consider the upper right rectangle in the plot above. Both s1 and s2 values are higher than the mean of s1 and s2, respectively. So, deviations are positive. When we multiply two positive values, we get a positive value. In the lower left rectangle, s1 and s2 values are lower than the mean of s1 and s2, respectively. Thus, deviations are negative but we get a positive number when two negative numbers are multiplied. For the points in lower right and upper left rectangle areas, deviations of s1 is positive when the deviation of s2 is negative and vice versa. So we get a negative number when two deviations are multiplied.

All the deviations are combined to get the covariance. Hence, if we have more points in negative regions than positive regions, we will get a negative covariance. Negative covariance indicates as the values in one variable increase, the values in the other variable tend to decrease. It is quite similar to correlation.