Investigating Numpy and Pandas the Delta Degree of Freedom
By: Shinichi Okada
Tags: Numpy, Dof, Programming, Data Science, Pandas

Data Science

Investigating Numpy and Pandas the Delta Degree of Freedom

Photo by ian dooley on Unsplash

Introduction

ddof means the Delta Degrees of Freedom. For example, when you calculate, a standard deviation, the divisor used in calculations is N - ddof, where N represents the number of elements. This ddof is also used in var, cov and other functions in Numpy and Pandas. In this article, I'd like to show how Numpy and Pandas have different default ddof values.

Sample data

Let’s create a sample data frame.

import pandas as pd

import numpy as np df = pd.DataFrame(

[

[7,3],

[6,4],

[5,4],

[3,2],

[6,4],

[8,9],

[9,7]

],

columns=['Set of A','Set of B']) # for numpy

x=df.iloc[:,0]

y=df.iloc[:,1]

Standard deviation

Numpy standard deviation

s_x=np.std(x)

s_x0=np.std(x,ddof=0)

s_x1=np.std(x,ddof=1)

print(f'Numpy std default: {s_x}')

print(f'Numpy std ddof=0: {s_x0}')

print(f'Numpy std ddof=1: {s_x1}')

Results:

Numpy std default: 1.8294640678379568

Numpy std ddof=0: 1.8294640678379568

Numpy std ddof=1: 1.9760470401187074

Pandas standard deviation

print(f'Pandas std default: {df.std()}')

print(f'Pandas std ddof=0: {df.std(ddof=0)}')

print(f'Pandas std ddof=1: {df.std(ddof=1)}')

Results:

Pandas std default: Set of A 1.976047

Set of B 2.429972

dtype: float64

Pandas std ddof=0: Set of A 1.829464

Set of B 2.249717

dtype: float64

Pandas std ddof=1: Set of A 1.976047

Set of B 2.429972

dtype: float64

The table of the std results

Variance

Numpy variance

var_n=np.var(x)

var_n0=np.var(x,ddof=0)

var_n1=np.var(x,ddof=1)

print(f'Numpy var default: {var_n}')

print(f'Numpy var ddof=0: {var_n0}')

print(f'Numpy std ddof=1: {var_n1}')

Results:

Numpy var default: 3.346938775510204

Numpy var ddof=0: 3.346938775510204

Numpy std ddof=1: 3.9047619047619047

Pandas variance

print(f'Pandas std default: {df.var()}')

print(f'Pandas std ddof=0: {df.var(ddof=0)}')

print(f'Pandas std ddof=1: {df.var(ddof=1)}')

Results:

Pandas std default: Set of A 3.904762

Set of B 5.904762

dtype: float64

Pandas std ddof=0: Set of A 3.346939

Set of B 5.061224

dtype: float64

Pandas std ddof=1: Set of A 3.904762

Set of B 5.904762

dtype: float64

The table of var results

covariance

Numpy covariance

cov_n=np.cov(x,y)[0][1]

cov_n0=np.cov(x,y,ddof=0)[0][1]

cov_n1=np.cov(x,y,ddof=1)[0][1]

print(f'Numpy cov default: {cov_n}')

print(f'Numpy cov ddof=0: {cov_n0}')

print(f'Numpy cov ddof=1: {cov_n1}')

Results:

Numpy cov default: 3.7619047619047614

Numpy cov ddof=0: 3.224489795918367

Numpy cov ddof=1: 3.7619047619047614

Pandas covariance

print(df.cov())

Results:

Set of A Set of B

Set of A 3.904762 3.761905

Set of B 3.761905 5.904762

Table of covariance results

Numpy functions using ddof

std, var, nanstd, nanvar, corrcoef, cov, describe, tvar, tstd, tsem, gstd, sem, chisquare, power_divergence, zmap, zscore.

Conclusion

Numpy has the default value of ddof=0 and Pandas has the default value of ddof=1. It is important to set the default value in order to avoid confusion.

Newsletter

If you like my article and would like to receive an occasional newsletter, please sign up.

Get full access to every story on Medium by becoming a member.

Reference