Using Standard Deviation in Python | by Reza Rajabi | Towards Data Science

As this article mentioned, with Standard Deviation you can get a handle on whether your data are close to the average or they are spread out over a wide range. For example, if an employer wants to determine if the salaries in one of his departments seem fair for all employees, or if there is a great disparity, he can use standard deviation. To do that, he can find the average of the salaries in that department and then calculate the standard deviation. One scenario could be like the following; He finds that the standard deviation is slightly higher than he expected, he examines the data further and finds that while most employees fall within a similar pay bracket, four loyal employees who have been in the department for 15 years or more, far longer than the others, are making far more due to their longevity with the company.
In general, a low standard deviation means that the data is very closely related to the average, thus very reliable and a high standard deviation means that there is a large variance between the data and the statistical average, thus not as reliable.
One of the most important applications of standard deviation is in comparing two datasets. If two datasets have the same average, it does not mean necessarily they are exactly the same, right? For example, the datasets 199, 200, 201 and 0, 200, 400 both have the same average (200), however the first dataset has a very small standard deviation (s=1) compared to the second dataset (s=200).
A population dataset contains all members of a specified group (the entire list of possible data values). For example, the population may be “ALL people living in Canada”. A sample dataset contains a part, or a subset, of a population. The size of a sample is always less than the size of the population from which it is taken. For example, the sample may be “SOME people living in Canada”.
We are normally interested in knowing the population standard deviation as it contains all the values we want to analyze. We would normally calculate the population standard deviation if we have the entire population or we have a sample of a larger population, but we do not want to generalize our findings to the population. In many cases, we have a sample of data from which we would like to generalize the analysis to a population though. In general, if you have only a sample of data and you want to make a statement about the population standard deviation from which the sample is drawn, you need to use the sample standard deviation. In this article, I will focus on the population standard deviation. The population standard deviation formula is the following:
For example, if our dataset is [13, 22, 26, 38, 36, 42,49, 50, 77, 81, 98, 110], the population mean or average will be: Summation of all individual items in the dataset divided by the number of items, and the result will be 53.5.
Now, subtract the mean from every item in the set and square the difference between each number and the mean. For example, for the first item, 13, we have:
(13–53.5)=(-40.5) and square of (-40.5) will be: 1,640.25
Then, sum all the squared differences (10,581)and divide this sum by the number of items. The result will be 881.75‬. This number is called the variance. Take the square root of the variance to find the standard deviation. So the standard deviation of this dataset will be 29.69.
The population mean and standard deviation of a dataset can be calculated using Numpy library in Python. The following code shows the work:
import numpy as npdataset=[13, 22, 26, 38, 36, 42,49, 50, 77, 81, 98, 110]print('Mean:', np.mean(dataset))print('Standard Deviation:', np.std(dataset))Mean:53.5Standard Deviation: 29.694275542602483
Two datasets below show the high temperatures (in degrees Fahrenheit) for two cities during a 15-day period. We want to compare the mean and standard deviation for the temperatures of this two cities.
City A= 36, 37, 36, 34, 39, 33, 30, 30, 32, 31, 31, 32, 32, 33, 35City B= 41, 35, 28, 29, 25, 36, 36, 32, 38, 40, 40, 34, 31, 28, 30
The mean temperature of two cities are 33.4 for City A and 33.5 for City B, which are very close to each other. Both cities are somehow cold. However, the standard deviation of City A is 2.60 far less than City B with 4.83. This indicates that temperatures in City A are more consistent than City B. In other words, temperatures in City B varies more from day to day.
To show this variation in a graph, I use error bar in Python. Error bars are useful to problem solvers because they show the confidence or precision in a set of calculated values. The mean and the standard deviation can be depicted using matlibplot library in Python. As the following figure shows, City A has less variation compared to City B.
Now I will explain how I created the error bar.
First of all, after importing the libraries, I calculate the mean and the standard deviation for both datasets:
City_A=[36,37,36,34,39,33,30,30,32,31,31,32,32,33,35] City_B=[41,35,28,29,25,36,36,32,38,40,40,34,31,28,30] Mean_City_A=np.mean(City_A)Mean_City_B=np.mean(City_B) STDV_City_A=np.std(City_A)STDV_City_B=np.std(City_B)
Then, I create a figure object and set the axis labels:
# Create a figure with customized sizefig = plt.figure(figsize=(10, 8))ax = fig.add_subplot(111) # Set the axis lablesax.set_xlabel('Day', fontsize = 18)ax.set_ylabel('Temperature', fontsize = 18)
We need to assign days (15-day period) to X axis:
# X axis is day numbers from 1 to 15xaxis = np.array(range(1,16))
As we are creating a error bar over lines, we can set the color and line setting of this graph as well:
# Line color for error barcolor_City_A = 'red'color_City_B = 'darkgreen' # Line style for each datasetlineStyle_City_A={"linestyle":"--", "linewidth":2, "markeredgewidth":2, "elinewidth":2, "capsize":3}lineStyle_City_B={"linestyle":"-", "linewidth":2, "markeredgewidth":2, "elinewidth":2, "capsize":3}
Now we create two error bars with each dataset items, the standard deviation, and the line styles we set.
# Create an error bar for each datasetline_City_A=ax.errorbar(xaxis, City_A, yerr=STDV_City_A, **lineStyle_City_A, color=color_City_A, label='City A')line_City_B=ax.errorbar(xaxis, City_B, yerr=STDV_City_B, **lineStyle_City_B, color=color_City_B, label='City B')
I would like to show the numbers on each line, that’s why, I use annotate function to add text to the points. xy in the function shows the position of numbers on the graph, and xytext shows the position of labels.
# Label each dataset on the graph, xytext is the label's position for i, txt in enumerate(City_A):        ax.annotate(txt, xy=(xaxis[i], City_A[i]),              xytext=(xaxis[i]+0.03,City_A[i]+0.3),             color=color_City_A) for i, txt in enumerate(City_B):             ax.annotate(txt, xy=(xaxis[i], City_B[i]),             xytext=(xaxis[i]+0.03,            City_B[i]+0.3),color=color_City_B)
Also, I will draw a legend for the graph, along with other settings about legend’s style, axis font style, and adding a grey grid to the graph:
# Customize the legend font and handle lengthparams = {'legend.fontsize': 13,          'legend.handlelength': 2}plt.rcParams.update(params) # Customize the fontfont = {'family' : 'Arial', 'weight':'bold','size': 12}matplotlib.rc('font', **font) # Draw a grid for the graph and set face color to the graphax.grid(color='lightgrey', linestyle='-')ax.set_facecolor('w')
Finally, the error bar is shown using plt.show() command.
Below find the Python code:
Thank you for reading this article!