p-value Basics with Python Code. What is p-value? It is the probability... | by Sujeewa Kumaratunga PhD | Towards Data Science

What is p-value? It is the probability that you will obtain a test result given an actual distribution. Or in an A/B test setting, it is the probability that we measure something, like an average order value, given an initial hypothesis, like ‘we believe the average order value is $170’.
The p-value is answering the question with a certain confidence level. You hear people often say, “I am 90% confident I will get that job” or “I am 99.99% confident I can not sing like Freddie Mercury” or something like that involving confidence level. This is sort of the same thing, except it will be an actual quantitative number.
We saw in the Central Limit Theorem post how, if we draw samples enough times the distribution of those sample means will be a normal distribution with the mean of that normal distribution approaching the population mean. So then a natural question to ask is how likely is it that we would observe a certain sample mean.
Let’s look at an example.
Let’s assume, like before, the average order value of our company’s customers is $170. That is, our population mean is $170 (and the standard deviation is $5). So our hypothesis is that our entire customer base has an average order value of $170.
Now we want to test our hypothesis. In general we can not test our entire population, so we resort to testing many smaller samples of the population. Now if we draw 10,000 samples, then the distribution of those sample means will look like this :
So you see that any given sample’s mean could vary anywhere from $150 to $190.
Then, let’s say we took a sample group of people and get their mean order value and that turned out to be $183. And we want to know the probability that this could happen. But in statistics we ask, what is the probability that a sample mean could be $183 or above, given the hypothesis that the population mean is $170.
This can be calculated by counting all the numbers that are above 183 in the above plot and dividing it by the total number of sample draws, which is 10000. Use the code at the and with:
pvalue_101(170.0, 5.0, 10000, 183.0) 
Percentage of numbers larger than 183.0 is 0.35%. It is a tiny percentage, but it is not zero. It would be wrong for you to reject the hypothesis that the population mean is $170, since we clearly derived this sample mean from that population distribution.
And similarly if you wanted to ask what is the probability of getting a sample mean that is more than $13 different from the population mean of $170? That is what is the probability of getting a sample mean that is less than $157 or more than $183?
Percentage of numbers further than the population mean of 170.0 by +/-13.0 is 0.77%. You see this is about double the percentage that the sample mean could be just only larger than $183. This is because a normal distribution is symmetrical around the mean.
It is important to understand this small but non-zero probability. Even if draw samples from the exact same population, there is a non-zero chance that the sample mean will vary by quite a lot from our population mean. So, if we run the A/B test for just one day, which is the equivalent of drawing one sample, we can not make a decision on the population mean. What we can do is estimate the probability that we would get this sample mean, given this population.
Now, instead, assume you wanted to know the reverse — how far from the population mean do 95% of the sample means lie? That is when the 68–95–99.7 rule comes in handy.
It says:
We are 68.2% confident that if we draw many random samples, the sample means will be between μ+/- σ. in our case 170 +/- 5, i.e., 68.2% of the time sample means will be between $165 and $17595.4% of the time our sample means will be between μ+/- 2σ. i.e., 95.4% of the time our sample means will be between $160 and $180.
We are 68.2% confident that if we draw many random samples, the sample means will be between μ+/- σ. in our case 170 +/- 5, i.e., 68.2% of the time sample means will be between $165 and $175
95.4% of the time our sample means will be between μ+/- 2σ. i.e., 95.4% of the time our sample means will be between $160 and $180.
In business people often talk about p-value. The p-value is closely related to the above rule. p-value measures the probability that a sample mean would be a certain value or more, given the population mean and standard deviation.
The p-value gives us the probability of observing what we observed, given a hypothesis is true. It does not tell us the probability that the null hypothesis is true.
In our example,
A p-value of 0.35% will give the probability that we get a sample mean that is more than $183, given the hypothesis that the population mean is $170.A p-value of 0.77% will give the probability that we get a sample mean that is more than $183 or less than $157, given the hypothesis that the population mean is $170.It does not give us the probability of the hypothesis being true. In fact it would be dangerous to reject the hypothesis that the population mean is $170, because we clearly got the sample mean from a population mean of $170.
A p-value of 0.35% will give the probability that we get a sample mean that is more than $183, given the hypothesis that the population mean is $170.
A p-value of 0.77% will give the probability that we get a sample mean that is more than $183 or less than $157, given the hypothesis that the population mean is $170.
It does not give us the probability of the hypothesis being true. In fact it would be dangerous to reject the hypothesis that the population mean is $170, because we clearly got the sample mean from a population mean of $170.
Often people would start an experiment saying they would want a 95% confidence level, which means they are expecting a p-value of 5% (which comes from 100%-95%). Then they would take the above sample mean of $183, take its p-value of 0.35% and say: “Since 0.35%<0.5%, our sample mean is further in the tails than the 5% that we have allowed. So we reject the hypothesis that the population mean is $170”. We know this is a wrong conclusion, because we used a population mean of $170 to generate some, albeit few, sample means above $183 (see graph above).
You can say nothing about if the hypothesis is true or not. In fact you can not even reject the hypothesis just with this one data point.
It is a difficult lesson to drive home. But in general, in life, we are not able to tell the probability that any hypothesis is true — all we can say is something about the probability of an observation given someone’s hypothesis.
You can generate the plots and p-values in this post with the following Python code.
def pvalue_101(mu, sigma, samp_size, samp_mean=0, deltam=0):    np.random.seed(1234)    s1 = np.random.normal(mu, sigma, samp_size)    if samp_mean > 0:        print(len(s1[s1>samp_mean]))        outliers = float(len(s1[s1>samp_mean])*100)/float(len(s1))        print('Percentage of numbers larger than {} is {}%'.format(samp_mean, outliers))    if deltam == 0:        deltam = abs(mu-samp_mean)    if deltam > 0 :        outliers = (float(len(s1[s1>(mu+deltam)]))                    +float(len(s1[s1<(mu-deltam)])))*100.0/float(len(s1))        print('Percentage of numbers further than the population mean of {} by +/-{} is {}%'.format(mu, deltam, outliers))    fig, ax = plt.subplots(figsize=(8,8))    fig.suptitle('Normal Distribution: population_mean={}'.format(mu) )    plt.hist(s1)    plt.axvline(x=mu+deltam, color='red')    plt.axvline(x=mu-deltam, color='green')    plt.show()