Repository: scipy/scipy
Pull request: DOC: stats: correct conclusion associated with high p-value
#### Reference issue
Closes gh-4952

#### What does this implement/fix?
gh-4952 noted that the [stats tutorial](https://scipy.github.io/devdocs/tutorial/stats/probability_distributions.html#subclassing-rv-discrete) includes the language:
> The pvalue in this case is high, so we can be quite confident that our random sample was actually generated by the distribution

which is not correct.
This PR summarizes what the test statistic and p-value mean in this example.

#### Additional information
There are other suggestions in gh-4952, but I think they have been mostly addressed over the years - at least enough to close the specific issue. I couldn't find broken links, and there is a lot more information about interpreting statistics and p-values in https://docs.scipy.org/doc/scipy/tutorial/stats/hypothesis_tests.html, especially https://docs.scipy.org/doc/scipy/tutorial/stats/resampling.html.


Problem statement:
DOC: stats: improvement suggestions for tutorial
First of all, I really enjoy reading through the stats tutorial, for the most part it makes a lot of sense.  However when it starts to get into detail I start having problems.  I guess I have a general lack of confidence when it comes to stats, always preferring mechanics.

Thus for me (and perhaps others) would benefit from either a bit more explanation and/or examples and/or a cleanup of the terminology used. For example I stumble reading the `Subclassing rv_discrete` and `T-test and KS-test` sections:
## Subclassing rv_discrete

```
The pvalue in this case is high, so we can be quite confident that our random sample was
actually generated by the distribution
```

Conclusion: a high pvalue means we feel confident about the result.
## T-test and KS-test

```
The pvalue is 0.7, this means that with an alpha error of, for example, 10%, we cannot
reject the hypothesis that...
```

Conclusion: a high pvalue means we **don't** feel confident about the result.

---

Do you see what I mean?  Somewhere there I feel a bit of logic goes missing.  It explains `Why we test`, but it doesn't fully explain `What we test`, which hypothesis do we currently test in the current example?

Also links to wikipedia would help and perhaps a brief (5-10 words) summary to explain the underlying concept of the methods used.

Also other problems, the source links give 404 page not found errors, and the inline source could look a little tidier, these problems though I can help with and will submit a PR as/when I find some time.

Thanks,
OceanWolf

Hints from issue discussion:
Docs improvements are always welcome

Commit message: Update doc/source/tutorial/stats/probability_distributions.rst

[docs only]

Co-authored-by: Matt Haberland <mhaberla@calpoly.edu>

Patch to review:
diff --git a/doc/source/tutorial/stats/probability_distributions.rst b/doc/source/tutorial/stats/probability_distributions.rst
index e4dcf8034b7b..c08aacc628db 100644
--- a/doc/source/tutorial/stats/probability_distributions.rst
+++ b/doc/source/tutorial/stats/probability_distributions.rst
@@ -602,11 +602,11 @@ the probabilities.
    :include-source: 0
 
 
-Next, we can test whether our sample was generated by our norm-discrete
-distribution. This also verifies whether the random numbers were generated
-correctly.
+Next, we can use a chi-squared test, `stats.chisquare`, to test the null
+hypothesis that the sample is distributed according to our norm-discrete
+distribution.
 
-The chisquare test requires that there are a minimum number of observations
+The test requires that there are a minimum number of observations
 in each bin. We combine the tail bins into larger bins so that they contain
 enough observations.
 
@@ -617,5 +617,12 @@ enough observations.
     >>> print('chisquare for normdiscrete: chi2 = %6.3f pvalue = %6.4f' % (ch2, pval))
     chisquare for normdiscrete: chi2 = 12.466 pvalue = 0.4090  # random
 
-The pvalue in this case is high, so we can be quite confident that
-our random sample was actually generated by the distribution.
+Conceptually, the test statistic ``chi2`` is sensitive to deviations between
+the frequencies of observations and their expected frequencies under the
+null hypothesis. The p-value is the probability of drawing samples from the
+hypothesized distribution that would produce a statistic value more extreme than
+the one we observed. Our statistic value is not very high; in fact, there is
+a 40.9% chance that the statistic would be higher than 12.466 if we were to draw a
+sample of the same size from the discrete distribution defined by ``p2``. Therefore,
+the test provides little evidence against the null hypothesis that the sample was
+drawn from our norm-discrete distribution.


Code review comments:
diff_hunk: @@ -602,11 +602,11 @@ the probabilities.
    :include-source: 0
 
 
-Next, we can test whether our sample was generated by our norm-discrete
-distribution. This also verifies whether the random numbers were generated
-correctly.
+Next, we can use a chi-squared test, `stats.chisquare`, to test the null
path: doc/source/tutorial/stats/probability_distributions.rst
text: @user1:
Just wanted to check whether this is intended as a clickable link?
```suggestion
Next, we can use a chi-squared test, :func:`scipy.stats.chisquare`, to test the null
```

@author:
Yes, thanks!