Repository: scipy/scipy

Issue:
BUG: Tippett’s and Pearson’s method for combine_pvalues are not monotonous
### Describe your issue.

Something is wrong with the methods `pearson` and `tippett` for `scipy.stats.combine_pvalues`. They should be monotonically increasing with all components of the input, yet they are monotonically decreasing. At first glance, the output seems to be 1 − _p_ when it should be _p._ It’s also straightforward to compute that Tippett’s method for an input of (½,½) should yield _p_ = ¾ but it does not.

@rlucas7 IIUC, you should know best about this.

### Reproducing Code Example

```python
import numpy as np
from scipy.stats import combine_pvalues

n = 5
ps = np.linspace(0.1,0.9,5)

for method in ("pearson","tippett"):
	combined_ps = [
			combine_pvalues( np.full(n,p), method )[1]
			for p in ps
		]
	assert np.all( np.diff(combined_ps) >= 0 )

assert combine_pvalues([0.5,0.5],method="tippett") == 0.75
```


### Error message

All assertions fail.

### SciPy/NumPy/Python version information

1.6.3 1.19.5 sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0)
BUG: Tippett’s and Pearson’s method for combine_pvalues are not monotonous
### Describe your issue.

Something is wrong with the methods `pearson` and `tippett` for `scipy.stats.combine_pvalues`. They should be monotonically increasing with all components of the input, yet they are monotonically decreasing. At first glance, the output seems to be 1 − _p_ when it should be _p._ It’s also straightforward to compute that Tippett’s method for an input of (½,½) should yield _p_ = ¾ but it does not.

@rlucas7 IIUC, you should know best about this.

### Reproducing Code Example

```python
import numpy as np
from scipy.stats import combine_pvalues

n = 5
ps = np.linspace(0.1,0.9,5)

for method in ("pearson","tippett"):
	combined_ps = [
			combine_pvalues( np.full(n,p), method )[1]
			for p in ps
		]
	assert np.all( np.diff(combined_ps) >= 0 )

assert combine_pvalues([0.5,0.5],method="tippett") == 0.75
```


### Error message

All assertions fail.

### SciPy/NumPy/Python version information

1.6.3 1.19.5 sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0)


Discussion:
According to https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.combine_pvalues.html, the "meta"-p-value of Tippet's method is `min(p_1, ..., p_N)`, Pearson's approach is `-2 sum_i log(1-p_i)`, see e.g., Heard, N. and Rubin-Delanchey, P. “Choosing between methods of combining p-values.” Biometrika 105.1 (2018): 239-246.

Why do you think the p-value should be 0.75 if method='tippett`?
@chrisb83 What I learned as Tippet's method defines the meta-p-value as `1 - (1 - min(p_1, ..., p_N))^N`. This is also the definition used by R's [Tippet's Method Package](https://search.r-project.org/CRAN/refmans/poolr/html/tippett.html), which cites the same 1931 book as the paper by Heard and Rubin-Delanchy. With this definition, we should expect the result for @Wrzlprmft's example to be 0.75.

Defined as above, the Tippet's meta-p-value is the probability of observing a minimum p-value among the N at least as small, assuming that the N p-values were generated iid from U(0, 1) (the uniform distribution with support [0 ,1]). It's possible that there were multiple methods discussed in Tippet's 1931 book, but I think it's more likely that this actually is a mistake in Heard and Rubin-Delanchy's paper, since I don't think there's any good theoretical justification for taking the min of the p-values as the meta-p-value. It is essentially the jellybeans method from [XKCD 882 ](https://xkcd.com/882/). 
Curiously, Heard and Rubin-Delanchy worked out that their version of Tippet's method is unsound, but didn't put together that they may have had the incorrect formula.

From [page 6 ](https://arxiv.org/pdf/1707.06897.pdf).

![image](https://user-images.githubusercontent.com/1953382/149042312-1b34b483-4bb1-4348-a899-4400e54eb655.png)

@Wrzlprmft thanks for pointing these out. I checked and your observation that SciPy’s Pearson p-value is actually 1 - p turns out to be correct. See for instance [this 2009 paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2771157/) by Won et al.

![image](https://user-images.githubusercontent.com/1953382/149048683-0ced55cf-5507-47a4-b2b7-3b1925bc76e7.png)

SciPy's implementation doesn't reverse the inequality in Pearson's method from that used in Fisher's method.

```python
    if method == 'fisher':
        statistic = -2 * np.sum(np.log(pvalues))
        pval = distributions.chi2.sf(statistic, 2 * len(pvalues))
    elif method == 'pearson':
        statistic = -2 * np.sum(np.log1p(-pvalues))
        pval = distributions.chi2.sf(statistic, 2 * len(pvalues))
```


clearly a bug

monotonicity in tippet goes in the wrong direction

Based on the example I guess the `1 - (1 - min(p_1, ..., p_N))^N` transformation is the same as `beta.cdf(min(pvalues), 1, n)`
with cdf instead of sf as steppi pointed out.

`1 - (1 - min(p_1, ..., p_N))^N`  looks like the Sidak multiple testing correction to me applied to the smallest p-value

> I think it's more likely that this actually is a mistake in Heard and Rubin-Delanchy's paper, since I don't think there's any good theoretical justification for taking the min of the p-values as the meta-p-value.

They don't do that, at least not skimming the definitions
min(pvalues) is the "test statistic", the actual p-value needs to come from the distribution of the test statistic, which is almost right in scipy except that's the complement, not left tail.

(based on a bit of fast skimming of small parts of articles)


> They don't do that, at least not skimming the definitions
> min(pvalues) is the "test statistic", the actual p-value needs to come from the distribution of the test statistic, which is almost right in scipy except that's the complement, not left tail.

You are correct. I skimmed too carelessly :)

Update: and proposition 2 that I posted a picture of but clearly didn’t read carefully is actually a pretty neat result 
According to https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.combine_pvalues.html, the "meta"-p-value of Tippet's method is `min(p_1, ..., p_N)`, Pearson's approach is `-2 sum_i log(1-p_i)`, see e.g., Heard, N. and Rubin-Delanchey, P. “Choosing between methods of combining p-values.” Biometrika 105.1 (2018): 239-246.

Why do you think the p-value should be 0.75 if method='tippett`?
@chrisb83 What I learned as Tippet's method defines the meta-p-value as `1 - (1 - min(p_1, ..., p_N))^N`. This is also the definition used by R's [Tippet's Method Package](https://search.r-project.org/CRAN/refmans/poolr/html/tippett.html), which cites the same 1931 book as the paper by Heard and Rubin-Delanchy. With this definition, we should expect the result for @Wrzlprmft's example to be 0.75.

Defined as above, the Tippet's meta-p-value is the probability of observing a minimum p-value among the N at least as small, assuming that the N p-values were generated iid from U(0, 1) (the uniform distribution with support [0 ,1]). It's possible that there were multiple methods discussed in Tippet's 1931 book, but I think it's more likely that this actually is a mistake in Heard and Rubin-Delanchy's paper, since I don't think there's any good theoretical justification for taking the min of the p-values as the meta-p-value. It is essentially the jellybeans method from [XKCD 882 ](https://xkcd.com/882/). 
Curiously, Heard and Rubin-Delanchy worked out that their version of Tippet's method is unsound, but didn't put together that they may have had the incorrect formula.

From [page 6 ](https://arxiv.org/pdf/1707.06897.pdf).

![image](https://user-images.githubusercontent.com/1953382/149042312-1b34b483-4bb1-4348-a899-4400e54eb655.png)

@Wrzlprmft thanks for pointing these out. I checked and your observation that SciPy’s Pearson p-value is actually 1 - p turns out to be correct. See for instance [this 2009 paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2771157/) by Won et al.

![image](https://user-images.githubusercontent.com/1953382/149048683-0ced55cf-5507-47a4-b2b7-3b1925bc76e7.png)

SciPy's implementation doesn't reverse the inequality in Pearson's method from that used in Fisher's method.

```python
    if method == 'fisher':
        statistic = -2 * np.sum(np.log(pvalues))
        pval = distributions.chi2.sf(statistic, 2 * len(pvalues))
    elif method == 'pearson':
        statistic = -2 * np.sum(np.log1p(-pvalues))
        pval = distributions.chi2.sf(statistic, 2 * len(pvalues))
```


clearly a bug

monotonicity in tippet goes in the wrong direction

Based on the example I guess the `1 - (1 - min(p_1, ..., p_N))^N` transformation is the same as `beta.cdf(min(pvalues), 1, n)`
with cdf instead of sf as steppi pointed out.

`1 - (1 - min(p_1, ..., p_N))^N`  looks like the Sidak multiple testing correction to me applied to the smallest p-value

> I think it's more likely that this actually is a mistake in Heard and Rubin-Delanchy's paper, since I don't think there's any good theoretical justification for taking the min of the p-values as the meta-p-value.

They don't do that, at least not skimming the definitions
min(pvalues) is the "test statistic", the actual p-value needs to come from the distribution of the test statistic, which is almost right in scipy except that's the complement, not left tail.

(based on a bit of fast skimming of small parts of articles)


> They don't do that, at least not skimming the definitions
> min(pvalues) is the "test statistic", the actual p-value needs to come from the distribution of the test statistic, which is almost right in scipy except that's the complement, not left tail.

You are correct. I skimmed too carelessly :)

Update: and proposition 2 that I posted a picture of but clearly didn’t read carefully is actually a pretty neat result 

Resolving patch:
diff --git a/scipy/stats/_stats_py.py b/scipy/stats/_stats_py.py
--- a/scipy/stats/_stats_py.py
+++ b/scipy/stats/_stats_py.py
@@ -8107,26 +8107,33 @@ def brunnermunzel(x, y, alternative="two-sided", distribution="t",
 
 def combine_pvalues(pvalues, method='fisher', weights=None):
     """
-    Combine p-values from independent tests bearing upon the same hypothesis.
+    Combine p-values from independent tests that bear upon the same hypothesis.
+
+    These methods are intended only for combining p-values from hypothesis
+    tests based upon continuous distributions.
+
+    Each method assumes that under the null hypothesis, the p-values are
+    sampled independently and uniformly from the interval [0, 1]. A test
+    statistic (different for each method) is computed and a combined
+    p-value is calculated based upon the distribution of this test statistic
+    under the null hypothesis.
 
     Parameters
     ----------
     pvalues : array_like, 1-D
-        Array of p-values assumed to come from independent tests.
-    method : {'fisher', 'pearson', 'tippett', 'stouffer',
-              'mudholkar_george'}, optional
+        Array of p-values assumed to come from independent tests based on
+        continuous distributions.
+    method : {'fisher', 'pearson', 'tippett', 'stouffer', 'mudholkar_george'}
 
         Name of method to use to combine p-values.
-        The following methods are available (default is 'fisher'):
-
-          * 'fisher': Fisher's method (Fisher's combined probability test), the
-            sum of the logarithm of the p-values
-          * 'pearson': Pearson's method (similar to Fisher's but uses sum of the
-            complement of the p-values inside the logarithms)
-          * 'tippett': Tippett's method (minimum of p-values)
-          * 'stouffer': Stouffer's Z-score method
-          * 'mudholkar_george': the difference of Fisher's and Pearson's methods
-            divided by 2
+
+        The available methods are (see Notes for details):
+
+        * 'fisher': Fisher's method (Fisher's combined probability test)
+        * 'pearson': Pearson's method
+        * 'mudholkar_george': Mudholkar's and George's method
+        * 'tippett': Tippett's method
+        * 'stouffer': Stouffer's Z-score method
     weights : array_like, 1-D, optional
         Optional array of weights used only for Stouffer's Z-score method.
 
@@ -8139,44 +8146,62 @@ def combine_pvalues(pvalues, method='fisher', weights=None):
 
     Notes
     -----
-    Fisher's method (also known as Fisher's combined probability test) [1]_ uses
-    a chi-squared statistic to compute a combined p-value. The closely related
-    Stouffer's Z-score method [2]_ uses Z-scores rather than p-values. The
-    advantage of Stouffer's method is that it is straightforward to introduce
-    weights, which can make Stouffer's method more powerful than Fisher's
-    method when the p-values are from studies of different size [6]_ [7]_.
-    The Pearson's method uses :math:`log(1-p_i)` inside the sum whereas Fisher's
-    method uses :math:`log(p_i)` [4]_. For Fisher's and Pearson's method, the
-    sum of the logarithms is multiplied by -2 in the implementation. This
-    quantity has a chi-square distribution that determines the p-value. The
-    `mudholkar_george` method is the difference of the Fisher's and Pearson's
-    test statistics, each of which include the -2 factor [4]_. However, the
-    `mudholkar_george` method does not include these -2 factors. The test
-    statistic of `mudholkar_george` is the sum of logisitic random variables and
-    equation 3.6 in [3]_ is used to approximate the p-value based on Student's
-    t-distribution.
+    If this function is applied to tests with a discrete statistics such as
+    any rank test or contingency-table test, it will yield systematically
+    wrong results, e.g. Fisher's method will systematically overestimate the
+    p-value [1]_. This problem becomes less severe for large sample sizes
+    when the discrete distributions become approximately continuous.
+
+    The differences between the methods can be best illustrated by their
+    statistics and what aspects of a combination of p-values they emphasise
+    when considering significance [2]_. For example, methods emphasising large
+    p-values are more sensitive to strong false and true negatives; conversely
+    methods focussing on small p-values are sensitive to positives.
+
+    * The statistics of Fisher's method (also known as Fisher's combined
+      probability test) [3]_ is :math:`-2\\sum_i \\log(p_i)`, which is
+      equivalent (as a test statistics) to the product of individual p-values:
+      :math:`\\prod_i p_i`. Under the null hypothesis, this statistics follows
+      a :math:`\\chi^2` distribution. This method emphasises small p-values.
+    * Pearson's method uses :math:`-2\\sum_i\\log(1-p_i)`, which is equivalent
+      to :math:`\\prod_i \\frac{1}{1-p_i}` [2]_.
+      It thus emphasises large p-values.
+    * Mudholkar and George compromise between Fisher's and Pearson's method by
+      averaging their statistics [4]_. Their method emphasises extreme
+      p-values, both close to 1 and 0.
+    * Stouffer's method [5]_ uses Z-scores and the statistic:
+      :math:`\\sum_i \\Phi^{-1} (p_i)`, where :math:`\\Phi` is the CDF of the
+      standard normal distribution. The advantage of this method is that it is
+      straightforward to introduce weights, which can make Stouffer's method
+      more powerful than Fisher's method when the p-values are from studies
+      of different size [6]_ [7]_.
+    * Tippett's method uses the smallest p-value as a statistic.
+      (Mind that this minimum is not the combined p-value.)
 
     Fisher's method may be extended to combine p-values from dependent tests
-    [5]_. Extensions such as Brown's method and Kost's method are not currently
+    [8]_. Extensions such as Brown's method and Kost's method are not currently
     implemented.
 
     .. versionadded:: 0.15.0
 
     References
     ----------
-    .. [1] https://en.wikipedia.org/wiki/Fisher%27s_method
-    .. [2] https://en.wikipedia.org/wiki/Fisher%27s_method#Relation_to_Stouffer.27s_Z-score_method
-    .. [3] George, E. O., and G. S. Mudholkar. "On the convolution of logistic
-           random variables." Metrika 30.1 (1983): 1-13.
-    .. [4] Heard, N. and Rubin-Delanchey, P. "Choosing between methods of
+    .. [1] Kincaid, W. M., "The Combination of Tests Based on Discrete
+           Distributions." Journal of the American Statistical Association 57,
+           no. 297 (1962), 10-19.
+    .. [2] Heard, N. and Rubin-Delanchey, P. "Choosing between methods of
            combining p-values."  Biometrika 105.1 (2018): 239-246.
-    .. [5] Whitlock, M. C. "Combining probability from independent tests: the
+    .. [3] https://en.wikipedia.org/wiki/Fisher%27s_method
+    .. [4] George, E. O., and G. S. Mudholkar. "On the convolution of logistic
+           random variables." Metrika 30.1 (1983): 1-13.
+    .. [5] https://en.wikipedia.org/wiki/Fisher%27s_method#Relation_to_Stouffer.27s_Z-score_method
+    .. [6] Whitlock, M. C. "Combining probability from independent tests: the
            weighted Z-method is superior to Fisher's approach." Journal of
            Evolutionary Biology 18, no. 5 (2005): 1368-1373.
-    .. [6] Zaykin, Dmitri V. "Optimally weighted Z-test is a powerful method
+    .. [7] Zaykin, Dmitri V. "Optimally weighted Z-test is a powerful method
            for combining probabilities in meta-analysis." Journal of
            Evolutionary Biology 24, no. 8 (2011): 1836-1841.
-    .. [7] https://en.wikipedia.org/wiki/Extensions_of_Fisher%27s_method
+    .. [8] https://en.wikipedia.org/wiki/Extensions_of_Fisher%27s_method
 
     """
     pvalues = np.asarray(pvalues)
@@ -8187,8 +8212,8 @@ def combine_pvalues(pvalues, method='fisher', weights=None):
         statistic = -2 * np.sum(np.log(pvalues))
         pval = distributions.chi2.sf(statistic, 2 * len(pvalues))
     elif method == 'pearson':
-        statistic = -2 * np.sum(np.log1p(-pvalues))
-        pval = distributions.chi2.sf(statistic, 2 * len(pvalues))
+        statistic = 2 * np.sum(np.log1p(-pvalues))
+        pval = distributions.chi2.cdf(-statistic, 2 * len(pvalues))
     elif method == 'mudholkar_george':
         normalizing_factor = np.sqrt(3/len(pvalues))/np.pi
         statistic = -np.sum(np.log(pvalues)) + np.sum(np.log1p(-pvalues))
@@ -8198,7 +8223,7 @@ def combine_pvalues(pvalues, method='fisher', weights=None):
                                   * approx_factor, nu)
     elif method == 'tippett':
         statistic = np.min(pvalues)
-        pval = distributions.beta.sf(statistic, 1, len(pvalues))
+        pval = distributions.beta.cdf(statistic, 1, len(pvalues))
     elif method == 'stouffer':
         if weights is None:
             weights = np.ones_like(pvalues)
@@ -8215,8 +8240,9 @@ def combine_pvalues(pvalues, method='fisher', weights=None):
 
     else:
         raise ValueError(
-            "Invalid method '%s'. Options are 'fisher', 'pearson', \
-            'mudholkar_george', 'tippett', 'or 'stouffer'", method)
+            f"Invalid method {method!r}. Valid methods are 'fisher', "
+            "'pearson', 'mudholkar_george', 'tippett', and 'stouffer'"
+        )
 
     return (statistic, pval)
 


Test patch:
diff --git a/scipy/stats/tests/test_stats.py b/scipy/stats/tests/test_stats.py
--- a/scipy/stats/tests/test_stats.py
+++ b/scipy/stats/tests/test_stats.py
@@ -6598,23 +6598,49 @@ def test_weighted_stouffer2(self):
 
     def test_pearson(self):
         Z, p = stats.combine_pvalues([.01, .2, .3], method='pearson')
-        assert_approx_equal(p, 0.97787, significant=4)
+        assert_approx_equal(p, 0.02213, significant=4)
 
     def test_tippett(self):
         Z, p = stats.combine_pvalues([.01, .2, .3], method='tippett')
-        assert_approx_equal(p, 0.970299, significant=4)
+        assert_approx_equal(p, 0.0297, significant=4)
 
     def test_mudholkar_george(self):
         Z, p = stats.combine_pvalues([.1, .1, .1], method='mudholkar_george')
         assert_approx_equal(p, 0.019462, significant=4)
 
-    def test_mudholkar_george_equal_fisher_minus_pearson(self):
+    def test_mudholkar_george_equal_fisher_pearson_average(self):
         Z, p = stats.combine_pvalues([.01, .2, .3], method='mudholkar_george')
         Z_f, p_f = stats.combine_pvalues([.01, .2, .3], method='fisher')
         Z_p, p_p = stats.combine_pvalues([.01, .2, .3], method='pearson')
-        # 0.5 here is because logistic = log(u) - log(1-u), i.e. no 2 factors
-        assert_approx_equal(0.5 * (Z_f-Z_p), Z, significant=4)
+        assert_approx_equal(0.5 * (Z_f+Z_p), Z, significant=4)
 
+    @pytest.mark.parametrize("variant", ["single", "all", "random"])
+    @pytest.mark.parametrize(
+        "method",
+        ["fisher", "pearson", "tippett", "stouffer", "mudholkar_george"],
+    )
+    def test_monotonicity(self, variant, method):
+        # Test that result increases monotonically with respect to input.
+        m, n = 10, 7
+        rng = np.random.default_rng(278448169958891062669391462690811630763)
+
+        # `pvaluess` is an m × n array of p values. Each row corresponds to
+        # a set of p values to be combined with p values increasing
+        # monotonically down one column (single), simultaneously down each
+        # column (all), or independently down each column (random).
+        if variant == "single":
+            pvaluess = np.full((m, n), rng.random(n))
+            pvaluess[:, 0] = np.linspace(0.1, 0.9, m)
+        elif variant == "all":
+            pvaluess = np.full((n, m), np.linspace(0.1, 0.9, m)).T
+        elif variant == "random":
+            pvaluess = np.sort(rng.uniform(0, 1, size=(m, n)), axis=0)
+
+        combined_pvalues = [
+            stats.combine_pvalues(pvalues, method=method)[1]
+            for pvalues in pvaluess
+        ]
+        assert np.all(np.diff(combined_pvalues) >= 0)
 
 class TestCdfDistanceValidation:
     """