Repository: scipy/scipy

Issue:
scipy.stats.combine_pvalues produces wrong results with method='mudholkar_george'
<!-- 
Thank you for taking the time to file a bug report. 
Please fill in the fields below, deleting the sections that 
don't apply to your issue. You can view the final output
by clicking the preview button above.
Note: This is a comment, and won't appear in the output.
-->

scipy.stats.combine_pvalues seems to work incorrectly with method=‘mudholkar_george’. In file scipy/scipy/stats/stats.py lines 7528-7533:
```
    elif method == 'mudholkar_george':
        statistic = -np.sum(np.log(pvalues)) + np.sum(np.log1p(-pvalues))
        nu = 5 * len(pvalues) + 4
        approx_factor = np.sqrt(nu / (nu - 2))
        pval = distributions.t.sf(statistic * approx_factor, nu)
```
According to the [original article](https://www.researchgate.net/publication/235117023_The_Logit_Statistic_for_Combining_Probabilities_-_An_Overview) approx_factor should be equal to np.sqrt(nu / (nu - 2)) / scipy.pi * np.sqrt(3 / len(pvalues)), not np.sqrt(nu / (nu - 2))


#### Reproducing code example:
<!-- 
If you place your code between the triple backticks below, 
it will be rendered as a code block. 
-->

```
from scipy import stats
stats.combine_pvalues([0.1, 0.1, 0.1], method='mudholkar_george')
```
The resulting p-value is 6.1e-07, however, the result should be 0.019, that can be achieved for example using R package [metap](https://CRAN.R-project.org/package=metap).

#### Scipy/Numpy/Python version information:
```
1.5.2 1.19.0 sys.version_info(major=3, minor=8, micro=5, releaselevel='final', serial=0)
```


Discussion:
Thanks for reporting the issue @M1kss 
During the [PR](https://github.com/scipy/scipy/pull/9660) I believe @chrisb83 had a similar question. 
When I implemented the Mudholker George method I looked at both methods:
https://gist.github.com/rlucas7/53ceae3b0ebbb1662892246d58f6a70a
and given the data in the paper: 
https://apps.dtic.mil/dtic/tr/fulltext/u2/a049996.pdf
it seemed like the method used had better numerical precision than the method used in:
https://apps.dtic.mil/dtic/tr/fulltext/u2/a049993.pdf

>The resulting p-value is 6.1e-07, however, the result should be 0.019,

How is the the conclusion

>the result should be 0.019

arrived at, is this purely a comparison against the results from the R-package?

Note I was not aware that the R package `metap` existed when I implemented these methods so I haven't compared nor investigated the implementations in that package.   
Thanks for the quick response!

1. "The result should be 0.019" is purely from the R-package. However, this corresponds to the fact, that the p-values from other methods applied to this data are of the same order of magnitude (0.032 for fisher, 0.013 for stouffer), while the result here is ~10^-7.

2. Following your gist, column x in table 1 of the https://doi.org/10.1007/BF02056895 represents the _standardized statisic_, that is denoted by the formula (3.1) in this article. So the both formulae, that you mention, (3.7) of this article and (2.5) of the original article are exactly the same.

3. Also in the gist provided there are two typos:
    1) line 10: >>> nu=5*k + 5 should be nu=5*k + 4
    2) in the comment, last line: F - ss.t.cdf(approx_factor * x, nu) should be F - ss.t.cdf(x/approx_factor, nu) to match the method, however, once again, this would not reproduce the results of table1, because the statistic x in the last article is already assumed to be standardized (scaled by sqrt(3/k)/pi).
Ok. I haven’t checked to see how this changes the numerical calculations (and if they give the same result as r package in this case) but I think I agree with conclusion of code misses the normalizing factor of sqrt((pi*3)/len(pvalues))
I think a line can be added ***before*** this one: 

https://github.com/scipy/scipy/blob/f7f31b4574c9665afd87068fb466f78a7bc37dd6/scipy/stats/stats.py#L7555

To add the normalizing factor which can then go into the calculation at this line:

https://github.com/scipy/scipy/blob/f7f31b4574c9665afd87068fb466f78a7bc37dd6/scipy/stats/stats.py#L7558
 
And IIUC this second line also needs to have: 
Statistic/approx_factor 
Rather than 
Statistic * approx_factor

So that end result would be: 

Statistic * normalizing_factor /approx_factor

Do you agree @M1kss ?

-Lucas Roberts 

> On Aug 2, 2020, at 6:48 AM, M1kss <notifications@github.com> wrote:
> 
> ﻿
> Thanks for the quick response!
> 
> "The result should be 0.019" is purely from the R-package. However, this corresponds to the fact, that the p-values from other methods applied to this data are of the same order of magnitude (0.032 for fisher, 0.013 for stouffer), while the result here is ~10^-7.
> 
> Following your gist, column x in table 1 of the https://doi.org/10.1007/BF02056895 represents the standardized statisic, that is denoted by the formula (3.1) in this article. So the both formulae, that you mention, (3.7) of this article and (2.5) of the original article are exactly the same.
> 
> Also in the gist provided there are two typos:
> 
> line 10: >>> nu=5k + 5 should be nu=5k + 4
> in the comment, last line: F - ss.t.cdf(approx_factor * x, nu) should be F - ss.t.cdf(x/approx_factor, nu) to match the method, however, once again, this would not reproduce the results of table1, because the statistic x in the last article is already assumed to be standardized (scaled by sqrt(3/k)/pi).
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub, or unsubscribe.

@rlucas7
Yes, adding the normalization factor yields the same result as the r package in the case discussed above ([0.1, 0.1, 0.1]) and a few of other test cases, but I haven't performed a detailed comparison.

As for the code suggestions,
1. Normalizing factor should be sqrt(3/len(pvalues))/pi, not sqrt((pi*3)/len(pvalues))
2. The statistic should be _multiplied_ by the approx factor, as it is sqrt(nu/(nu-2)) here, not sqrt((nu-2)/nu), that was in the gist.

So, considering these notes, I see the code as follows:
```python3
elif method == 'mudholkar_george':
        normalizing_factor = np.sqrt(3/len(pvalues))/np.pi
        statistic = -np.sum(np.log(pvalues)) + np.sum(np.log1p(-pvalues))
        nu = 5 * len(pvalues) + 4
        approx_factor = np.sqrt(nu / (nu - 2))
        pval = distributions.t.sf(statistic * normalizing_factor * approx_factor, nu)
```

> Ok. I haven’t checked to see how this changes the numerical calculations (and if they give the same result as r package in this case) but I think I agree with conclusion of code misses the normalizing factor of sqrt((pi*3)/len(pvalues)) I think a line can be added ***before*** this one: https://github.com/scipy/scipy/blob/f7f31b4574c9665afd87068fb466f78a7bc37dd6/scipy/stats/stats.py#L7555 To add the normalizing factor which can then go into the calculation at this line: https://github.com/scipy/scipy/blob/f7f31b4574c9665afd87068fb466f78a7bc37dd6/scipy/stats/stats.py#L7558 And IIUC this second line also needs to have: Statistic/approx_factor Rather than Statistic * approx_factor So that end result would be: Statistic * normalizing_factor /approx_factor Do you agree @M1kss ?
> […](#)
> -Lucas Roberts
> On Aug 2, 2020, at 6:48 AM, M1kss ***@***.***> wrote: ﻿ Thanks for the quick response! "The result should be 0.019" is purely from the R-package. However, this corresponds to the fact, that the p-values from other methods applied to this data are of the same order of magnitude (0.032 for fisher, 0.013 for stouffer), while the result here is ~10^-7. Following your gist, column x in table 1 of the https://doi.org/10.1007/BF02056895 represents the standardized statisic, that is denoted by the formula (3.1) in this article. So the both formulae, that you mention, (3.7) of this article and (2.5) of the original article are exactly the same. Also in the gist provided there are two typos: line 10: >>> nu=5k + 5 should be nu=5k + 4 in the comment, last line: F - ss.t.cdf(approx_factor * x, nu) should be F - ss.t.cdf(x/approx_factor, nu) to match the method, however, once again, this would not reproduce the results of table1, because the statistic x in the last article is already assumed to be standardized (scaled by sqrt(3/k)/pi). — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.


@M1kss @rlucas7 I'm new to SciPy but I'd like to help if this is a good first issue. It seems like the change should be as follows from above; any thoughts on if it would be any different than just making the below change / testing / PR? 

```python 
elif method == 'mudholkar_george':
        normalizing_factor = np.sqrt(3/len(pvalues))/np.pi
        statistic = -np.sum(np.log(pvalues)) + np.sum(np.log1p(-pvalues))
        nu = 5 * len(pvalues) + 4
        approx_factor = np.sqrt(nu / (nu - 2))
        pval = distributions.t.sf(statistic * normalizing_factor * approx_factor, nu)
```

@WillTirone, I've started looking into it but need a bit more time to understand things. Hoping to have some time tomorrow to look into this to make sure I agree with the changes. It also should involve some additional testing against the R package `metap` too. I haven't looked into that R package extensively but would like to do so before moving forward. 
There may be other discrepancies as well. If you are familiar with R you could start with setting up a bunch of (random) experiments comparing 'mudholkar_george' and other methods for agreement/disagreement and document your findings here. 
@rlucas7 ah I see, that makes sense. Unfortunately I've never used R, probably an issue better suited to someone with more experience in that area. Thanks!


> On Aug 11, 2020, at 12:29 AM, WillTirone <notifications@github.com> wrote:
> 
> ﻿
> @rlucas7 ah I see, that makes sense. Unfortunately I've never used R, probably an issue better suited to someone with more experience in that area. Thanks!
> 
Sure no problem. 

Thanks for inquiring. As I look into this a bit more I’ll keep my eyes posted for isolated increments.

> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

@rlucas7 
If this in any way helps, here is the `metap` R source code, that can be found here https://cran.r-project.org/web/packages/metap/index.html:

```r
logitp <-
function(p)  {
   keep <- (p > 0) & (p < 1)
   invalid <- sum(1L * keep) < 2
   if(invalid) {
      warning("Must have at least two valid p values")
      res <- list(t = NA_real_, df = NA_integer_, p = NA_real_,
         validp = p[keep])
   } else {
      psum <- sum(log(p[keep] / (1 - p[keep])))
      k <- length(p[keep])
      mult <- -1 / sqrt(k * pi ^ 2 * (5 * k + 2) / (3 * (5 * k + 4)))
      if(k != length(p)) {
         warning("Some studies omitted")
      }
      t <- mult * psum
      df <- (5 * k + 4)
      res <- list(t = t, df = df, p = pt(t, df, lower.tail = FALSE),
         validp = p[keep])
   }
   class(res) <- c("logitp", "metap")
   res
}
print.logitp <- function(x, ...) {
   cat("t = ", x$t, " with df = ", x$df, " p = ", x$p, "\n")
   invisible(x)
}
```
The algorithm seems to be exactly the same as discussed earlier, however 'metap' solution filters out p=0 and p=1 as they're leading to the logit statistic of positive infinity or negative infinity. Following 'stouffer' method in 'stats.combine_pvalues', these p-values are not filtered out, but lead to the aggregated p-values of 0 and 1 respectively. So, it might be logical not to filter out 0-s and 1-s here too.

Except for this filtering, the algorithm is identical to the one discussed [above](https://github.com/scipy/scipy/issues/12654#issuecomment-669242874).
@M1kss 

Thanks for the pointer (and code).

Hmm, I can see the value with some logic for handling edge cases (p=0,1, etc) but that would impact all methods so I think that should be handled as a separate issue. 

@WillTirone are you interested in giving this a shot then? I’d be happy to review a PR with the fix.

Sincerely, 

-Lucas Roberts 

>> On Aug 12, 2020, at 12:06 PM, Alexandr Boytsov <notifications@github.com> wrote:
> ﻿
> If this in any way helps, here is the metap R source code, that can be found here https://cran.r-project.org/web/packages/metap/index.html:
> 
> logitp <-
> function(p)  {
>    keep <- (p > 0) & (p < 1)
>    invalid <- sum(1L * keep) < 2
>    if(invalid) {
>       warning("Must have at least two valid p values")
>       res <- list(t = NA_real_, df = NA_integer_, p = NA_real_,
>          validp = p[keep])
>    } else {
>       psum <- sum(log(p[keep] / (1 - p[keep])))
>       k <- length(p[keep])
>       mult <- -1 / sqrt(k * pi ^ 2 * (5 * k + 2) / (3 * (5 * k + 4)))
>       if(k != length(p)) {
>          warning("Some studies omitted")
>       }
>       t <- mult * psum
>       df <- (5 * k + 4)
>       res <- list(t = t, df = df, p = pt(t, df, lower.tail = FALSE),
>          validp = p[keep])
>    }
>    class(res) <- c("logitp", "metap")
>    res
> }
> print.logitp <- function(x, ...) {
>    cat("t = ", x$t, " with df = ", x$df, " p = ", x$p, "\n")
>    invisible(x)
> }
> The algorithm seems to be exactly the same as discussed earlier, however 'metap' solution filters out p=0 and p=1 as they're leading to the logit statistic of positive infinity or negative infinity. Following 'stouffer' method in 'stats.combine_pvalues', these p-values are not filtered out, but lead to the aggregated p-values of 0 and 1 respectively. So, it might be logical not to filter out 0-s and 1-s here too.
> 
> Except for this filtering, the algorithm is identical to the one discussed [above].(#12654 (comment).)
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

Response got truncated (email on phone is finicky)

@WillTirone I’d be happy to review a PR with the changes discussed here. 

-Lucas Roberts 

> On Aug 14, 2020, at 8:10 AM, rlucas7@vt.edu wrote:
> 
> ﻿@M1kss 
> 
> Thanks for the pointer (and code).
> 
> Hmm, I can see the value with some logic for handling edge cases (p=0,1, etc) but that would impact all methods so I think that should be handled as a separate issue. 
> 
> @Will
> 
> Sincerely, 
> 
> -Lucas Roberts 
> 
>>> On Aug 12, 2020, at 12:06 PM, Alexandr Boytsov <notifications@github.com> wrote:
>>> 
>> ﻿
>> If this in any way helps, here is the metap R source code, that can be found here https://cran.r-project.org/web/packages/metap/index.html:
>> 
>> logitp <-
>> function(p)  {
>>    keep <- (p > 0) & (p < 1)
>>    invalid <- sum(1L * keep) < 2
>>    if(invalid) {
>>       warning("Must have at least two valid p values")
>>       res <- list(t = NA_real_, df = NA_integer_, p = NA_real_,
>>          validp = p[keep])
>>    } else {
>>       psum <- sum(log(p[keep] / (1 - p[keep])))
>>       k <- length(p[keep])
>>       mult <- -1 / sqrt(k * pi ^ 2 * (5 * k + 2) / (3 * (5 * k + 4)))
>>       if(k != length(p)) {
>>          warning("Some studies omitted")
>>       }
>>       t <- mult * psum
>>       df <- (5 * k + 4)
>>       res <- list(t = t, df = df, p = pt(t, df, lower.tail = FALSE),
>>          validp = p[keep])
>>    }
>>    class(res) <- c("logitp", "metap")
>>    res
>> }
>> print.logitp <- function(x, ...) {
>>    cat("t = ", x$t, " with df = ", x$df, " p = ", x$p, "\n")
>>    invisible(x)
>> }
>> The algorithm seems to be exactly the same as discussed earlier, however 'metap' solution filters out p=0 and p=1 as they're leading to the logit statistic of positive infinity or negative infinity. Following 'stouffer' method in 'stats.combine_pvalues', these p-values are not filtered out, but lead to the aggregated p-values of 0 and 1 respectively. So, it might be logical not to filter out 0-s and 1-s here too.
>> 
>> Except for this filtering, the algorithm is identical to the one discussed [above].(#12654 (comment).)
>> 
>> —
>> You are receiving this because you were mentioned.
>> Reply to this email directly, view it on GitHub, or unsubscribe.

@rlucas7 sure I'd be happy to take a shot! I'll just incorporate the changes listed above and send in a PR. Thanks!

Resolving patch:
diff --git a/scipy/stats/stats.py b/scipy/stats/stats.py
--- a/scipy/stats/stats.py
+++ b/scipy/stats/stats.py
@@ -7552,10 +7552,11 @@ def combine_pvalues(pvalues, method='fisher', weights=None):
         statistic = -2 * np.sum(np.log1p(-pvalues))
         pval = distributions.chi2.sf(statistic, 2 * len(pvalues))
     elif method == 'mudholkar_george':
+        normalizing_factor = np.sqrt(3/len(pvalues))/np.pi
         statistic = -np.sum(np.log(pvalues)) + np.sum(np.log1p(-pvalues))
         nu = 5 * len(pvalues) + 4
         approx_factor = np.sqrt(nu / (nu - 2))
-        pval = distributions.t.sf(statistic * approx_factor, nu)
+        pval = distributions.t.sf(statistic * normalizing_factor * approx_factor, nu)
     elif method == 'tippett':
         statistic = np.min(pvalues)
         pval = distributions.beta.sf(statistic, 1, len(pvalues))


Test patch:
diff --git a/scipy/stats/tests/test_stats.py b/scipy/stats/tests/test_stats.py
--- a/scipy/stats/tests/test_stats.py
+++ b/scipy/stats/tests/test_stats.py
@@ -5021,8 +5021,8 @@ def test_tippett(self):
         assert_approx_equal(p, 0.970299, significant=4)
 
     def test_mudholkar_george(self):
-        Z, p = stats.combine_pvalues([.01, .2, .3], method='mudholkar_george')
-        assert_approx_equal(p, 3.7191571041915e-07, significant=4)
+        Z, p = stats.combine_pvalues([.1, .1, .1], method='mudholkar_george')
+        assert_approx_equal(p, 0.019462, significant=4)
 
     def test_mudholkar_george_equal_fisher_minus_pearson(self):
         Z, p = stats.combine_pvalues([.01, .2, .3], method='mudholkar_george')