Repository: scipy/scipy

Issue:
Non-negative values of `stats.hypergeom.logcdf`
The `logcdf` method for `stats.hypergeom.logcdf` produces for certain parameters non-negative outputs, leading to probabilities > 1.

### Reproducing code example:
```python
import numpy as np
import scipy.stats as st

x, M, n, N = 40, 1600, 50, 300
lcdf = st.hypergeom.logcdf(x, M, n, N)
print("logcdf:", lcdf)
print("probab:", repr(np.exp(lcdf)))
```
results on my system (MacOS w/scipy installed from Anaconda) in:

```
logcdf: 7.578382366088447e-13
probab: 1.0000000000007578
```

For reference, using this R code (with the same values of x, M, n and N as above):

```R
lcdf <- phyper(x, n, M-n, N, lower.tail=TRUE, log=TRUE)
cat("logcdf:", sprintf("%.12e", lcdf), "\n")
cat("probab:", exp(lcdf))
```

I get:

```
logcdf: -7.565148879229e-23 
probab: 1
 ```

Note the call to `phyper` needs to be made with slight adjustments to the parameters to match the conventions in the R definition, as warned in the [scipy.stats docs](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.hypergeom.html#scipy.stats.hypergeom). See the R docs for this function [here](https://www.rdocumentation.org/packages/stats/versions/3.4.3/topics/Hypergeometric).

### Scipy/Numpy/Python version information:
```
scipy: 1.0.1
numpy: 1.14.2
sys:   sys.version_info(major=3, minor=6, micro=5, releaselevel='final', serial=0)
 ```


Discussion:
For reference, here are a couple of other issues that could be relevant:

#2238: old, closed one but has good discussion about the special functions code that underlies hypergeom. Some of that code could still be involved in the one I'm reporting here.

#4982: precision problem fixed, though the issue remains open due to potential performance problems.

And as additional confirmation that for less extreme values things work fine in scipy and match R, here's a check for both scipy and R (run in the notebook with `%load_ext rpy2.ipython`):

```python
x, M, n, N = 3, 20, 7, 12
lcdf = st.hypergeom.logcdf(x, M, n, N)
print("logcdf:", lcdf)
print("probab:", repr(np.exp(lcdf)))
```
->
```
logcdf: -1.3832031685502197
probab: 0.25077399380804904
```

and R:

```R
%%R -i x,M,n,N
lcdf <- phyper(x, n, M-n, N, lower.tail=TRUE, log=TRUE)
cat("logcdf:", sprintf("%.12e", lcdf), "\n")
cat("probab:", exp(lcdf))
```

->

```
logcdf: -1.383203168550e+00 
probab: 0.250774
```
cc @mattip 
Digging into the code, the core function being called is `_pmf` in `scipy/stats/_discrete_distns.py`, which calls ` exp(self._logpmf(k, M, n, N))`. `logpmf` is implemented as a sum of `betaln` factors, which is implemented as a NumPy `ufunc` in cython from `special/_ufuncs.pyx`. Not clear to me whether the `logpmf` implementation is a couple of `eps` off or whether it is inaccuracies in ``betaln` itself. Can we get to the core implementation in `R` to compare?
I've been testing the code as well, and one of things I've noticed is that when I try to calculate the mathematical expression for `_logcdf ` based on MATLAB's definition here: https://www.mathworks.com/help/stats/hygecdf.html i.e. `print(np.sum(np.divide(np.multiply(comb(K_arr, k),comb(M_K_arr, np.subtract(N_arr, k))),comb(M_arr, N_arr))))`. In this case, comb is scipy.special.comb and k is a numpy array from 0 to the x parameter passed in, inclusive. Trying to run this results in an RuntimeWarning: overflow encountered in multiply. Not sure how betaln/gammaln deal with overflow. 

Is there a specific way to get the core implementation in R? Simply stepping through the code in RStudio doesn't work.
Also, the Runtime warning doesn’t appear in the less extreme example.
Careful with the R code, we cannot use it.
It seems the case boils down to this function
```
352          def _logpmf(self, k, M, n, N):
353              tot, good = M, n
354              bad = tot - good
355              import pdb;pdb.set_trace()
356              result = (betaln(good+1, 1) + betaln(bad+1, 1) + betaln(tot-N+1, N+1) -
357  ->                      betaln(k+1, good-k+1) - betaln(N-k+1, bad-N+k+1) -
358                        betaln(tot+1, 1))
359              return result
```
The end result is `np.sum(exp(result)) = 1.0 + 7.5e-13`, the problem seems to stem from numeric instability in `betaln() - betaln()`. In the code for `betaln` (found in `scipy/special/cephes/beta.c `, function `lbeta`, there is some consideration of what happens when computing `lgam(a + b) - lgam(a)`. Perhaps the code on line 357 needs to be refactored to take into account cases where M is large but n is small. Note that reversing the expression and summing
```
np.exp(st.hypergeom.logcdf(x, M, n, N)) + np.exp(st.hypergeom.logcdf(N-x, M, M - n, N))
```
shows the numeric instability, and that the non-log form is also inaccurate, for `M`=1600, `n` = 50, `N` = 300

```
st.hypergeom.cdf(x, M, n, N) + st.hypergeom.logcdf(N-x, M, M - n, N)
```

This inaccuracy holds, on my computer, for `0>x>49`, for `49 < x < 300` the sum of the two `cdf` s is 1.0 (for the `logcdf`. The cdf cutoff is 33 or so).

Also, slightly confusing that `scipy/special/cdflib/betaln.f` has the function `betaln` that apparently is not used here. Maybe that's a historical artifact or something.
It seems there needs to be code to detect the edge cases. Once that is established, 

- SciPy should emit a warning
- Would it be acceptable to do the inverse, i.e. instead of calculating `st.hypergeom.logcdf(x, M, n, N)`, calculate `np.log(1 - st.hypergeom.cdf(N-x, M, M - n, N))`? Is there precedent for this in the stats module?
@mattip By code to detect edge cases, are you referring to within the function to check when the bug happens, a test case, or both? I believe MATLAB's implementation specifically has a flag to pass in for extreme upper tail probabilities that calculates the upper tail 1 - p, but I'm not sure if that would serve as a precedent for SciPy to start using this method for more accuracy.
The problem is that the code for the hypergeometric cdf and sf functions needs to calculate the smaller tail and apply the "1-" if the tails have been swapped. See notes on [6840](https://github.com/scipy/scipy/issues/6840), which suffers the same problem.
It seems as if logcdf is implemented as `log(cdf(...))`, where logpmf is calculated in a more stable way
(From #5010)
```
 def _logsf(self, k, M, n, N):
        """
        More precise calculation than log(sf)
        """
        res = []
        for quant, tot, good, draw in zip(k, M, n, N):
            # Integration over probability mass function using logsumexp
            k2 = np.arange(quant + 1, draw + 1)
            res.append(logsumexp(self._logpmf(k2, tot, good, draw)))
        return np.asarray(res)
```

With minor changes we can define logcdf as
```
 def _logcdf(self, k, M, n, N):
        """
        More precise calculation than log(sf)
        """
        res = []
        for quant, tot, good, draw in zip(k, M, n, N):
            # Integration over probability mass function using logsumexp
            k2 = np.arange(0, quant+1)
            res.append(logsumexp(self._logpmf(k2, tot, good, draw)))
        return np.asarray(res)
```

Any suggestions as to how to validate that this is accurate?
This is what I'm currently using for my calculations
If you look at the Notes on Hypergeometric distribution functions in 
[https://stats.stackexchange.com/questions/279793/computational-shortcuts-approximations-for-beta-binomial-and-beta-negative-binom?noredirect=1&lq=1](url)
they will tell you how to calculate the pmf, cdf & sf for the Hypergeometric distribution.

The _logpmf function used in your code loses accuracy when the parameter values are large but it is accurate enough to get _logcdf correct to about 12 figures for the example reported as giving the problem.

With the recommended method for calculating the cdf in the Notes referenced above
for (i+½).(m+½) <= (n-½).(k-½)
ln(Pr(X<=i;n,k,m)) = ln(cf(i))+_logpmf(X=i;n,k,m)
for (i+½).(m+½) > (n-½).(k-½)
ln(Pr(X<=i;n,k,m)) = log1p(-Pr(X<=(n-i-1);n,m-k,m))

If you think about it, you cannot reasonably expect to use logsumexp to operate on all the _logpmf values and get the answer 0.

That said, I can provide you either with accurate answers for any examples you are interested in or with examples and answers if you would prefer. 

If you are interested in the code for the Hypergeometric distribution, I can either add parameters to the pmf, cdf and sf functions to indicate that logs of the answers are required or add logpmf, logcdf and logsf functions if you would prefer. You are then welcome to modify the code yourself.

I'm afraid I don't know how things are supposed to work around here and I certainly don't want to offend anyone, so please ignore this if I'm upsetting everyone.

> how

Can you provide some sample code? This would be very helpful
Code for cdf_hypergeometric & sf_hypergeometric is available in #7102 in the text file referred to by the mystatsBetaBinomialpy.txt link.

Both functions use the hypergeometricTerm and hypergeometric functions, which are parameterised in what might seem an odd way but it is best suited for use with the BetaNegativeBinomial distribution. Additionally, hypergeometric has a "comp" parameter to return cdf or sf (the complement of the cdf). An additional parameter to indicate that the log of the values should be returned, can easily be added to both hypergeometricTerm and hypergeometric. For hypergeometricTerm, it is clearly quite simple to modify the code to do this. For hypergeometric, it is only slightly more complicated.

If you want, I can do this and add logcdf_hypergeometric & logsf_hypergeometric functions or I can leave you to modify the code yourself.


 


Here is a text version of a module which has some test cases for logcdf&logsf_hypergeometric in it.

[HyperGeometriclogCdf.py.txt](https://github.com/scipy/scipy/files/2738806/HyperGeometriclogCdf.py.txt)

and these are the results it gives

pmf&logpmf_hypergeometric(107,1000.0,215.0,10000.0)     4.8410543401313856e-51 -115.8547072068211
pmf&logpmf_hypergeometric(40, 50, 300, 1600)     1.4743579253865967e-21 -47.96606436267847
Accurate values for cdf and sf in issue 6840
   cdf&sf_hypergeometric(107,1000.0,215.0,10000.0)     1.0 5.3908059082006355e-52
   cdf&sf_hypergeometric(107,1200.0,215.0,10000.0)     1.0 4.927676545540008e-44
   cdf&sf_hypergeometric(107,2000.0,215.0,10000.0)     1.0 1.3569340968179074e-23
   cdf&sf_hypergeometric(107,3000.0,215.0,10000.0)     0.9999999997226765 2.7732346732176105e-10
Accurate values for cdf and sf in issue 6840

Accurate values for cdf and sf in issue 8692
   cdf&sf_hypergeometric(40, 50, 300, 1600)     1.0 7.565148879228534e-23
Accurate values for cdf and sf in issue 8692

Accurate values for logcdf and logsf in issue 6840
   logcdf&logsf_hypergeometric(107,1000.0,215.0,10000.0)     -5.390805908200666e-52 -118.04972994279966
   logcdf&logsf_hypergeometric(107,1200.0,215.0,10000.0)     -4.927676545540018e-44 -99.71887650371207
   logcdf&logsf_hypergeometric(107,2000.0,215.0,10000.0)     -1.356934096817912e-23 -52.65422932454123
   logcdf&logsf_hypergeometric(107,3000.0,215.0,10000.0)     -2.7732346736021487e-10 -22.0058365389692
Accurate values for cdf and sf in issue 6840

Accurate values for logcdf and logsf in issue 6840
   logcdf&logsf_hypergeometric(892,1000.0,9785.0,10000.0)     -118.04972994279966 -5.390805908200666e-52
   logcdf&logsf_hypergeometric(1092,1200.0,9785.0,10000.0)     -99.71887650371207 -4.927676545540018e-44
   logcdf&logsf_hypergeometric(1892,2000.0,9785.0,10000.0)     -52.65422932454123 -1.356934096817912e-23
   logcdf&logsf_hypergeometric(2892,3000.0,9785.0,10000.0)     -22.0058365389692 -2.7732346736021487e-10
Accurate values for cdf and sf in issue 6840

Accurate values for cdf and sf in issue 8692
   logcdf&logsf_hypergeometric(40, 50, 300, 1600)     -7.565148879228508e-23 -50.93590511181261
Accurate values for cdf and sf in issue 8692

      cdf&sf_hypergeometric(30,4363,12390,13397950)  1.0 1.3299429823231134e-17
logcdf&logsf_hypergeometric(30,4363,12390,13397950)  -1.3299429823231123e-17 -38.858810510017825
      cdf&sf_hypergeometric(20,4363,9260,13397950)  0.9999999999877109 1.2289025467637998e-11
logcdf&logsf_hypergeometric(20,4363,9260,13397950)  -1.228902546771351e-11 -25.1223144902352

pmf&logpmf_hypergeometric(9939300,2e7,2e7,4e7)     0.0 -745.1873275735092
cdf&sf_hypergeometric(9939300,2e7,2e7,4e7)  0.0 1.0
logcdf&logsf_hypergeometric(9939300,2e7,2e7,4e7)  -741.4577911289049 -0.0
logcdf&logsf_hypergeometric(10060699,2e7,2e7,4e7)  -0.0 -741.4577911289049
pmf&logpmf_hypergeometric(981200,2e6,2e6,4e6)     7.7611053948566e-311 -714.054839149307
cdf&sf_hypergeometric(981200,2e6,2e6,4e6)  1.070504233120402e-309 1.0
logcdf&logsf_hypergeometric(981200,2e6,2e6,4e6)  -711.4306639517808 -1.070504233120402e-309
logcdf&logsf_hypergeometric(1018799,2e6,2e6,4e6)  -1.070504233120402e-309 -711.4306639517808
pmf&logpmf_hypergeometric(0,2,3,1e8)     0.9999999400000005 -6.000000130068628e-08
cdf&sf_hypergeometric(0,2,3,1e8)  0.9999999400000006 5.999999940000001e-08
logcdf&logsf_hypergeometric(0,2,3,1e8)  -6.000000120000003e-08 -16.62892128472431
logcdf&logsf_hypergeometric(1,2,1e8-3,1e8)  -16.62892128472431 -6.000000120000003e-08
pmf&logpmf_hypergeometric(1,2,3,1e8)     5.99999988e-08 -16.62892129472431
cdf&sf_hypergeometric(1,2,3,1e8)  0.9999999999999994 6.000000059999989e-16
logcdf&logsf_hypergeometric(1,2,3,1e8)  -6.000000059999983e-16 -35.04960200867668
logcdf&logsf_hypergeometric(0,2,1e8-3,1e8)  -35.04960200867668 -6.000000059999983e-16
pmf&logpmf_hypergeometric(5e7-1,5e7,1e8-1,1e8)     0.5 -0.6931471805599452
cdf&sf_hypergeometric(5e7-1,5e7,1e8-1,1e8)  0.4999999999999999 0.5000000000000001
logcdf&logsf_hypergeometric(5e7-1,5e7,1e8-1,1e8)  -0.6931471805599455 -0.6931471805599452
logcdf&logsf_hypergeometric(0,5e7,1,1e8)  -0.6931471805599455 -0.6931471805599452
cdf&sf_hypergeometric(5000000,10000000,50000000,100000000)  0.5001329807567715 0.4998670192432285
Time taken  0.015621662139892578
logcdf_hypergeometric(5000000,10000000,50000000,100000000)  -0.6928812544078974 -0.6934131774475221

cdf&sf_hypergeometric(1e12,2e12+1,2e12+1,4e12+2)     0.5000000000000013 0.49999999999999867
Time taken  0.14062213897705078
logcdf_hypergeometric(5000000,10000000,50000000,100000000)  -0.6931471805599414 -0.693147180559949
logcdf&logsf_hypergeometric(1e4,5e4,1e6,1e7)  -0.0 -2239.7712499203985
logcdf_&logsfhypergeometric(1,5e4,1e6,1e7)  -5273.335151655274 -0.0
>>> 



Resolving patch:
diff --git a/scipy/stats/_discrete_distns.py b/scipy/stats/_discrete_distns.py
--- a/scipy/stats/_discrete_distns.py
+++ b/scipy/stats/_discrete_distns.py
@@ -395,7 +395,6 @@ def _entropy(self, M, n, N):
         return np.sum(entr(vals), axis=0)
 
     def _sf(self, k, M, n, N):
-        """More precise calculation, 1 - cdf doesn't cut it."""
         # This for loop is needed because `k` can be an array. If that's the
         # case, the sf() method makes M, n and N arrays of the same shape. We
         # therefore unpack all inputs args, so we can do the manual
@@ -409,14 +408,27 @@ def _sf(self, k, M, n, N):
         return np.asarray(res)
 
     def _logsf(self, k, M, n, N):
-        """
-        More precise calculation than log(sf)
-        """
         res = []
         for quant, tot, good, draw in zip(k, M, n, N):
-            # Integration over probability mass function using logsumexp
-            k2 = np.arange(quant + 1, draw + 1)
-            res.append(logsumexp(self._logpmf(k2, tot, good, draw)))
+            if (quant + 0.5) * (tot + 0.5) < (good - 0.5) * (draw - 0.5):
+                # Less terms to sum if we calculate log(1-cdf)
+                res.append(log1p(-exp(self.logcdf(quant, tot, good, draw))))
+            else:
+                # Integration over probability mass function using logsumexp
+                k2 = np.arange(quant + 1, draw + 1)
+                res.append(logsumexp(self._logpmf(k2, tot, good, draw)))
+        return np.asarray(res)
+
+    def _logcdf(self, k, M, n, N):
+        res = []
+        for quant, tot, good, draw in zip(k, M, n, N):
+            if (quant + 0.5) * (tot + 0.5) > (good - 0.5) * (draw - 0.5):
+                # Less terms to sum if we calculate log(1-sf)
+                res.append(log1p(-exp(self.logsf(quant, tot, good, draw))))
+            else:
+                # Integration over probability mass function using logsumexp
+                k2 = np.arange(0, quant + 1)
+                res.append(logsumexp(self._logpmf(k2, tot, good, draw)))
         return np.asarray(res)
 
 
@@ -763,7 +775,7 @@ class zipf_gen(rv_discrete):
 
     for :math:`k \ge 1`.
 
-    `zipf` takes :math:`a` as shape parameter. :math:`\zeta` is the 
+    `zipf` takes :math:`a` as shape parameter. :math:`\zeta` is the
     Riemann zeta function (`scipy.special.zeta`)
 
     %(after_notes)s


Test patch:
diff --git a/scipy/stats/tests/test_distributions.py b/scipy/stats/tests/test_distributions.py
--- a/scipy/stats/tests/test_distributions.py
+++ b/scipy/stats/tests/test_distributions.py
@@ -589,8 +589,63 @@ def test_logsf(self):
         N = 5e4
 
         result = stats.hypergeom.logsf(k, M, n, N)
-        exspected = -2239.771   # From R
-        assert_almost_equal(result, exspected, decimal=3)
+        expected = -2239.771   # From R
+        assert_almost_equal(result, expected, decimal=3)
+
+        k = 1
+        M = 1600
+        n = 600
+        N = 300
+
+        result = stats.hypergeom.logsf(k, M, n, N)
+        expected = -2.566567e-68   # From R
+        assert_almost_equal(result, expected, decimal=15)
+
+    def test_logcdf(self):
+        # Test logcdf for very large numbers. See issue #8692
+        # Results compare with those from R (v3.3.2):
+        # phyper(k, n, M-n, N, lower.tail=TRUE, log.p=TRUE)
+        # -5273.335
+
+        k = 1
+        M = 1e7
+        n = 1e6
+        N = 5e4
+
+        result = stats.hypergeom.logcdf(k, M, n, N)
+        expected = -5273.335   # From R
+        assert_almost_equal(result, expected, decimal=3)
+
+        # Same example as in issue #8692
+        k = 40
+        M = 1600
+        n = 50
+        N = 300
+
+        result = stats.hypergeom.logcdf(k, M, n, N)
+        expected = -7.565148879229e-23    # From R
+        assert_almost_equal(result, expected, decimal=15)
+
+        k = 125
+        M = 1600
+        n = 250
+        N = 500
+
+        result = stats.hypergeom.logcdf(k, M, n, N)
+        expected = -4.242688e-12    # From R
+        assert_almost_equal(result, expected, decimal=15)
+
+        # test broadcasting robustness based on reviewer
+        # concerns in PR 9603; using an array version of
+        # the example from issue #8692
+        k = np.array([40, 40, 40])
+        M = 1600
+        n = 50
+        N = 300
+
+        result = stats.hypergeom.logcdf(k, M, n, N)
+        expected = np.full(3, -7.565148879229e-23)  # filled from R result
+        assert_almost_equal(result, expected, decimal=15)
 
 
 class TestLoggamma(object):