Repository: statsmodels/statsmodels

Issue:
BUG: proportion confint returns nan, negative and larger than 1
partially duplicate of #2609

see https://github.com/josef-pkt/misc/blob/proportion_one/notebooks/proportion_one_power.ipynb
(not yet committed and pushed)
example copied below

beta/exact confidence interval contains a NaN if the count is all of the same kind, 
normal and agresti_coull return proportions that are negative or larger than one.

This might also be the reason for the comment that binom_test inversion fails, it uses normal ci for the bounds of root search, which should fail if probability is negative or larger than 1.

TODO: trim/clip to [0,1], and explicitly set bounds to 0 or 1 for beta in corner case.

```
import numpy as np
import statsmodels.stats.proportion as smprop
import pandas as pd

confint_methods = ['beta', 'wilson', 'normal', 'agresti_coull', 'jeffrey']
nobs = 30
count_ = 0
confints0 = [smprop.proportion_confint(count_, nobs, alpha=0.05, method=method) for method in confint_methods]
count_ = 1
confints1 = [smprop.proportion_confint(count_, nobs, alpha=0.05, method=method) for method in confint_methods]
count_ = nobs - 1
confintsnm1 = [smprop.proportion_confint(count_, nobs, alpha=0.05, method=method) for method in confint_methods]
count_ = nobs
confintsn = [smprop.proportion_confint(count_, nobs, alpha=0.05, method=method) for method in confint_methods]
pd.DataFrame(np.column_stack((confints0, confints1, confintsnm1, confintsn)), index=confint_methods, 
             columns=['0 lower', '0 upper', '1 lower', '1 upper', 'n-1 lower', 'n-1 upper', 'n lower', 'n upper'])
```



Discussion:
I would like to work on this.
In case of 'beta', ci_low and ci_low are calculated using stats.beta function. So, should I implement clipping within the implementation of stats.beta?

The confidence intervals need clipping, beta needs clipping or special casing of the boundaries.

My example can be used for unit tests to check that the values are not nan and in between 0 and 1.

I have create a PR #2789 But the travis CI build showing some bugs. I don't think the error is related to my commits because the exact same error details are shown for PR #2779 

see my comment in #2789  
PR #2779 is not merged because it assumes at least 1-D when it can be scalar.
PR #2789 assumes scalar when it can be vectorized

@josef-pkt Thanks. You not only saved my time, but also provided me with really helpful inputs. It's nice to discuss doubts with you.

It's a `type-bug-wrong`, but the wrong numbers are not misleading
PR #3058 merged better array handling, so examples will not run in the same way anymore.
correction that is for proportion_ztest
we need a similar change for proportion_conf

AFAICS, no change in nan and out of [0, 1] range
(all normal distribution based approximation will have the possibility of out of range values, unless appropriate endpoint transformation is used for the confidence interval.)

'beta' and 'jeffreys' us beta distribution and will not go outside [0,1]. Those two are similar, except jeffreys add 0.5 and will stay away from the corner case (AFAICS)
i.e. we need to np.clip(..., 0, 1) for all except those two
current behavior

asarray is missing: 
- using numpy array and pandas Series both work. list raises an exception.
- if we add np.asarray, then we don't return pandas anymore if pandas was used for the data.
  currently:   
  wilson, normal and agresti_coul return pandas,   
  beta and jeffreys return array with pandas Seris as input

wilson doesn't return values outside [0, 1] in the example, so it's only 'normal' and 'agresti-coull' that needs clipping
my current WIP version allows pandas broadcasting after np.asarray

count_ is a DataFrame, nobs is a Series

```
>>> count_.shape
(3, 1)
>>> nobs_.shape
(3,)
>>> np.asarray(count_) / np.asarray(nobs_)
array([[ 0.        ,  0.        ,  0.        ],
       [ 0.1       ,  0.1       ,  0.03333333],
       [ 1.        ,  1.        ,  0.33333333]])
```

maybe that's too much, but it is a side effect of allowing numpy broadcasting inside the function.
There is no squeeze, so Series and a one column DataFrame are handled in a different way.

I haven't changed the column labels to account for this, the index comes from the `count` argument.
```
>>> type(count_)
<class 'pandas.core.frame.DataFrame'>
>>> type(nobs_)
<class 'pandas.core.series.Series'>
>>> for method in confint_methods:
...     print(method)
...     r = smprop.proportion_confint(count_, nobs_, alpha=0.05, method=method)
...     print(r)   
... 
beta
(          0         1         2
0  0.000000  0.000000  0.000000
1  0.002529  0.002529  0.000844
2  0.691503  0.691503  0.172874,           
0         1         2
0  0.308497  0.308497  0.115703
1  0.445016  0.445016  0.172169
2  1.000000  1.000000  0.528120)
wilson
(          0         1         2
0  0.000000  0.000000  0.000000
1  0.017876  0.017876  0.005909
2  0.722467  0.722467  0.192305,           
0         1         2
0  0.277533  0.277533  0.113513
1  0.404150  0.404150  0.166704
2  1.000000  1.000000  0.512199)
normal
(     0    1         2
0  0.0  0.0  0.000000
1  0.0  0.0  0.000000
2  1.0  1.0  0.164646,          
 0         1         2
0  0.000000  0.000000  0.000000
1  0.285939  0.285939  0.097567
2  1.000000  1.000000  0.502020)
agresti_coull
(          0         1         2
0  0.000000  0.000000  0.000000
1  0.000000  0.000000  0.000000
2  0.679113  0.679113  0.191316,           
0         1         2
0  0.320887  0.320887  0.134712
1  0.425968  0.425968  0.180918
2  1.000000  1.000000  0.513188)
jeffrey
(          0         1         2
0  0.000048  0.000048  0.000016
1  0.011012  0.011012  0.003620
2  0.782804  0.782804  0.185979,           
0         1         2
0  0.217196  0.217196  0.079678
1  0.381315  0.381315  0.145418
2  0.999952  0.999952  0.511148)
```

that is current behavior is, except for labels, like using
`c, n = map(pd.DataFrame, np.broadcast_arrays(np.asarray(count_), np.asarray(nobs_)))`
(or whatever the pandas version might possibly be, if it exists)

proportion_confint is an elementwise function
unit tests, the current unit tests fail if we impose [0,1] range

It looks like the R library for the unit test has one value larger than 1.
i.e. 1.0314784 for upper for 'normal' or 'asymptotic'

```
> library(binom)
> bci = binom.confint(x = c(18), n = 20, tol = 1e-8)
> bci
          method  x  n      mean     lower     upper
1  agresti-coull 18 20 0.9000000 0.6867561 0.9843438
2     asymptotic 18 20 0.9000000 0.7685216 1.0314784
3          bayes 18 20 0.8809524 0.7460094 0.9907234
4        cloglog 18 20 0.9000000 0.6560307 0.9740102
5          exact 18 20 0.9000000 0.6830173 0.9876515
6          logit 18 20 0.9000000 0.6761980 0.9748664
7         probit 18 20 0.9000000 0.7027685 0.9788585
8        profile 18 20 0.9000000 0.7220529 0.9823182
9            lrt 18 20 0.9000000 0.7220576 0.9826399
10     prop.test 18 20 0.9000000 0.6687224 0.9824874
11        wilson 18 20 0.9000000 0.6989664 0.9721335
```
The two corner cases in R binom

```
> bci = binom.confint(x = c(20), n = 20, tol = 1e-8)
> bci
          method  x  n      mean     lower    upper
1  agresti-coull 20 20 1.0000000 0.8101904 1.028684
2     asymptotic 20 20 1.0000000 1.0000000 1.000000
3          bayes 20 20 0.9761905 0.9095236 1.000000
4        cloglog 20 20 1.0000000 0.8315665 1.000000
5          exact 20 20 1.0000000 0.8315665 1.000000
6          logit 20 20 1.0000000 0.8315665 1.000000
7         probit 20 20 1.0000000 0.8315665 1.000000
8        profile 20 20 1.0000000 0.8497732 1.000000
9            lrt 20 20 1.0000000 0.9084309 1.000000
10     prop.test 20 20 1.0000000 0.7995467 1.000000
11        wilson 20 20 1.0000000 0.8388748 1.000000

> bci = binom.confint(x = c(0), n = 20, tol = 1e-8)
> bci
          method x  n       mean         lower      upper
1  agresti-coull 0 20 0.00000000 -2.868440e-02 0.18980956
2     asymptotic 0 20 0.00000000  0.000000e+00 0.00000000
3          bayes 0 20 0.02380952  0.000000e+00 0.09047643
4        cloglog 0 20 0.00000000  0.000000e+00 0.16843347
5          exact 0 20 0.00000000  0.000000e+00 0.16843347
6          logit 0 20 0.00000000  0.000000e+00 0.16843347
7         probit 0 20 0.00000000  0.000000e+00 0.16843347
8        profile 0 20 0.00000000  0.000000e+00 0.15022677
9            lrt 0 20 0.00000000  0.000000e+00 0.09156912
10     prop.test 0 20 0.00000000  0.000000e+00 0.20045335
11        wilson 0 20 0.00000000  1.164173e-17 0.16112516
```

Resolving patch:
diff --git a/statsmodels/stats/proportion.py b/statsmodels/stats/proportion.py
--- a/statsmodels/stats/proportion.py
+++ b/statsmodels/stats/proportion.py
@@ -20,8 +20,8 @@ def proportion_confint(count, nobs, alpha=0.05, method='normal'):
 
     Parameters
     ----------
-    count : int or array
-        number of successes
+    count : int or array_array_like
+        number of successes, can be pandas Series or DataFrame
     nobs : int
         total number of trials
     alpha : float in (0, 1)
@@ -39,16 +39,24 @@ def proportion_confint(count, nobs, alpha=0.05, method='normal'):
 
     Returns
     -------
-    ci_low, ci_upp : float
+    ci_low, ci_upp : float, ndarray, or pandas Series or DataFrame
         lower and upper confidence level with coverage (approximately) 1-alpha.
-        Note: Beta has coverage
-        coverage is only 1-alpha on average for some other methods.)
+        When a pandas object is returned, then the index is taken from the
+        `count`.
 
     Notes
     -----
-    Beta, the Clopper-Pearson interval has coverage at least 1-alpha, but is
-    in general conservative. Most of the other methods have average coverage
-    equal to 1-alpha, but will have smaller coverage in some cases.
+    Beta, the Clopper-Pearson exact interval has coverage at least 1-alpha,
+    but is in general conservative. Most of the other methods have average
+    coverage equal to 1-alpha, but will have smaller coverage in some cases.
+
+    The 'beta' and 'jeffreys' interval are central, they use alpha/2 in each
+    tail, and alpha is not adjusted at the boundaries. In the extreme case
+    when `count` is zero or equal to `nobs`, then the coverage will be only
+    1 - alpha/2 in the case of 'beta'.
+
+    The confidence intervals are clipped to be in the [0, 1] interval in the
+    case of 'normal' and 'agresti_coull'.
 
     Method "binom_test" directly inverts the binomial test in scipy.stats.
     which has discrete steps.
@@ -67,6 +75,13 @@ def proportion_confint(count, nobs, alpha=0.05, method='normal'):
 
     '''
 
+    pd_index = getattr(count, 'index', None)
+    if pd_index is not None and hasattr(pd_index, '__call__'):
+        # this rules out lists, lists have an index method
+        pd_index = None
+    count = np.asarray(count)
+    nobs = np.asarray(nobs)
+
     q_ = count * 1. / nobs
     alpha_2 = 0.5 * alpha
 
@@ -93,6 +108,13 @@ def func(qi):
         ci_low = stats.beta.ppf(alpha_2, count, nobs - count + 1)
         ci_upp = stats.beta.isf(alpha_2, count + 1, nobs - count)
 
+        if np.ndim(ci_low) > 0:
+            ci_low[q_ == 0] = 0
+            ci_upp[q_ == 1] = 1
+        else:
+            ci_low = ci_low if (q_ != 0) else 0
+            ci_upp = ci_upp if (q_ != 1) else 1
+
     elif method == 'agresti_coull':
         crit = stats.norm.isf(alpha / 2.)
         nobs_c = nobs + crit**2
@@ -119,6 +141,19 @@ def func(qi):
 
     else:
         raise NotImplementedError('method "%s" is not available' % method)
+
+    if method in ['normal', 'agresti_coull']:
+        ci_low = np.clip(ci_low, 0, 1)
+        ci_upp = np.clip(ci_upp, 0, 1)
+    if pd_index is not None and np.ndim(ci_low) > 0:
+        import pandas as pd
+        if np.ndim(ci_low) == 1:
+            ci_low = pd.Series(ci_low, index=pd_index)
+            ci_upp = pd.Series(ci_upp, index=pd_index)
+        if np.ndim(ci_low) == 2:
+            ci_low = pd.DataFrame(ci_low, index=pd_index)
+            ci_upp = pd.DataFrame(ci_upp, index=pd_index)
+
     return ci_low, ci_upp
 
 


Test patch:
diff --git a/statsmodels/stats/tests/results/results_proportion.py b/statsmodels/stats/tests/results/results_proportion.py
--- a/statsmodels/stats/tests/results/results_proportion.py
+++ b/statsmodels/stats/tests/results/results_proportion.py
@@ -87,3 +87,40 @@ class Holder(object):
      0.974623779100809, 0.974626983311416, 0.974392083257476,
      0.972617354399236
     ])
+
+# > bci = binom.confint(x = c(1), n = 30, tol = 1e-8)
+res_binom[(1, 30)].ci_low = np.array([
+    -8.305484e-03, -3.090070e-02,  6.903016e-05, 2.494567e-03,
+     8.435709e-04,  4.675346e-03,  3.475014e-03, 3.012987e-03,
+     1.932430e-03,  1.742467e-03,  5.908590e-03])
+
+res_binom[(1, 30)].ci_upp = np.array([
+    0.18091798, 0.09756737, 0.12314380, 0.14513807,
+    0.17216946, 0.20200244, 0.16637241, 0.13868254,
+    0.13868375, 0.19053022, 0.16670391])
+
+# > bci = binom.confint(x = c(29), n = 30, tol = 1e-8)
+res_binom[(29, 30)].ci_low = np.array([
+    0.8190820, 0.9024326, 0.8768562, 0.7860836,
+    0.8278305, 0.7979976, 0.8336276, 0.8613175,
+    0.8613162, 0.8094698, 0.8332961])
+res_binom[(29, 30)].ci_upp = np.array([
+    1.0083055, 1.0309007, 0.9999310, 0.9952363,
+    0.9991564, 0.9953247, 0.9965250, 0.9969870,
+    0.9980676, 0.9982575, 0.9940914])
+
+# > bci = binom.confint(x = c(0), n = 30, tol = 1e-8)
+    # this ci_low clips one negative value to 0
+res_binom[(0, 30)].ci_low = np.zeros(11)
+res_binom[(0, 30)].ci_upp = np.array([
+    0.13471170, 0.00000000, 0.06151672, 0.11570331,
+    0.11570331, 0.11570331, 0.11570331, 0.10402893,
+    0.06201781, 0.14132048, 0.11351339])
+
+# > bci = binom.confint(x = c(30), n = 30, tol = 1e-8)
+res_binom[(30, 30)].ci_low = np.array([
+    0.8652883, 1.0000000, 0.9384833, 0.8842967,
+    0.8842967, 0.8842967, 0.8842967, 0.8959711,
+    0.9379822, 0.8586795, 0.8864866])
+    # this ci_upp clips one value > 1
+res_binom[(30, 30)].ci_upp = np.ones(11)
diff --git a/statsmodels/stats/tests/test_proportion.py b/statsmodels/stats/tests/test_proportion.py
--- a/statsmodels/stats/tests/test_proportion.py
+++ b/statsmodels/stats/tests/test_proportion.py
@@ -8,6 +8,7 @@
 import warnings
 
 import numpy as np
+import pandas as pd
 from numpy.testing import (assert_almost_equal, assert_equal, assert_array_less,
                            assert_raises, assert_allclose)
 
@@ -20,30 +21,77 @@
 class Holder(object):
     pass
 
-
+probci_methods = {'agresti_coull' : 'agresti-coull',
+                  'normal' : 'asymptotic',
+                  'beta' : 'exact',
+                  'wilson' : 'wilson',
+                  'jeffreys' : 'bayes'
+                  }
 
 def test_confint_proportion():
     from .results.results_proportion import res_binom, res_binom_methods
-    methods = {'agresti_coull' : 'agresti-coull',
-               'normal' : 'asymptotic',
-               'beta' : 'exact',
-               'wilson' : 'wilson',
-               'jeffrey' : 'bayes'
-               }
+
 
     for case in res_binom:
         count, nobs = case
-        for method in methods:
-            idx = res_binom_methods.index(methods[method])
+        for method in probci_methods:
+            idx = res_binom_methods.index(probci_methods[method])
             res_low = res_binom[case].ci_low[idx]
             res_upp = res_binom[case].ci_upp[idx]
             if np.isnan(res_low) or np.isnan(res_upp):
                 continue
+            if (count == 0 or count == nobs) and method == 'jeffreys':
+                # maybe a bug or different corner case definition
+                continue
+            if method == 'jeffreys' and nobs == 30:
+                # something is strange in extreme case e.g 0/30 or 1/30
+                continue
             ci = proportion_confint(count, nobs, alpha=0.05, method=method)
-
+            # we impose that confint is in [0, 1]
+            res_low = max(res_low, 0)
+            res_upp = min(res_upp, 1)
             assert_almost_equal(ci, [res_low, res_upp], decimal=6,
                                 err_msg=repr(case) + method)
 
+
+def test_confint_proportion_ndim():
+    # check that it works with 1-D, 2-D and pandas
+
+    count = np.arange(6).reshape(2, 3)
+    nobs = 10 * np.ones((2, 3))
+
+    count_pd = pd.DataFrame(count)
+    nobs_pd =  pd.DataFrame(nobs)
+
+    for method in probci_methods:
+        ci_arr = proportion_confint(count, nobs, alpha=0.05, method=method)
+        ci_pd = proportion_confint(count_pd, nobs_pd, alpha=0.05,
+                                   method=method)
+        assert_allclose(ci_arr, (ci_pd[0].values, ci_pd[1].values), rtol=1e-13)
+        # spot checking one value
+        ci12 = proportion_confint(count[1, 2], nobs[1, 2], alpha=0.05,
+                                  method=method)
+        assert_allclose((ci_pd[0].values[1, 2], ci_pd[1].values[1, 2]), ci12,
+                        rtol=1e-13)
+        assert_allclose((ci_arr[0][1, 2], ci_arr[1][1, 2]), ci12, rtol=1e-13)
+
+        # check that lists work as input
+        ci_li = proportion_confint(count.tolist(), nobs.tolist(), alpha=0.05,
+                                   method=method)
+        assert_allclose(ci_arr, (ci_li[0], ci_li[1]), rtol=1e-13)
+
+        # check pandas Series, 1-D
+        ci_pds = proportion_confint(count_pd.iloc[0], nobs_pd.iloc[0],
+                                    alpha=0.05, method=method)
+        assert_allclose((ci_pds[0].values, ci_pds[1].values),
+                        (ci_pd[0].values[0], ci_pd[1].values[0]), rtol=1e-13)
+
+        # check scalar nobs, verifying one value
+        ci_arr2 = proportion_confint(count, nobs[1, 2], alpha=0.05,
+                                     method=method)
+        assert_allclose((ci_arr2[0][1, 2], ci_arr[1][1, 2]), ci12, rtol=1e-13)
+
+
 def test_samplesize_confidenceinterval_prop():
     #consistency test for samplesize to achieve confidence_interval
     nobs = 20