Repository: statsmodels/statsmodels

Issue:
proportions_chisquare produces erroneous p-value with large sample size
Function Path: from statsmodels.stats.proportion.proportions_chisquare

With large sample sizes, proportions_chisquare produces a p-value of 1, even though the difference in proportions is statistically significantly large, that should yield a p-value close to 0 (which is what we get when we use the scipy.stats.chi2_contingency function with the same input parameters). We present an example of this below.

Say we have 2 groups "a" and "b", with the following trials and successes.
nsucc_groupa = 640
ntrials_groupa = 679
nsucc_groupb = 100
ntrials_groupb = 53000

```
INPUT: proportions_chisquare([nsucc_groupa,nsucc_groupb],[ntrials_groupa,ntrials_groupb])[1]
OUTPUT:  1

INPUT: chi2_contingency([[nsucc_groupa,nsucc_groupb],[ntrials_groupa-nsucc_groupa,ntrials_groupb-nsucc_groupb]])[1]
OUTPUT:  0
```

Is this behavior expected? If so what's the underlying reason for this observation?


Discussion:
BUG counts or nobs needs to be float, then pvalue is 0

problem is in the helper function, after that it just delegates to scipy stats

```
_table_proportion(count.astype(int), nobs.astype(int))
(array([[  640,    39],
        [  100, 52900]]),
 array([[ 9.36045753e+00,  6.69639542e+02],
        [ 7.30639542e+02, -2.77426982e+04]]),
 2)

_table_proportion(count.astype(float), nobs.astype(float))
(array([[6.40e+02, 3.90e+01],
        [1.00e+02, 5.29e+04]]),
 array([[9.36045753e+00, 6.69639542e+02],
        [7.30639542e+02, 5.22693605e+04]]),
 2)
```

using
`count, nobs = np.asarray([nsucc_groupa,nsucc_groupb]), np.asarray([ntrials_groupa,ntrials_groupb])`
I guess it's an int overflow, that causes a negative integer. and numpy is not upcasting by default.

using int64 works for the example

```
count.astype(int).dtype
dtype('int32')

_table_proportion(count.astype(np.int64), nobs.astype(np.int64))
(array([[  640,    39],
        [  100, 52900]], dtype=int64),
 array([[9.36045753e+00, 6.69639542e+02],
        [7.30639542e+02, 5.22693605e+04]]),
 2)
```

the BUG is that `proportions_chisquare` is missing asarray, `asarray` defaults to float64 casting.
```
    count = np.asarray(count)
    nobs = np.asarray(nobs)
```

However, if the numpy array has dtype with int dtype, then asarray doesn't convert.
In that case we would still need to upcast to at least float, or better float64

```
np.asarray(count.astype(np.int8)).dtype
dtype('int8')
```
something is still weird

atleast_1d(nobs) is used in proportions_chisquare and it upcasts in my notebook

```
np.atleast_1d([ntrials_groupa, ntrials_groupb]).dtype
dtype('float64')
```

but not for counts
```
np.atleast_1d([nsucc_groupa,nsucc_groupb]).dtype
dtype('int32')
```

This problem with `int` dtype might affect other functions for count models, when there are no computations early on that upcast the data.

for whatever reason atleast_1d doesn't upcast

```
res = proportions_chisquare([nsucc_groupa,nsucc_groupb],[ntrials_groupa,ntrials_groupb])
res[2][0].dtype
dtype('int32')

table = res[2][0]
table.sum()
53679

table.sum(0) * table.sum(1)[:,None]
array([[     502460,    35945581],
       [   39220000, -1489200296]])
```

In the actual computation, conversion to float comes too late, e.g. reversing sequence works to force computation to be in float.

```
1. / table.sum() * table.sum(0) * table.sum(1)[:,None]
array([[9.36045753e+00, 6.69639542e+02],
       [7.30639542e+02, 5.22693605e+04]])
```

**edit** `sum` could also overflow if table is small int type, so that wouldn't help in those cases.
We need a float64 table.

aside: since division between ints produces a float, we stopped using `1.*` as quick casting  trick.

Obviously, we still need it for upcasting small integers.
@ronitshaw Thanks for reporting. I never thought of checking examples like this.

We are not upcasting by default, because there are usecases where users want lower precision to save memory.
But in these cases, we still need to make sure computations are at full precision, and memory should not be an issue when we use summary statistics like counts.
another thought

I haven't been closely following numpy developement in a while. 
Over the longer term, casting rules change, and code that initially worked, might stop working correctly.

As defensive programming, we should/could make more explicit upcasting, instead of relying on implicit upcasting.
e.g. use asarray with dtype= `at least float64`
(we might want to allow complex numbers in some cases by default)

e.g. sum has a dtype option for the accumulator

```
np.array([990000000, 2100000000], np.int32).sum()
-1204967296

np.array([990000000, 2100000000], np.int32).sum(dtype=np.float64)
3090000000.0
```

at least dtype can be implemented with np.promote_types

```
np.promote_types(np.array([990000000, 2100000000], np.int32).dtype, np.float64)
dtype('float64')

np.promote_types(np.array([990000000, 2100000000], np.complex128).dtype, np.float64)
dtype('complex128')
```

Resolving patch:
diff --git a/statsmodels/stats/proportion.py b/statsmodels/stats/proportion.py
--- a/statsmodels/stats/proportion.py
+++ b/statsmodels/stats/proportion.py
@@ -748,8 +748,11 @@ def _table_proportion(count, nobs):
     recent scipy has more elaborate contingency table functions
 
     '''
+    count = np.asarray(count)
+    dt = np.promote_types(count.dtype, np.float64)
+    count = np.asarray(count, dtype=dt)
     table = np.column_stack((count, nobs - count))
-    expected = table.sum(0) * table.sum(1)[:,None] * 1. / table.sum()
+    expected = table.sum(0) * table.sum(1)[:, None] * 1. / table.sum()
     n_rows = table.shape[0]
     return table, expected, n_rows
 


Test patch:
diff --git a/statsmodels/stats/tests/test_proportion.py b/statsmodels/stats/tests/test_proportion.py
--- a/statsmodels/stats/tests/test_proportion.py
+++ b/statsmodels/stats/tests/test_proportion.py
@@ -583,6 +583,16 @@ def test_proportion_ztests():
     # test only p-value
     assert_almost_equal(res1[1], res2[1], decimal=13)
 
+    # test with integers, issue #7603
+    res1 = smprop.proportions_ztest(np.asarray([15, 10]),
+                                    np.asarray([20, 50000]),
+                                    value=0, prop_var=None)
+    res2 = smprop.proportions_chisquare(np.asarray([15, 10]),
+                                        np.asarray([20, 50000]))
+    # test only p-value
+    assert_almost_equal(res1[1], res2[1], decimal=13)
+    assert_array_less(0, res2[-1][1])  # expected should be positive
+
 
 def test_confint_2indep():
     # alpha = 0.05