Repository: statsmodels/statsmodels

Issue:
Sampling of zero-inflated models with high means is very slow
Hi,

I’ve been using the zero-inflated distributions of StatsModels for a while. Works well! I have notice that the sampling is particularly slow when the mean(parameter mu) is high.

When timing the sampling from two zipoisson with a mean of 10 and 10000. We notice that the sampling is slower when the mean is greater.

```
import timeit
import statsmodels.distributions as sm

print(timeit.timeit('sm.zipoisson(mu=10000, w=0.3).rvs(size=100)', number=10, setup="import statsmodels.distributions as sm"))
print(timeit.timeit('sm.zipoisson(mu=10, w=0.3).rvs(size=100)', number=10, setup="import statsmodels.distributions as sm"))
```
The above snippet of code returns the following two lines with :
```
368.92
0.81
```
I have tested on W10 and OSX with StatsModels `v0.11.1` and `0.10.1` respectively with SciPy `1.3.1` under the hood.

Sampling from a `zipoisson` with `mu = 10000`(or any big `mu`) is very very slow. After profiling and looking at the code of the `.rvs()` method of StatsModels class `zipoisson_gen` that inherits from SciPy’s `rv_discrete`, I think that the slowdown comes from the use of the inverse cdf algorithm in SciPy’s`.rvs()` to generate samples from the `pmf` and a uniform distribution.

SciPy often overwrites this method with a quicker way to compute the samples depending on the distribution.s 
I thought we could do this for `zipoisson` and `zinegbin` by multiplying a binary vector with a vector from the non-zero-inflated distribution; something like:
```
        # Bernouilli of parameter 1-w
zero_inflated_sampling = np.random.binomial(1, 1 - w, size=size)
        # then we sample from the Poisson of parameter mu
distibution_sampling = np.random.poisson(mu, size=size)
        # we multiply both to obtain the Zero-inflated Poisson
samples = zero_inflated_sampling * distibution_sampling
```
This is more than a thousand time faster and still complies with the definition of a zero-inflated distribution.
I’d be interested to look at it. What are your thoughts?
Thanks



Discussion:
We are definitely interested in adding additional methods to the distributions.

We didn't implement rvs because the code/PR was only focused on estimation, and we didn't have a sampling usecase.

Also cdf could be implemented in an efficient way using the mixing.

Generalized Poisson might be more difficult, it's not available in numpy/scipy, and I never looked at distribution properties other than loglike/pdf
While working on this issue, I noticed a weird behavior from `test_count_model.py`.

With the latest master branch all tests are green(40s to test this file). 
The `nobs`(l _215_, l _318_ & l _430_ in the code) variables that dictate the size of the `exog` and `endog` of the testing scenario are set to 200, 200 and 500 for `zip`, `zigp` and `zinb`. If we set them to 2000, 2000 and 5000 then six tests are failing(50s to test). If we set them to 20000, 20000 and 50000 then four tests are failing(200s to test). So simply changing nobs sparks failures.


I found what I believe are small bugs in the tests code to resolve four of the six failing tests; so that 20000, 20000 and 50000 is all green now.
 
I still got an issue with two of them, one in `zigp` and one in `zinb`:

```
________________________________ TestZeroInflatedGeneralizedPoisson_predict.test_mean _________________________________

self = <statsmodels.discrete.tests.test_count_model.TestZeroInflatedGeneralizedPoisson_predict object at 0x0000019265143F98>

    def test_mean(self):
        assert_allclose(self.res.predict().mean(), self.endog.mean(),
>                       atol=1e-4, rtol=1e-4)
E       AssertionError:
E       Not equal to tolerance rtol=0.0001, atol=0.0001
E
E       Mismatch: 100%
E       Max absolute difference: 0.00171753
E       Max relative difference: 0.00190731
E        x: array(0.902218)
E        y: array(0.9005)

statsmodels\discrete\tests\test_count_model.py:329: AssertionError
```
 and 

```
self = <statsmodels.discrete.tests.test_count_model.TestZeroInflatedNegativeBinomialP_predict object at 0x0000019265295438>

    def test_predict_generic_zi(self):
        # These tests do not use numbers from other packages.
        # Tests are on closeness of estimated to true/DGP values
        # and theoretical relationship between quantities
        res = self.res
        endog = self.endog
        exog = self.res.model.exog
[...]
        probs_main_unique = res.predict(exog=[[1, 0], [1, 1]],
                                        exog_infl=np.asarray([[1], [1]]),
                                        which='prob-main')
        probs_main = res.predict(which='prob-main')
        probs_main[[0,-1]]
        assert_allclose(probs_main_unique, probs_main[[0,-1]],  rtol=1e-10)
>       assert_allclose(probs_main_unique, 1 - prob_infl, atol=0.01)
E       AssertionError:
E       Not equal to tolerance rtol=1e-07, atol=0.01
E
E       Mismatch: 100%
E       Max absolute difference: 0.01677652
E       Max relative difference: 0.01973709
E        x: array([0.833223, 0.833223])
E        y: array(0.85)

statsmodels\discrete\tests\test_count_model.py:504: AssertionError
```
The weird part is that those two tests are not failing for 200, 200, 500 nor  20000, 20000 and 50000 but are indeed failing for  2000, 2000 and 5000. 

I've seen on #3928 that it is not the first time and I don't really know if those error margins are acceptable or if it shows a real failure somewhere in the code of count models.


we don't have confidence intervals and standard errors for those predictions yet. so it's difficult to tell what the precision should be.

If just a few cases fail, then it could be just "unlucky" random numbers.

AFAIR, `sm.distributions.zinegbin.rvs` has not been separately verified. I don't know how good that one is with generic rvs.

Resolving patch:
diff --git a/statsmodels/distributions/discrete.py b/statsmodels/distributions/discrete.py
--- a/statsmodels/distributions/discrete.py
+++ b/statsmodels/distributions/discrete.py
@@ -1,5 +1,5 @@
 import numpy as np
-from scipy.stats import rv_discrete, nbinom
+from scipy.stats import rv_discrete, nbinom, poisson
 from scipy.special import gammaln
 from statsmodels.compat.scipy import _lazywhere
 
@@ -39,6 +39,30 @@ def _logpmf(self, x, mu, w):
     def _pmf(self, x, mu, w):
         return np.exp(self._logpmf(x, mu, w))
 
+    def _cdf(self, x, mu, w):
+        # construct cdf from standard poisson's cdf and the w inflation of zero
+        return w + poisson(mu=mu).cdf(x) * (1 - w)
+
+    def _ppf(self, q, mu, w):
+        # we just translated and stretched q to remove zi
+        q_mod = (q - w) / (1 - w)
+        x = poisson(mu=mu).ppf(q_mod)
+        # set to zero if in the zi range
+        x[q < w] = 0
+        return x
+
+    def _mean(self, mu, w):
+        return (1 - w) * mu
+
+    def _var(self, mu, w):
+        dispersion_factor = 1 + w * mu
+        var = (dispersion_factor * self._mean(mu, w)).mean()
+        return var
+
+    def _moment(self, n, mu, w):
+        return (1 - w) * poisson(mu).moment(n)
+
+
 zipoisson = zipoisson_gen(name='zipoisson',
                           longname='Zero Inflated Poisson')
 
@@ -58,9 +82,21 @@ def _logpmf(self, x, mu, alpha, p, w):
     def _pmf(self, x, mu, alpha, p, w):
         return np.exp(self._logpmf(x, mu, alpha, p, w))
 
-zigenpoisson = zigeneralizedpoisson_gen(name='zigenpoisson',
+    def _mean(self, mu, alpha, p, w):
+        return (1 - w) * mu
+
+    def _var(self, mu, alpha, p, w):
+        p = p - 1
+        dispersion_factor = (1 + alpha * mu ** p) ** 2 + w * mu
+        var = (dispersion_factor * self._mean(mu, alpha, p, w)).mean()
+        return var
+
+
+zigenpoisson = zigeneralizedpoisson_gen(
+    name='zigenpoisson',
     longname='Zero Inflated Generalized Poisson')
 
+
 class zinegativebinomial_gen(rv_discrete):
     '''Zero Inflated Generalized Negative Binomial distribution
     '''
@@ -78,6 +114,33 @@ def _logpmf(self, x, mu, alpha, p, w):
     def _pmf(self, x, mu, alpha, p, w):
         return np.exp(self._logpmf(x, mu, alpha, p, w))
 
+    def _cdf(self, x, mu, alpha, p, w):
+        s, p = self.convert_params(mu, alpha, p)
+        # construct cdf from standard negative binomial cdf
+        # and the w inflation of zero
+        return w + nbinom.cdf(x, s, p) * (1 - w)
+
+    def _ppf(self, q, mu, alpha, p, w):
+        s, p = self.convert_params(mu, alpha, p)
+        # we just translated and stretched q to remove zi
+        q_mod = (q - w) / (1 - w)
+        x = nbinom.ppf(q_mod, s, p)
+        # set to zero if in the zi range
+        x[q < w] = 0
+        return x
+
+    def _mean(self, mu, alpha, p, w):
+        return (1 - w) * mu
+
+    def _var(self, mu, alpha, p, w):
+        dispersion_factor = 1 + alpha * mu ** (p - 1) + w * mu
+        var = (dispersion_factor * self._mean(mu, alpha, p, w)).mean()
+        return var
+
+    def _moment(self, n, mu, alpha, p, w):
+        s, p = self.convert_params(mu, alpha, p)
+        return (1 - w) * nbinom.moment(n, s, p)
+
     def convert_params(self, mu, alpha, p):
         size = 1. / alpha * mu**(2-p)
         prob = size / (size + mu)


Test patch:
diff --git a/statsmodels/distributions/tests/test_discrete.py b/statsmodels/distributions/tests/test_discrete.py
--- a/statsmodels/distributions/tests/test_discrete.py
+++ b/statsmodels/distributions/tests/test_discrete.py
@@ -56,6 +56,32 @@ def test_logpmf(self):
         zipoisson_logpmf = sm.distributions.zipoisson.logpmf(7, 3, 0.1)
         assert_allclose(poisson_logpmf, zipoisson_logpmf, rtol=5e-2, atol=5e-2)
 
+    def test_cdf_zero(self):
+        poisson_cdf = poisson.cdf(3, 2)
+        zipoisson_cdf = sm.distributions.zipoisson.cdf(3, 2, 0)
+        assert_allclose(poisson_cdf, zipoisson_cdf, rtol=1e-12)
+
+    def test_ppf_zero(self):
+        poisson_ppf = poisson.ppf(5, 1)
+        zipoisson_ppf = sm.distributions.zipoisson.ppf(5, 1, 0)
+        assert_allclose(poisson_ppf, zipoisson_ppf, rtol=1e-12)
+
+    def test_mean_var(self):
+        poisson_mean, poisson_var = poisson.mean(12), poisson.var(12)
+        zipoisson_mean = sm.distributions.zipoisson.mean(12, 0)
+        zipoisson_var = sm.distributions.zipoisson.mean(12, 0)
+        assert_allclose(poisson_mean, zipoisson_mean, rtol=1e-10)
+        assert_allclose(poisson_var, zipoisson_var, rtol=1e-10)
+
+    def test_moments(self):
+        poisson_m1, poisson_m2 = poisson.moment(1, 12), poisson.moment(2, 12)
+        zip_m0 = sm.distributions.zipoisson.moment(0, 12, 0)
+        zip_m1 = sm.distributions.zipoisson.moment(1, 12, 0)
+        zip_m2 = sm.distributions.zipoisson.moment(2, 12, 0)
+        assert_allclose(1, zip_m0, rtol=1e-10)
+        assert_allclose(poisson_m1, zip_m1, rtol=1e-10)
+        assert_allclose(poisson_m2, zip_m2, rtol=1e-10)
+
 
 class TestZIGeneralizedPoisson(object):
     def test_pmf_zero(self):
@@ -95,6 +121,36 @@ def test_logpmf_p2(self):
         tnb_logpmf = sm.distributions.zinegbin.logpmf(200, 10, 1, 2, 0.01)
         assert_allclose(nb_logpmf, tnb_logpmf, rtol=1e-2, atol=1e-2)
 
+    def test_cdf_p2(self):
+        n, p = sm.distributions.zinegbin.convert_params(30, 0.1, 2)
+        nbinom_cdf = nbinom.cdf(10, n, p)
+        zinbinom_cdf = sm.distributions.zinegbin.cdf(10, 30, 0.1, 2, 0)
+        assert_allclose(nbinom_cdf, zinbinom_cdf, rtol=1e-12, atol=1e-12)
+
+    def test_ppf_p2(self):
+        n, p = sm.distributions.zinegbin.convert_params(100, 1, 2)
+        nbinom_ppf = nbinom.ppf(0.27, n, p)
+        zinbinom_ppf = sm.distributions.zinegbin.ppf(0.27, 100, 1, 2, 0)
+        assert_allclose(nbinom_ppf, zinbinom_ppf, rtol=1e-12, atol=1e-12)
+
+    def test_mran_var_p2(self):
+        n, p = sm.distributions.zinegbin.convert_params(7, 1, 2)
+        nbinom_mean, nbinom_var = nbinom.mean(n, p), nbinom.var(n, p)
+        zinb_mean = sm.distributions.zinegbin.mean(7, 1, 2, 0)
+        zinb_var = sm.distributions.zinegbin.var(7, 1, 2, 0)
+        assert_allclose(nbinom_mean, zinb_mean, rtol=1e-10)
+        assert_allclose(nbinom_var, zinb_var, rtol=1e-10)
+
+    def test_moments_p2(self):
+        n, p = sm.distributions.zinegbin.convert_params(7, 1, 2)
+        nb_m1, nb_m2 = nbinom.moment(1, n, p), nbinom.moment(2, n, p)
+        zinb_m0 = sm.distributions.zinegbin.moment(0, 7, 1, 2, 0)
+        zinb_m1 = sm.distributions.zinegbin.moment(1, 7, 1, 2, 0)
+        zinb_m2 = sm.distributions.zinegbin.moment(2, 7, 1, 2, 0)
+        assert_allclose(1, zinb_m0, rtol=1e-10)
+        assert_allclose(nb_m1, zinb_m1, rtol=1e-10)
+        assert_allclose(nb_m2, zinb_m2, rtol=1e-10)
+
     def test_pmf(self):
         n, p = sm.distributions.zinegbin.convert_params(1, 0.9, 1)
         nb_logpmf = nbinom.pmf(2, n, p)
@@ -106,3 +162,44 @@ def test_logpmf(self):
         nb_logpmf = nbinom.logpmf(2, n, p)
         tnb_logpmf = sm.distributions.zinegbin.logpmf(2, 5, 1, 1, 0.005)
         assert_allclose(nb_logpmf, tnb_logpmf, rtol=1e-2, atol=1e-2)
+
+    def test_cdf(self):
+        n, p = sm.distributions.zinegbin.convert_params(1, 0.9, 1)
+        nbinom_cdf = nbinom.cdf(2, n, p)
+        zinbinom_cdf = sm.distributions.zinegbin.cdf(2, 1, 0.9, 2, 0)
+        assert_allclose(nbinom_cdf, zinbinom_cdf, rtol=1e-12, atol=1e-12)
+
+    def test_ppf(self):
+        n, p = sm.distributions.zinegbin.convert_params(5, 1, 1)
+        nbinom_ppf = nbinom.ppf(0.71, n, p)
+        zinbinom_ppf = sm.distributions.zinegbin.ppf(0.71, 5, 1, 1, 0)
+        assert_allclose(nbinom_ppf, zinbinom_ppf, rtol=1e-12, atol=1e-12)
+
+    def test_convert(self):
+        n, p = sm.distributions.zinegbin.convert_params(25, 0.85, 2)
+        n_true, p_true = 1.1764705882352942, 0.04494382022471911
+        assert_allclose(n, n_true, rtol=1e-12, atol=1e-12)
+        assert_allclose(p, p_true, rtol=1e-12, atol=1e-12)
+
+        n, p = sm.distributions.zinegbin.convert_params(7, 0.17, 1)
+        n_true, p_true = 41.17647058823529, 0.8547008547008547
+        assert_allclose(n, n_true, rtol=1e-12, atol=1e-12)
+        assert_allclose(p, p_true, rtol=1e-12, atol=1e-12)
+
+    def test_mean_var(self):
+        n, p = sm.distributions.zinegbin.convert_params(9, 1, 1)
+        nbinom_mean, nbinom_var = nbinom.mean(n, p), nbinom.var(n, p)
+        zinb_mean = sm.distributions.zinegbin.mean(9, 1, 1, 0)
+        zinb_var = sm.distributions.zinegbin.var(9, 1, 1, 0)
+        assert_allclose(nbinom_mean, zinb_mean, rtol=1e-10)
+        assert_allclose(nbinom_var, zinb_var, rtol=1e-10)
+
+    def test_moments(self):
+        n, p = sm.distributions.zinegbin.convert_params(9, 1, 1)
+        nb_m1, nb_m2 = nbinom.moment(1, n, p), nbinom.moment(2, n, p)
+        zinb_m0 = sm.distributions.zinegbin.moment(0, 9, 1, 1, 0)
+        zinb_m1 = sm.distributions.zinegbin.moment(1, 9, 1, 1, 0)
+        zinb_m2 = sm.distributions.zinegbin.moment(2, 9, 1, 1, 0)
+        assert_allclose(1, zinb_m0, rtol=1e-10)
+        assert_allclose(nb_m1, zinb_m1, rtol=1e-10)
+        assert_allclose(nb_m2, zinb_m2, rtol=1e-10)