Repository: statsmodels/statsmodels

Issue:
smf.mixedlm with small sample sizes, integer data, missing values, and skwed distribution.
I'm using statsmodels' MixedLM a lot recently, and some of the data I am working with is becoming quirkier and quirkier. I am wondering whether any of the following data wold be inappropriate for the function. Some of my data has:

1. as little as 2 values in one of the (e.g.) time:treatment bins (and e.g. up to 8 in others). I guess this is a compounded issue of small sample size and missing data.
2. discrete (integer) dependent variable values, e.g. event per time or feature per area counts. Technically, these could also be expressed as floats, and seem continuous, but I would still count them as discrete.
3. a squeezed distribution towards 0 and a long tail extending upwards. Is this still “normal enough”? Is there any test I can run beforehand, or would statsmodels just warn me?

In case any of these characteristics preclude usage of MixedLM with my data, is there any other modelling tool in statsmodels (particularly one which supports repeated measures/random effects) which I can use? 


Discussion:
For #1 you might have low power but there is no reason that this would
impact a mixed linear model any more than any other type of model.

For #2, we have Poisson mixed GLM
(statsmodels.genmod.bayes_mixed_glm.PoissonMixedGLM).  It works but is
still being refined.

For #3, hard to say without more information.  There is no built-in
specification testing so you will not get a warning.  You might want to log
or sqrt transform the data and then use mixed linear model

On Wed, Jul 18, 2018 at 7:05 PM Horea Christian <notifications@github.com>
wrote:

> I'm using statsmodels' MixedLM a lot recently, and some of the data I am
> working with is becoming quirkier and quirkier. I am wondering whether any
> of the following data wold be inappropriate for the function:
>
>    1. Some of my data has as little as 2 values in some of the (e.g.)
>    time:treatment bins (and e.g. up to 8 in others).
>    2. Discrete (integer) dependent variable values, e.g. event per time
>    or feature per area counts. Technically, these could also be expressed as
>    floats, and seem continuous, but I would still count them as discrete.
>    3. A lot of my data has a squeezed distribution towards 0 and a long
>    tail extending upwards. Is this still normal enough? Is there any test I
>    can run beforehand, or would statsmodels just warn me?
>
> In case any of these characteristics preclude usage of MixedLM with my
> data, is there any other modelling tool in statsmodels (particularly one
> which supports repeated measures/random effects) which I can use?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/4801>, or mute the
> thread
> <https://github.com/notifications/unsubscribe-auth/ACiww2i032SBaKThk4hHPuSMTfaqjO8Oks5uH79AgaJpZM4VVgId>
> .
>

@kshedden --- do the answers ally the same way to OLS?
If you do not have repeated measures or other form of dependence in the data, you can use Poisson GLM or negative binomial GLM to handle hetroscedasticity   Transforming the response variable is another option in many cases.  It is not a panacea, but handles certain forms of heteroscedasticity well.

Other tools to look at would be GEE (for handling dependence/repeated measures), splines (for handling nonlinearities), and penalized estimation (for handling sparsity and collinearity).  There is some GAM code and GP code in branches, not sure how mature or applicable it is for you.   I mentioned the BayesMixedGLM code that handles heteroscedasticity and dependence.  It is reasonably stable now.
Sadly (or rather luckily) I do have repeated measures.

What about [PoissonBayesMixedGLM](http://www.statsmodels.org/dev/generated/statsmodels.genmod.bayes_mixed_glm.PoissonBayesMixedGLM.html)? Sounds like the combo that would address all my problems?

Also, are there any (more) usage examples for the BayesMixedGLM classes? I'd be curious to see how all of the other parameters might be used.
You can use the test file to see some usage examples.  I don't have any
other examples at this point that I can share.

On Thu, Jul 19, 2018 at 2:55 PM Horea Christian <notifications@github.com>
wrote:

> Sadly (or rather luckily) I do have repeated measures.
>
> What about PoissonBayesMixedGLM
> <http://www.statsmodels.org/dev/generated/statsmodels.genmod.bayes_mixed_glm.PoissonBayesMixedGLM.html>?
> Sounds like the combo that would address all my problems?
>
> Also, are there any (more) usage examples for the BayesMixedGLM classes?
> I'd be curious to see how all of the other parameters might be used.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/4801#issuecomment-406379350>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACiww8qvpBMuJfNxSON89MtnqKLQ3DReks5uINYbgaJpZM4VVgId>
> .
>

@kshedden as demonstrated in #4807 I am having slight difficulties with adopting a bayesian angle here. Additionally, it would be weird to analyze all my continuous data with a frequentist GLM, and the discrete one with a bayesian GLM.

So I guess what I am looking for is a frequentist Poisson Mixed GLM --- one which allowes me to run an omnibus ANOVAcontrast and post-hoc t-tests. Does statsmodels provide such a function? I have browsed the code a bit, and it seems the only poisson Mixed GLM is PoissonBayesMixedGLM --- is that correct?
I think all the issues you found with the Poisson mixed GLM have been fixed
by some code I merged about a week ago.  If you find otherwise please let
me know.

Yes, this the only mixed GLM we have.

On Sat, Aug 4, 2018 at 4:50 PM Horea Christian <notifications@github.com>
wrote:

> @kshedden <https://github.com/kshedden> as demonstrated in #4807
> <https://github.com/statsmodels/statsmodels/issues/4807> I am having
> slight difficulties with adopting a bayesian angle here. Additionally, it
> would be weird to analyze all my continuous data with a frequentist GLM,
> and the discrete one with a bayesian GLM.
>
> So I guess what I am looking for is a frequentist Poisson Mixed GLM ---
> one which allowes me to run an omnibus ANOVAcontrast and post-hoc t-tests.
> Does statsmodels provide such a function? I have browsed the code a bit,
> and it seems the only poisson Mixed GLM is PoissonBayesMixedGLM --- is that
> correct?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/4801#issuecomment-410476760>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACiwwyN1l-Nvu1f90zBnD9aLBdVqa4hXks5uNgj6gaJpZM4VVgId>
> .
>

Is there any way to run a t-test or otherwise find out which treatment:time effects signifincatnly differ from each other. The normal approach ([minimal self-contained example](http://chymera.eu/debug/statsmodels/multi_longitudinal_treatment_bayes-1.zip)) fails with:

```
/usr/lib64/python3.6/site-packages/statsmodels/genmod/bayes_mixed_glm.py:483: UserWarning: Laplace fitting did not converge, |gradient|=0.015467
  warnings.warn(msg)
Traceback (most recent call last):
  File "model.py", line 8, in <module>
    print(fit.t_test(np.array([[0,0,1,0,-1]])))
AttributeError: 'BayesMixedGLMResults' object has no attribute 't_test'
```

Inspecting the object with `.__dir__()` doesn't show any supported test methods either:

```
['model', 'params', '_cov_params', 'optim_retvals', 'fe_mean', 'vcp_mean', 'vc_mean', 'fe_sd', 'vcp_sd', 'vc_sd', '__module__', '__doc__', '__init__', 'cov_params', 'summary', 'random_effects', 'predict', '__dict__', '__weakref__', '__repr__', '__hash__', '__str__', '__getattribute__', '__setattr__', '__delattr__', '__lt__', '__le__', '__eq__', '__ne__', '__gt__', '__ge__', '__new__', '__reduce_ex__', '__reduce__', '__subclasshook__', '__init_subclass__', '__format__', '__sizeof__', '__dir__', '__class__']
```

What can I do?


Additionally, while the comparison of treatments to water (but not among each other) seems to work with the [approach you suggested](https://github.com/statsmodels/statsmodels/issues/4807#issuecomment-408621739), I cannot find or produce any significance levels. I have the posterior mean and sd, so I can kind-of say something is within 2SD or not, but that completely ignores sample size.
A posterior standard deviation is analogous to a frequentist standard error
(i.e. it will get smaller with increasing sample size and reflects the
uncertainty about the parameter's value based on all the information in the
data).  So you would be on very solid ground using the posterior mean
divided by the posterior SD as a Z-score, and then base a t-test on it.


On Sat, Aug 4, 2018 at 5:22 PM Horea Christian <notifications@github.com>
wrote:

> Is there any way to run a t-test or otherwise find out which
> treatment:time effects signifincatnly differ from each other. The normal
> approach (minimal self-contained example
> <http://chymera.eu/debug/statsmodels/multi_longitudinal_treatment_bayes-1.zip>)
> fails with:
>
> /usr/lib64/python3.6/site-packages/statsmodels/genmod/bayes_mixed_glm.py:483: UserWarning: Laplace fitting did not converge, |gradient|=0.015467
>   warnings.warn(msg)
> Traceback (most recent call last):
>   File "model.py", line 8, in <module>
>     print(fit.t_test(np.array([[0,0,1,0,-1]])))
> AttributeError: 'BayesMixedGLMResults' object has no attribute 't_test'
>
> Inspecting the object with .__dir__() doesn't show any supported test
> methods either:
>
> ['model', 'params', '_cov_params', 'optim_retvals', 'fe_mean', 'vcp_mean', 'vc_mean', 'fe_sd', 'vcp_sd', 'vc_sd', '__module__', '__doc__', '__init__', 'cov_params', 'summary', 'random_effects', 'predict', '__dict__', '__weakref__', '__repr__', '__hash__', '__str__', '__getattribute__', '__setattr__', '__delattr__', '__lt__', '__le__', '__eq__', '__ne__', '__gt__', '__ge__', '__new__', '__reduce_ex__', '__reduce__', '__subclasshook__', '__init_subclass__', '__format__', '__sizeof__', '__dir__', '__class__']
>
> What can I do?
>
> Additionally, while the comparison of treatments to water (but not among
> each other) seems to work with the approach you suggested
> <https://github.com/statsmodels/statsmodels/issues/4807#issuecomment-408621739>
> --- I can't find or produce any significance levels. I have the posterior
> mean and sd, so I can kind-of say something is within 2SD or not, but that
> completely ignores sample size.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/4801#issuecomment-410478419>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACiww-Poax3HRvwgPCtTn3oj00HDkzQyks5uNhCpgaJpZM4VVgId>
> .
>

Ah, that does make it --- at least conceptually --- easy then. However, putting everything (especially an ANOVA) together by hand would take a while. Does statsmodels have any ANOVA or t-test bindings, which would take mean and SEM as inputs?

Also, on the conceptual level --- am I not throwing the entire robustness-to-heteroscedasticity part out the window, if I am at any step simply summarizing my data as a mean+SEM?
You aren't summarizing the data as mean +/- SEM.  You are fitting the
Poisson mixed model to the data, then summarizing what you know about the
Poisson model's parameters in terms of a point estimate (posterior mean)
and uncertainty level (posterior SD).  This is similar to the frequentist
approach where you summarize what you know about model parameters using a
point estimate (MLE) and uncertainty level (standard error).

On Sat, Aug 4, 2018 at 5:38 PM Horea Christian <notifications@github.com>
wrote:

> Ah, that does make it --- at least conceptually --- easy then. However,
> putting everything (especially an ANOVA) together by hand would take a
> while. Does statsmodels have any ANOVA or t-test bindings, which would take
> mean and SEM as inputs?
>
> Also, on the conceptual level --- am I not throwing the entire
> robustness-to-heteroscedasticity part out the window, if I am at any step
> simply summarizing my data as a mean+SEM?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/4801#issuecomment-410479171>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACiww2zQAsvLV0w8zE4EXf0ufXz_XVHdks5uNhREgaJpZM4VVgId>
> .
>

what about bindings for ANOVA and the t-test? I have tried passing the fit object from my last example to [`anova_lm`](http://www.statsmodels.org/devel/generated/statsmodels.stats.anova.anova_lm.html#statsmodels.stats.anova.anova_lm) but that fails with:

```
    anova = sm.stats.anova_lm(fit, typ=2)
  File "/usr/lib64/python3.6/site-packages/statsmodels/stats/anova.py", line 326, in anova_lm
    return anova_single(model, **kwargs)
  File "/usr/lib64/python3.6/site-packages/statsmodels/stats/anova.py", line 83, in anova_single
    robust)
  File "/usr/lib64/python3.6/site-packages/statsmodels/stats/anova.py", line 178, in anova2_lm_single
    cov = _get_covariance(model, None)
  File "/usr/lib64/python3.6/site-packages/statsmodels/stats/anova.py", line 15, in _get_covariance
    return model.cov_params()
  File "/usr/lib64/python3.6/site-packages/statsmodels/genmod/bayes_mixed_glm.py", line 858, in cov_params
    self.model.vc_names)
TypeError: can only concatenate list (not "NoneType") to list
```
I haven't looked closely at anova_lm, but I think it is for models fit using least squares or MLE.  It's not just a matter of adding bindings.  A basic Z-test for the fixed parameters (regression coefficients) is just this:

```
z = rslt.fe_mean / rslt.fe_sd
p = 2*sp.norm.cdf(-np.abs(z))
```

where sp is scipy.special
For the record, it appears that's actually `scipy.stats.norm.cdf()` on `=scipy-1.1.0`. But in any case that seems to work, thank you!

How would I compute the p-value for the comparison between two factors? I assume I'd just subtract the means --- but what about the standard deviations?
Using the Laplace approximation (`fit` or `fit_map`) you should be able to create a contrast vector `d` having the same length as `result.params` containing one 1 and one -1 in positions corresponding to the two parameters you want to contrast.  Then the point estimate is `np.dot(d, result.params)`, which should give the same value as differencing the two parameter estimates.  The standard deviation for this contrast estimate is

```
np.sqrt(np.dot(d, np.dot(result.cov_params(), d)))
```
(this is a posterior standard deviation so behaves like a standard error in a frequentist analysis).

This approach will not work with the VB mean field approximation (`fit_vb`), which may in some cases give a better approximation to the posterior distribution than the Laplace approximation.  The VB method does not estimate posterior cross-covariances so cannot be used to do inference on parameter contrasts.  However you may be able to reparameterize your model so that what was once a contrast becomes a single parameter, and then the VB method can be used.

Performance trade-offs among different forms of Posterior approximation are not yet well understood.



@kshedden while `fit.summary()` shows only 5 factors (and a 6th entry for the variance term), `fit.params` gives me 39 values. Thus I get:

```
ValueError: shapes (5,) and (39,) not aligned: 5 (dim 0) != 39 (dim 0)
```

Am I doing something wrong?
You get that error when doing what?

On Tue, Aug 7, 2018 at 10:01 PM Horea Christian <notifications@github.com>
wrote:

> @kshedden <https://github.com/kshedden> while fit.summary() shows only 5
> factors (and a 6th entry for the variance term), fit.params gives me 39
> values. Thus I get:
>
> ValueError: shapes (5,) and (39,) not aligned: 5 (dim 0) != 39 (dim 0)
>
> Am I doing something wrong?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/4801#issuecomment-411259052>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACiwwy24QukkZr2ShRTwTs7ufUP6C5EXks5uOkZagaJpZM4VVgId>
> .
>

while running `np.dot`:

```
Traceback (most recent call last):
  File "sprout-countANOVA.py", line 55, in <module>
    mean = np.dot(contrast, fit.params)
ValueError: shapes (5,) and (39,) not aligned: 5 (dim 0) != 39 (dim 0)
```

Is this unexpected? Would you need a minimal working example including data?
Summary only shows the fixed effects parameters and parameters for the
variance structure.  You will need to expand your contrast vector to have
length 39 (the remaining components are the realizations of the random
effects).  The fixed effects parameters come first, so if you are only
contrasting within the fixed effects, you can just pad with zeros.

On Tue, Aug 7, 2018 at 10:44 PM Horea Christian <notifications@github.com>
wrote:

> while running np.dot:
>
> Traceback (most recent call last):
>   File "sprout-countANOVA.py", line 55, in <module>
>     mean = np.dot(contrast, fit.params)
> ValueError: shapes (5,) and (39,) not aligned: 5 (dim 0) != 39 (dim 0)
>
> Is this unexpected? Would you need a minimal working example including
> data?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/4801#issuecomment-411265891>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACiww3IH1EvjZtvnKt9k5z5FOD2JRX24ks5uOlCVgaJpZM4VVgId>
> .
>

@kshedden ok, I'm working on it. But you sure these z-tests are going the correct route? After all, I am dealing with discrete data (the very reason I needed the Poisson GLM), and now all my means ahave values like 0.3274 --- whereas the probability of any non-interger value is actually zero. 
are you sure `fit.cov_params()` is what I need to get the standard deviations? 

`fit.cov_params()` gives me:

```
Traceback (most recent call last):
  File "sprout-countANOVA.py", line 59, in <module>
    print(fit.cov_params())
  File "/usr/lib64/python3.6/site-packages/statsmodels/genmod/bayes_mixed_glm.py", line 858, in cov_params
    self.model.vc_names)
TypeError: can only concatenate list (not "NoneType") to list
```

`fit.cov_params` gives me:

```
<bound method BayesMixedGLMResults.cov_params of <statsmodels.genmod.bayes_mixed_glm.BayesMixedGLMResults object at 0x7f433dde05c0>>
```

`fit._cov_params` is what seems to give me the values I need. Is that correct?
@kshedden I just went ahead and used `fit._cov_params`, and the p-values look about right. Overall rather smaller (so “better”) than I expected, but they vary correctly between conditions which look more similar or less similar.

At any rate, is there any way to compute an omnibus anova? Or do the holm-bonferroni corrected post-hoc t-tests suffice for multiple comparison correction. I actually think they should, but I find it customary in the literature to only reference them if an omnibus ANOVA shows the variance between groups to be significantly greater than the variance within them.

Resolving patch:
diff --git a/statsmodels/genmod/bayes_mixed_glm.py b/statsmodels/genmod/bayes_mixed_glm.py
--- a/statsmodels/genmod/bayes_mixed_glm.py
+++ b/statsmodels/genmod/bayes_mixed_glm.py
@@ -853,10 +853,17 @@ def __init__(self, model, params, cov_params, optim_retvals=None):
         self.vc_sd = np.sqrt(self.vc_sd)
 
     def cov_params(self):
+
         if hasattr(self.model.data, "frame"):
+            # Return the covariance matrix as a dataframe or series
             na = (self.model.fep_names + self.model.vcp_names +
                   self.model.vc_names)
-            return pd.DataFrame(self._cov_params, index=na, columns=na)
+            if self._cov_params.ndim == 2:
+                return pd.DataFrame(self._cov_params, index=na, columns=na)
+            else:
+                return pd.Series(self._cov_params, index=na)
+
+        # Return the covariance matrix as a ndarray
         return self._cov_params
 
     def summary(self):
@@ -1068,7 +1075,7 @@ def __init__(self,
             family=families.Poisson(),
             fep_names=fep_names,
             vcp_names=vcp_names,
-            vc_names=None)
+            vc_names=vc_names)
 
     @classmethod
     def from_formula(cls,


Test patch:
diff --git a/statsmodels/genmod/tests/test_bayes_mixed_glm.py b/statsmodels/genmod/tests/test_bayes_mixed_glm.py
--- a/statsmodels/genmod/tests/test_bayes_mixed_glm.py
+++ b/statsmodels/genmod/tests/test_bayes_mixed_glm.py
@@ -161,6 +161,13 @@ def test_simple_poisson_map():
                 assert_equal(pr2.min() >= 0, True)
                 assert_equal(pr3.min() >= 0, True)
 
+    # Check dimensions and PSD status of cov_params
+    for rslt in rslt1, rslt2:
+        cp = rslt.cov_params()
+        p = len(rslt.params)
+        assert_equal(cp.shape, np.r_[p, p])
+        np.linalg.cholesky(cp)
+
 
 def test_crossed_logit_map():
 
@@ -175,6 +182,12 @@ def test_crossed_logit_map():
         np.zeros_like(rslt.params),
         atol=1e-4)
 
+    # Check dimensions and PSD status of cov_params
+    cp = rslt.cov_params()
+    p = len(rslt.params)
+    assert_equal(cp.shape, np.r_[p, p])
+    np.linalg.cholesky(cp)
+
 
 def test_crossed_poisson_map():
 
@@ -189,6 +202,11 @@ def test_crossed_poisson_map():
         np.zeros_like(rslt.params),
         atol=1e-4)
 
+    # Check dimensions and PSD status of cov_params
+    cp = rslt.cov_params()
+    p = len(rslt.params)
+    assert_equal(cp.shape, np.r_[p, p])
+    np.linalg.cholesky(cp)
 
 def test_logit_map_crossed_formula():
 
@@ -209,10 +227,11 @@ def test_logit_map_crossed_formula():
     assert_allclose(
         r.iloc[0, :].values, np.r_[-0.02004904, 0.094014], atol=1e-4)
 
+    # Check dimensions and PSD status of cov_params
     cm = rslt.cov_params()
     p = rslt.params.shape[0]
     assert_equal(list(cm.shape), [p, p])
-
+    np.linalg.cholesky(cm)
 
 def test_elbo_grad():
 
@@ -301,6 +320,15 @@ def test_simple_logit_vb():
         rtol=1e-4,
         atol=1e-4)
 
+    for rslt in rslt1, rslt2:
+        cp = rslt.cov_params()
+        p = len(rslt.params)
+        if rslt is rslt1:
+            assert_equal(cp.shape, np.r_[p, p])
+            np.linalg.cholesky(cp)
+        else:
+            assert_equal(cp.shape, np.r_[p,])
+            assert_equal(cp > 0, True*np.ones(p))
 
 def test_simple_poisson_vb():
 
@@ -340,6 +368,16 @@ def test_simple_poisson_vb():
         rtol=1e-4,
         atol=1e-4)
 
+    for rslt in rslt1, rslt2:
+        cp = rslt.cov_params()
+        p = len(rslt.params)
+        if rslt is rslt1:
+            assert_equal(cp.shape, np.r_[p, p])
+            np.linalg.cholesky(cp)
+        else:
+            assert_equal(cp.shape, np.r_[p,])
+            assert_equal(cp > 0, True*np.ones(p))
+
 
 def test_crossed_logit_vb():
 
@@ -382,6 +420,16 @@ def test_crossed_logit_vb():
         rtol=1e-4,
         atol=1e-4)
 
+    for rslt in rslt1, rslt2:
+        cp = rslt.cov_params()
+        p = len(rslt.params)
+        if rslt is rslt1:
+            assert_equal(cp.shape, np.r_[p, p])
+            np.linalg.cholesky(cp)
+        else:
+            assert_equal(cp.shape, np.r_[p,])
+            assert_equal(cp > 0, True*np.ones(p))
+
 
 def test_crossed_logit_vb_formula():
 
@@ -401,6 +449,16 @@ def test_crossed_logit_vb_formula():
     rslt1.summary()
     rslt2.summary()
 
+    for rslt in rslt1, rslt2:
+        cp = rslt.cov_params()
+        p = len(rslt.params)
+        if rslt is rslt1:
+            assert_equal(cp.shape, np.r_[p,])
+            assert_equal(cp > 0, True*np.ones(p))
+        else:
+            assert_equal(cp.shape, np.r_[p,])
+            assert_equal(cp > 0, True*np.ones(p))
+
 
 def test_crossed_poisson_vb():
 
@@ -429,6 +487,16 @@ def test_crossed_poisson_vb():
         rtol=1e-4,
         atol=1e-4)
 
+    for rslt in rslt1, rslt2:
+        cp = rslt.cov_params()
+        p = len(rslt.params)
+        if rslt is rslt1:
+            assert_equal(cp.shape, np.r_[p, p])
+            np.linalg.cholesky(cp)
+        else:
+            assert_equal(cp.shape, np.r_[p,])
+            assert_equal(cp > 0, True*np.ones(p))
+
 
 def test_poisson_formula():
 
@@ -467,6 +535,15 @@ def test_poisson_formula():
 
         assert_allclose(rslt1.params, rslt2.params, rtol=1e-5)
 
+        for rslt in rslt1, rslt2:
+            cp = rslt.cov_params()
+            p = len(rslt.params)
+            if vb:
+                assert_equal(cp.shape, np.r_[p,])
+                assert_equal(cp > 0, True*np.ones(p))
+            else:
+                assert_equal(cp.shape, np.r_[p, p])
+                np.linalg.cholesky(cp)
 
 def test_scale_vb():