Repository: statsmodels/statsmodels

Issue:
BUG: MixedLM random effects BLUPs with variance components
This fixes a bug that prevented prediction covariances of BLUP's from being extracted from models having variance components.  

I also changed the name index for the variance component BLUP's from being the position to being the actual name (otherwise it is sometimes impossible to know which BLUP corresponds to which actual group in the data).  This could produce backwards incompatibilities so I will add a note to the release notes.  People who were relying on the previous names lining up with the actual group labels may not have been getting what they though they were getting, so I think this is an appropriate change to make despite the compatibility break. 


Discussion:
This pull request **introduces 1 alert** when merging cfa1bb24639c1e72ba68da6d7b546c98166c7c60 into 5a01211ab0c73b8b9182396e6d551efe220c2ee9 - [view on lgtm.com](https://lgtm.com/projects/g/statsmodels/statsmodels/rev/pr-9a75e1e34347e701fc2561915bbb0683f71c1d3c)

**new alerts:**

* 1 for Potentially uninitialized local variable

---

*Comment posted by [lgtm.com](https://lgtm.com)*

[![Coverage Status](https://coveralls.io/builds/16280557/badge)](https://coveralls.io/builds/16280557)

Coverage increased (+0.01%) to 82.695% when pulling **fccd687aca0a709e191bc6f044e5d43fc6106ba4 on kshedden:mixxedlm_vc_names** into **5a01211ab0c73b8b9182396e6d551efe220c2ee9 on statsmodels:master**.

# [Codecov](https://codecov.io/gh/statsmodels/statsmodels/pull/4413?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@9f2ed83`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/statsmodels/statsmodels/pull/4413/graphs/tree.svg?width=650&token=Og706qJV5S&height=150&src=pr)](https://codecov.io/gh/statsmodels/statsmodels/pull/4413?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master    #4413   +/-   ##
=========================================
  Coverage          ?   80.18%           
=========================================
  Files             ?      560           
  Lines             ?    84727           
  Branches          ?     9580           
=========================================
  Hits              ?    67937           
  Misses            ?    14578           
  Partials          ?     2212
```


| [Impacted Files](https://codecov.io/gh/statsmodels/statsmodels/pull/4413?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [statsmodels/regression/tests/test\_lme.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4413/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvcmVncmVzc2lvbi90ZXN0cy90ZXN0X2xtZS5weQ==) | `87.81% <100%> (ø)` | |
| [statsmodels/regression/mixed\_linear\_model.py](https://codecov.io/gh/statsmodels/statsmodels/pull/4413/diff?src=pr&el=tree#diff-c3RhdHNtb2RlbHMvcmVncmVzc2lvbi9taXhlZF9saW5lYXJfbW9kZWwucHk=) | `81.4% <100%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/statsmodels/statsmodels/pull/4413?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/statsmodels/statsmodels/pull/4413?src=pr&el=footer). Last update [9f2ed83...e0e0e42](https://codecov.io/gh/statsmodels/statsmodels/pull/4413?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

This pull request **introduces 1 alert** when merging fccd687aca0a709e191bc6f044e5d43fc6106ba4 into 5a01211ab0c73b8b9182396e6d551efe220c2ee9 - [view on lgtm.com](https://lgtm.com/projects/g/statsmodels/statsmodels/rev/pr-cb8d5383804a9c33fc5613c48df010bcbf3bdcb9)

**new alerts:**

* 1 for Potentially uninitialized local variable

---

*Comment posted by [lgtm.com](https://lgtm.com)*
@kshedden  There is another possible candidate for 0.9. But I have no idea about this.
Yes, I would like this to go into 0.9 as well.  But give me 1-2 days to
review it again then I will merge.



On Wed, Apr 18, 2018 at 10:07 PM, Josef Perktold <notifications@github.com>
wrote:

> @kshedden <https://github.com/kshedden> There is another possible
> candidate for 0.9. But I have no idea about this.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/pull/4413#issuecomment-382586218>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACiwwzjPDWBXP-7Ehi92HQVMGrlGnptYks5tp_FygaJpZM4TCgrX>
> .
>

My current (!) plans are to get the rc1 out in the middle of next week. I have one more day of code merges, and then there are still quite a few doc and release issues.
OK I will try to do this merge tomorrow.

On Wed, Apr 18, 2018 at 10:28 PM, Josef Perktold <notifications@github.com>
wrote:

> My current (!) plans are to get the rc1 out in the middle of next week. I
> have one more day of code merges, and then there are still quite a few doc
> and release issues.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/pull/4413#issuecomment-382589624>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACiww1MhAYTux_b5Bz86c4YkiHsqpxkPks5tp_Y0gaJpZM4TCgrX>
> .
>

Hi, I'm getting the same error calling 'random_effects_cov' with 0.9.0 as with '0.8.0' for variance components models. I note though that the expected change to labelling of groups in 'random_effects' has taken place. Could I possible have a dependency issue?

Resolving patch:
diff --git a/statsmodels/regression/mixed_linear_model.py b/statsmodels/regression/mixed_linear_model.py
--- a/statsmodels/regression/mixed_linear_model.py
+++ b/statsmodels/regression/mixed_linear_model.py
@@ -2308,7 +2308,7 @@ def random_effects_cov(self):
             ex_r = self.model._aex_r[group_ix]
             ex2_r = self.model._aex_r2[group_ix]
             label = self.model.group_labels[group_ix]
-            vc_var = self.model._expand_vcomp(vcomp, group_ix)
+            vc_var = self.model._expand_vcomp(vcomp, label)
 
             solver = _smw_solver(self.scale, ex_r, ex2_r, cov_re_inv,
                                  1 / vc_var)
@@ -2325,7 +2325,7 @@ def random_effects_cov(self):
             v[0:m, 0:m] += self.cov_re
             ix = np.arange(m, v.shape[0])
             v[ix, ix] += vc_var
-            na = self._expand_re_names(group_ix)
+            na = self._expand_re_names(label)
             v = pd.DataFrame(v, index=na, columns=na)
             ranef_dict[label] = v
 


Test patch:
diff --git a/statsmodels/regression/tests/test_lme.py b/statsmodels/regression/tests/test_lme.py
--- a/statsmodels/regression/tests/test_lme.py
+++ b/statsmodels/regression/tests/test_lme.py
@@ -1106,7 +1106,7 @@ def test_random_effects_getters():
         y.append(yy)
         x.append(xx)
         z.append(zz)
-        g.append(i * np.ones(m))
+        g.append(["g%d" % i] * m)
 
     y = np.concatenate(y)
     x = np.concatenate(x)
@@ -1142,12 +1142,12 @@ def test_random_effects_getters():
     result = model.fit()
 
     ref = result.random_effects
-    b0 = [ref[k][0:2] for k in range(ng)]
+    b0 = [ref["g%d" % k][0:2] for k in range(ng)]
     b0 = np.asarray(b0)
     assert (np.corrcoef(b0[:, 0], b[:, 0])[0, 1] > 0.8)
     assert (np.corrcoef(b0[:, 1], b[:, 1])[0, 1] > 0.8)
 
-    cf0 = [ref[k][2:6] for k in range(ng)]
+    cf0 = [ref["g%d" % k][2:6] for k in range(ng)]
     cf0 = np.asarray(cf0)
     for k in range(4):
         assert (np.corrcoef(cf0[:, k], cc[:, k])[0, 1] > 0.8)