Repository: statsmodels/statsmodels

Issue:
BUG/MAINT  test error, pandas numpy compat
the newest python on travis and appveyor errors in MICE

fails since numpy 1.13 was released, and I guess conda updated to it

```
numpy:            1.13.0-py36_0     
pandas:           0.20.2-np113py36_0
```

there are 12 errors like the following with 2-d versus 1-d conflict, all in the same line
```
/imputation/mice.py", line 395, in _store_changes
    self.data[col].iloc[ix] = vals
```

https://s3.amazonaws.com/archive.travis-ci.org/jobs/241936042/log.txt?X-Amz-Expires=30&X-Amz-Date=20170612T112837Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJRYRXRSVGNKPKO5A/20170612/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=ad275790b6e71f77e6b3eb179ebd824d3277aed6dd386b36223a50a05452ca1e

```

======================================================================
ERROR: statsmodels.imputation.tests.test_mice.TestMICE.test_MICE
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/tests/test_mice.py", line 287, in test_MICE
    result = mi.fit(1, 3)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 1228, in fit
    self.data.update_all(n_burnin)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 415, in update_all
    self.update(vname)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 1005, in update
    self.impute(vname)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 988, in impute
    self.impute_pmm(vname)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 1080, in impute_pmm
    self._store_changes(vname, imputed_miss)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 395, in _store_changes
    self.data[col].iloc[ix] = vals
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/pandas/core/indexing.py", line 179, in __setitem__
    self._setitem_with_indexer(indexer, value)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/pandas/core/indexing.py", line 619, in _setitem_with_indexer
    value=value)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/pandas/core/internals.py", line 3203, in setitem
    return self.apply('setitem', **kwargs)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/pandas/core/internals.py", line 3091, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/pandas/core/internals.py", line 758, in setitem
    values[indexer] = value
ValueError: shape mismatch: value array of shape (5,1) could not be broadcast to indexing result of shape (5,)
```
BUG/MAINT  test error, pandas numpy compat
the newest python on travis and appveyor errors in MICE

fails since numpy 1.13 was released, and I guess conda updated to it

```
numpy:            1.13.0-py36_0     
pandas:           0.20.2-np113py36_0
```

there are 12 errors like the following with 2-d versus 1-d conflict, all in the same line
```
/imputation/mice.py", line 395, in _store_changes
    self.data[col].iloc[ix] = vals
```

https://s3.amazonaws.com/archive.travis-ci.org/jobs/241936042/log.txt?X-Amz-Expires=30&X-Amz-Date=20170612T112837Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJRYRXRSVGNKPKO5A/20170612/us-east-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=ad275790b6e71f77e6b3eb179ebd824d3277aed6dd386b36223a50a05452ca1e

```

======================================================================
ERROR: statsmodels.imputation.tests.test_mice.TestMICE.test_MICE
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/tests/test_mice.py", line 287, in test_MICE
    result = mi.fit(1, 3)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 1228, in fit
    self.data.update_all(n_burnin)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 415, in update_all
    self.update(vname)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 1005, in update
    self.impute(vname)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 988, in impute
    self.impute_pmm(vname)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 1080, in impute_pmm
    self._store_changes(vname, imputed_miss)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/statsmodels-0.8.0-py3.6-linux-x86_64.egg/statsmodels/imputation/mice.py", line 395, in _store_changes
    self.data[col].iloc[ix] = vals
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/pandas/core/indexing.py", line 179, in __setitem__
    self._setitem_with_indexer(indexer, value)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/pandas/core/indexing.py", line 619, in _setitem_with_indexer
    value=value)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/pandas/core/internals.py", line 3203, in setitem
    return self.apply('setitem', **kwargs)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/pandas/core/internals.py", line 3091, in apply
    applied = getattr(b, f)(**kwargs)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python3.6/site-packages/pandas/core/internals.py", line 758, in setitem
    values[indexer] = value
ValueError: shape mismatch: value array of shape (5,1) could not be broadcast to indexing result of shape (5,)
```


Discussion:
either numpy indexing has changed, 
or (less likely because change happened a while ago) it's a late effect that predict returns now a pandas Series if dataframe was used

quick fix would be to just np.squeeze before assignment, but that might fix the symptom and not the source of the problem.

starting in mice at line 1074
```
        # Unwind the indices
        jj = np.arange(dxi.shape[0])
        ix = dxi[[jj, ir]]
        iz = ixm[[jj, ix]]

        imputed_miss = np.array(endog_obs[iz])
        self._store_changes(vname, imputed_miss)
```

I have no idea whether endog_obs is a pandas Series or ndarray, and no idea what the indexing does and why imputed_miss is 2-d column-vector.
needs debugging with numpy 1.13  which I don't have yet.
I think it's time for the quick-fix, so we have a green Travis across PRs.


either numpy indexing has changed, 
or (less likely because change happened a while ago) it's a late effect that predict returns now a pandas Series if dataframe was used

quick fix would be to just np.squeeze before assignment, but that might fix the symptom and not the source of the problem.

starting in mice at line 1074
```
        # Unwind the indices
        jj = np.arange(dxi.shape[0])
        ix = dxi[[jj, ir]]
        iz = ixm[[jj, ix]]

        imputed_miss = np.array(endog_obs[iz])
        self._store_changes(vname, imputed_miss)
```

I have no idea whether endog_obs is a pandas Series or ndarray, and no idea what the indexing does and why imputed_miss is 2-d column-vector.
needs debugging with numpy 1.13  which I don't have yet.
I think it's time for the quick-fix, so we have a green Travis across PRs.



Resolving patch:
diff --git a/statsmodels/imputation/mice.py b/statsmodels/imputation/mice.py
--- a/statsmodels/imputation/mice.py
+++ b/statsmodels/imputation/mice.py
@@ -1076,7 +1076,7 @@ def impute_pmm(self, vname):
         ix = dxi[[jj, ir]]
         iz = ixm[[jj, ix]]
 
-        imputed_miss = np.array(endog_obs[iz])
+        imputed_miss = np.array(endog_obs[iz]).squeeze()
         self._store_changes(vname, imputed_miss)
 
 


Test patch: