Repository: statsmodels/statsmodels

Issue:
BUG: predict fails with nan in exog when building series
see last case in http://stackoverflow.com/questions/38045310/statsmodels-what-are-the-allowable-formats-to-give-to-result-predict-for-out/38087993#38087993

```
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

df3 = pd.DataFrame({"A": [10,20,30,324,2353,2353],
                    "B": [20, 30, 10, 100, 2332, np.nan],
                    "C": [0, -30, 120, 11, 2, 2 ]})
result3 = smf.ols(formula="A ~ B + C", data=df3).fit()
print(result3.summary())
test3 = result3.predict(df3)     # Fails
```

```
Traceback (most recent call last):
  File "E:\josef_new_notebook\scripts\bug_missing_formula.py", line 10, in <modu
le>
    test3 = result3.predict(df3)     # Fails
  File "E:\josef_new_notebook\git\statsmodels_py34_pr\statsmodels\base\model.py"
, line 767, in predict
    return pd.Series(predict_results, index=exog_index)
  File "C:\Users\josef\Downloads\WinPython-64bit-3.4.4.2\python-3.4.4.amd64\lib\
site-packages\pandas\core\series.py", line 228, in __init__
    data = SingleBlockManager(data, index, fastpath=True)
  File "C:\Users\josef\Downloads\WinPython-64bit-3.4.4.2\python-3.4.4.amd64\lib\
site-packages\pandas\core\internals.py", line 3752, in __init__
    fastpath=True)
  File "C:\Users\josef\Downloads\WinPython-64bit-3.4.4.2\python-3.4.4.amd64\lib\
site-packages\pandas\core\internals.py", line 2461, in make_block
    return klass(values, ndim=ndim, fastpath=fastpath, placement=placement)
  File "C:\Users\josef\Downloads\WinPython-64bit-3.4.4.2\python-3.4.4.amd64\lib\
site-packages\pandas\core\internals.py", line 84, in __init__
    len(self.mgr_locs)))
ValueError: Wrong number of items passed 5, placement implies 6
```

reference: returning pandas object has been added in #2884 (new in 0.8)



Discussion:
Yes, it's patsy's overzealos drop nan/missing policy

```
>>> dmatrix(result3.model.data.design_info.builder, df3, return_type='dataframe'
   Intercept       B      C
0        1.0    20.0    0.0
1        1.0    30.0  -30.0
2        1.0    10.0  120.0
3        1.0   100.0   11.0
4        1.0  2332.0    2.0


>>> df3
      A       B    C
0    10    20.0    0
1    20    30.0  -30
2    30    10.0  120
3   324   100.0   11
4  2353  2332.0    2
5  2353     NaN    2
>>>
```

dropping nan before adding the index was pretty nasty because there was no information which predictions correspond to which row, if number of rows was reduced.
So, assume nobody cared or didn't have a case with nan.
However, dropping nan in cross-validation or similar if only summary statistics are calculated is useful.

I think dropping nans with formulas in statsmodels 0.8.0, was a mistake and we add them back in, even if it breaks backwards compatibility for that case.

I guess, to get minimal overhead we check if formula was used, if yes, then use dmatrix with pandas return, otherwise ignore nans and work as it is now. 
I guess we need a reindex or something like that. if len(exog) before and after patsy are not the same.


Resolving patch:
diff --git a/statsmodels/base/model.py b/statsmodels/base/model.py
--- a/statsmodels/base/model.py
+++ b/statsmodels/base/model.py
@@ -764,7 +764,14 @@ def predict(self, exog=None, transform=True, *args, **kwargs):
         if transform and hasattr(self.model, 'formula') and exog is not None:
             from patsy import dmatrix
             exog = dmatrix(self.model.data.design_info.builder,
-                           exog)
+                           exog, return_type="dataframe")
+            if len(exog) < len(exog_index):
+                # missing values, rows have been dropped
+                if exog_index is not None:
+                    exog = exog.reindex(exog_index)
+                else:
+                    import warnings
+                    warnings.warn("nan rows have been dropped", ValueWarning)
 
         if exog is not None:
             exog = np.asarray(exog)


Test patch:
diff --git a/statsmodels/base/tests/test_generic_methods.py b/statsmodels/base/tests/test_generic_methods.py
--- a/statsmodels/base/tests/test_generic_methods.py
+++ b/statsmodels/base/tests/test_generic_methods.py
@@ -440,6 +440,23 @@ def initialize(cls):
         cls.res = mod.fit()  # default use_t=True
 
 
+    def test_predict_missing(self):
+        ex = self.data[:5].copy()
+        ex.iloc[0, 1] = np.nan
+        predicted1 = self.res.predict(ex)
+        predicted2 = self.res.predict(ex[1:])
+        from pandas.util.testing import assert_series_equal
+        try:
+            from pandas.util.testing import assert_index_equal
+        except ImportError:
+            # for old pandas
+            from numpy.testing import assert_array_equal as assert_index_equal
+
+        assert_index_equal(predicted1.index, ex.index)
+        assert_series_equal(predicted1[1:], predicted2)
+        assert_equal(predicted1.values[0], np.nan)
+
+
 class TestWaldAnovaGLM(CheckAnovaMixin):
 
     @classmethod