Repository: statsmodels/statsmodels

Issue:
predict accepts a DataFrame but returns a numpy array (rather than Series)
This loses any `Index` information on the input `DataFrame`.

This would not be an issue except for the fact that the shape of the input DataFrame does not match the returned numpy array which makes combining the two tricky.

This is because some of the data is dropped when performing the transform:

```
exog = dmatrix(self.model.data.orig_exog.design_info.builder, exog)
```

This actually seems to be a rather simple fix.

`dmatrix` can be told to return a "dataframe" and then that DataFrame can be dot'ed with the params.



Discussion:
Do you have an example where the length of the returned prediction is different from the number of rows in the DataFrame?
I thought this should always be the same. 
Is it the case if there are missing values in your predict exog? Or, is there a transform that changes the number of observations?

The default argument for `dmatrix` is `NA_action='drop'`.

I have a transform that can produce NaNs.

Example:

```
s = pd.DataFrame({'s':pd.Series(range(10))})

def shifter(x):
    return x.shift()

model = "s ~ shifter(s)"
fit = ols(model, data=s).fit()
```

```
In [18]: s.shape
Out[18]: (10,)

In [17]: fit.predict(s).shape
Out[17]: (9,)
```

Here is a more fleshed out example of how this can be improved:

```
import pandas as pd
from statsmodels.formula.api import ols
from patsy import dmatrix

s = pd.Series(range(10),index=pd.date_range("2013-1-1", periods=10)).to_frame()

def shifter(x):
    return x.shift()

model = "s ~ shifter(s)"

fit = ols(model, data=s).fit()
```

Current prediction:

```
In [19]:
fit.predict(s)

Out[19]:
array([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])
```

New prediction:

```
In [21]:
def better_predict(fit, exog):
    dm = dmatrix(fit.model.data.orig_exog.design_info.builder, exog, return_type="dataframe", )
    return dm.dot(fit.params)

In [22]:
better_predict(fit, s)

Out[22]:
2013-01-02    1
2013-01-03    2
2013-01-04    3
2013-01-05    4
2013-01-06    5
2013-01-07    6
2013-01-08    7
2013-01-09    8
2013-01-10    9
dtype: float64
```

Hmm, yes, I quite like this approach. It won't work for predict and forecast in TSA, but that should be overwritten already anyway.

I'll have to look at the speed penalty vs. pure arrays for bootstrapping, etc. but I doubt it's that substantial.

We need to avoid any non-numpy overhead for bootstrapping and cross-validation.

While I agree, my new rule is we need timings before discussing any timings. It's a concern, but I don't know how much yet. We're going to be able to remove some checks that are in there already that patsy would also do (more robustly), so the overhead might really be negligible. What's negligible for bootstrap? We decide when we have timings I guess.

A few ways forward that come to mind.

1) This magic goes in the Results.predict. Bootstrap relies on Model.predict, which Results.predict also calls. Everyone wins.
2) We provide the predict somehow in the wrapper classes and bootstrap relies on Results._results.predict.

I'm leaning towards 1. Seems simple and straight-forward.

I think we just need a simple type check; `if types(exog) is np.array` or something like that.
also what happens if exog is None and we calculate the fittedvalues, which are separately wrapped AFAIR, and should be arrays in _results.

Timing: a simple LOO crossval would do it.

I [encountered this recently](http://stackoverflow.com/q/22580477/656912) and ended up very confused about what was happening. I'm glad to see it's not just me. Including pandas index information in the results from `predict` (at least optionally), or preserving length (to match the length of `predict`s argument) would make predict much easier to use.

I opened a SUMMary issue #2150 and started to look into prediction methods. 
I'd like a general solution instead of patching code in various places.

see also #2145 as a more general duplicate

Putting this on my radar for 0.7. That's annoying.

I started part of pandas handling in my branch for prediction intervals.
I want to remove all data/exog handling from predict and outsource it, which is another issue

We need design decision and helper functions, Kerby has also PRs for prediction.

Ok, we have general wrapping code that I wrote for MixedLM, so it should be simple to just fix this issue without getting too far afield.

+1

+1


Resolving patch:
diff --git a/statsmodels/base/model.py b/statsmodels/base/model.py
--- a/statsmodels/base/model.py
+++ b/statsmodels/base/model.py
@@ -3,6 +3,7 @@
 import numpy as np
 from scipy import stats
 from statsmodels.base.data import handle_data
+from statsmodels.tools.data import _is_using_pandas
 from statsmodels.tools.tools import recipr, nan_dot
 from statsmodels.stats.contrast import ContrastResults, WaldTestResults
 from statsmodels.tools.decorators import resettable_cache, cache_readonly
@@ -735,10 +736,14 @@ def predict(self, exog=None, transform=True, *args, **kwargs):
 
         Returns
         -------
-        prediction : ndarray or pandas.Series
+        prediction : ndarray, pandas.Series or pandas.DataFrame
             See self.model.predict
 
         """
+        import pandas as pd
+
+        exog_index = exog.index if _is_using_pandas(exog, None) else None
+
         if transform and hasattr(self.model, 'formula') and exog is not None:
             from patsy import dmatrix
             exog = dmatrix(self.model.data.design_info.builder,
@@ -751,7 +756,18 @@ def predict(self, exog=None, transform=True, *args, **kwargs):
                 exog = exog[:, None]
             exog = np.atleast_2d(exog)  # needed in count model shape[1]
 
-        return self.model.predict(self.params, exog, *args, **kwargs)
+        predict_results = self.model.predict(self.params, exog, *args, **kwargs)
+
+        if exog_index is not None and not hasattr(predict_results, 'predicted_values'):
+
+            if predict_results.ndim == 1:
+                return pd.Series(predict_results, index=exog_index)
+            else:
+                return pd.DataFrame(predict_results, index=exog_index)
+
+        else:
+
+            return predict_results
 
 
 #TODO: public method?
diff --git a/statsmodels/regression/linear_model.py b/statsmodels/regression/linear_model.py
--- a/statsmodels/regression/linear_model.py
+++ b/statsmodels/regression/linear_model.py
@@ -32,6 +32,7 @@
 """
 
 from __future__ import print_function
+
 from statsmodels.compat.python import lrange, lzip, range
 __docformat__ = 'restructuredtext en'
 
@@ -370,10 +371,11 @@ def predict(self, params, exog=None):
         """
         #JP: this doesn't look correct for GLMAR
         #SS: it needs its own predict method
+
         if exog is None:
             exog = self.exog
-        return np.dot(exog, params)
 
+        return np.dot(exog, params)
 
     def get_distribution(self, params, scale, exog=None, dist_class=None):
         """
diff --git a/statsmodels/sandbox/mice/mice.py b/statsmodels/sandbox/mice/mice.py
--- a/statsmodels/sandbox/mice/mice.py
+++ b/statsmodels/sandbox/mice/mice.py
@@ -775,6 +775,7 @@ def plot_fit_obs(self, col_name, lowess_args=None,
 
         from statsmodels.graphics import utils as gutils
         from statsmodels.nonparametric.smoothers_lowess import lowess
+        import pandas as pd
 
         if lowess_args is None:
             lowess_args = {}
@@ -1006,10 +1007,12 @@ def _get_predicted(self, obj):
 
         if isinstance(obj, np.ndarray):
             return obj
+        elif isinstance(obj, pd.Series):
+            return obj.values
         elif hasattr(obj, 'predicted_values'):
             return obj.predicted_values
         else:
-            raise "cannot obtain predicted values from %s" % obj.__class__
+            raise ValueError("cannot obtain predicted values from %s" % obj.__class__)
 
 
     def impute_pmm(self, vname):


Test patch:
diff --git a/statsmodels/base/tests/test_generic_methods.py b/statsmodels/base/tests/test_generic_methods.py
--- a/statsmodels/base/tests/test_generic_methods.py
+++ b/statsmodels/base/tests/test_generic_methods.py
@@ -120,6 +120,7 @@ def test_fitted(self):
         assert_allclose(fitted, res.predict(), rtol=1e-12)
 
     def test_predict_types(self):
+
         res = self.results
         # squeeze to make 1d for single regressor test case
         p_exog = np.squeeze(np.asarray(res.model.exog[:2]))
@@ -141,6 +142,10 @@ def test_predict_types(self):
             res.predict(p_exog.tolist())
             res.predict(p_exog[0].tolist())
         else:
+
+            import pandas as pd
+            from pandas.util.testing import assert_series_equal
+
             fitted = res.fittedvalues[:2]
             assert_allclose(fitted, res.predict(p_exog), rtol=1e-12)
             # this needs reshape to column-vector:
@@ -152,11 +157,27 @@ def test_predict_types(self):
             assert_allclose(fitted[:1], res.predict(p_exog[0]),
                             rtol=1e-12)
 
-            # predict doesn't preserve DataFrame, e.g. dot converts to ndarray
-#             import pandas
-#             predicted = res.predict(pandas.DataFrame(p_exog))
-#             assert_(isinstance(predicted, pandas.DataFrame))
-#             assert_allclose(predicted, fitted, rtol=1e-12)
+            exog_index = range(len(p_exog))
+            predicted = res.predict(p_exog)
+
+            if p_exog.ndim == 1:
+                predicted_pandas = res.predict(pd.Series(p_exog, index=exog_index))
+
+            else:
+                predicted_pandas = res.predict(pd.DataFrame(p_exog, index=exog_index))
+
+            if predicted.ndim == 1:
+
+                assert_(isinstance(predicted_pandas, pd.Series))
+
+                predicted_expected = pd.Series(predicted, index=exog_index)
+                assert_series_equal(predicted_expected, predicted_pandas)
+
+            else:
+                assert_(isinstance(predicted_pandas, pd.DataFrame))
+
+                predicted_expected = pd.DataFrame(predicted, index=exog_index)
+                assert_(predicted_expected.equals(predicted_pandas))
 
 
 #########  subclasses for individual models, unchanged from test_shrink_pickle
diff --git a/statsmodels/base/tests/test_shrink_pickle.py b/statsmodels/base/tests/test_shrink_pickle.py
--- a/statsmodels/base/tests/test_shrink_pickle.py
+++ b/statsmodels/base/tests/test_shrink_pickle.py
@@ -51,6 +51,10 @@ def setup_class(self):
         self.l_max = 20000
 
     def test_remove_data_pickle(self):
+
+        import pandas as pd
+        from pandas.util.testing import assert_series_equal
+
         if winoldnp:
             raise SkipTest
         results = self.results
@@ -72,7 +76,15 @@ def test_remove_data_pickle(self):
         #remove data arrays, check predict still works
         results.remove_data()
         pred2 = results.predict(xf, **pred_kwds)
-        np.testing.assert_equal(pred2, pred1)
+
+        if isinstance(pred1, pd.Series) and isinstance(pred2, pd.Series):
+            assert_series_equal(pred1, pred2)
+
+        elif isinstance(pred1, pd.DataFrame) and isinstance(pred2, pd.DataFrame):
+            assert_(pred1.equals(pred2))
+
+        else:
+            np.testing.assert_equal(pred2, pred1)
 
         #pickle, unpickle reduced array
         res, l = check_pickle(results._results)
@@ -86,7 +98,15 @@ def test_remove_data_pickle(self):
 
 
         pred3 = results.predict(xf, **pred_kwds)
-        np.testing.assert_equal(pred3, pred1)
+
+        if isinstance(pred1, pd.Series) and isinstance(pred3, pd.Series):
+            assert_series_equal(pred1, pred3)
+
+        elif isinstance(pred1, pd.DataFrame) and isinstance(pred3, pd.DataFrame):
+            assert_(pred1.equals(pred3))
+
+        else:
+            np.testing.assert_equal(pred3, pred1)
 
     def test_remove_data_docstring(self):
         assert_(self.results.remove_data.__doc__ is not None)
diff --git a/statsmodels/regression/tests/test_regression.py b/statsmodels/regression/tests/test_regression.py
--- a/statsmodels/regression/tests/test_regression.py
+++ b/statsmodels/regression/tests/test_regression.py
@@ -1089,7 +1089,6 @@ def test_fvalue_implicit_constant():
     res.summary()
 
 
-
 if __name__=="__main__":
 
     import nose