Repository: statsmodels/statsmodels

Issue:
Predict fails for data frame
For the following script:

```
import pandas as pd
import statsmodels.formula.api as smf

df = pd.DataFrame([[3, 0.030], [10, 0.060], [20, 0.120]], columns=['BSA', 'Absorbance'])

model = smf.ols('Absorbance ~ BSA', data=df)
fit = model.fit()
fit.predict([0.25])
```

I get an error ending

```
PatsyError: Error evaluating factor: NameError: name 'BSA' is not defined
    Absorbance ~ BSA
                 ^^^
```

Is this expected?

```
In [3]: statsmodels.__version__
Out[3]: '0.8.0.dev0+ad70c62'
```




Discussion:
Yes, that is expected.

The default in predict when using formulas is to go through patsy. patsy needs to be able to identify which variable the data corresponds to. So the exog needs to be a DataFrame or something dict like
e.g.

```
>>> fit.predict({'BSA': [0.25]})
0    0.012568
dtype: float64
```

otherwise, it is possible to provide the full predict design matrix and avoid patsy with `transform=False` (no categorical or similar transformations are done in this case)

```
>>> fit.predict([1, 0.25], transform=False)
array([ 0.01256849])
```

There were some problems in the last year after we changed to returning a pandas Series or DataFrame where we missed some possible cases of input types. (I'm using current master, but we still have one open PR to allow other possible argument types for exog in predict.)



However, it looks like patsy doesn't like Series

`>>> fit.predict(df.mean(0))`
raises an exception in patsy

```
>>> fit.predict(df.mean(0).to_frame().T)
0    0.07
dtype: float64

>>> fit.predict([1, 11.], transform=False)
array([ 0.07])
```

and this looks like a bug coming from the new indexing of the predicted return
(we predict correctly but have the wrong index, I guess)

```
>>> fit.predict(pd.Series([1, 11.]), transform=False)
0    0.07
1    0.07
dtype: float64
```
(Life is much simpler in a pure numpy world)
It is possible to give a more obvious error message?
I'm not sure or I doubt it's practical.

"NameError: name 'BSA' is not defined" sounds pretty descriptive to me.
The error is raised by patsy, so we don't have direct control over the exception message, short of catching the error and rewriting exception messages. 
Well I can say that the error message was confusing enough for me that I did a bug report...

I had the implicit hope that, in this case, the machinery could guess what I meant, but if not, it would be nice to know why (as in something like "we need to know what RHS values you passed in order to do the prediction".   Can you not check that the input is a data frame etc?
I have a similar issue, though I have used a dataframe - 

```
import pandas as pd 
import numpy as np
import matplotlib.pyplot as plt

calo=pd.read_csv("/Users/Sanjeev/Desktop/Excel R Assignments/Simple Linear Regression/calories_consumed.csv")

import statsmodels.formula.api as smf
model = smf.ols("wt~cal",data=calo).fit()

pred = model.predict(pd.DataFrame(calo['wt']))
```

The Error is - 

```
pred = model.predict(pd.DataFrame(calo['wt']))
Traceback (most recent call last):

  File "<ipython-input-53-4fcbf1ee1921>", line 1, in <module>
    pred = model.predict(pd.DataFrame(calo['wt']))

  File "/anaconda3/lib/python3.6/site-packages/statsmodels/base/model.py", line 837, in predict
    exog = dmatrix(design_info, exog, return_type="dataframe")

  File "/anaconda3/lib/python3.6/site-packages/patsy/highlevel.py", line 291, in dmatrix
    NA_action, return_type)

  File "/anaconda3/lib/python3.6/site-packages/patsy/highlevel.py", line 169, in _do_highlevel_design
    return_type=return_type)

  File "/anaconda3/lib/python3.6/site-packages/patsy/build.py", line 888, in build_design_matrices
    value, is_NA = _eval_factor(factor_info, data, NA_action)

  File "/anaconda3/lib/python3.6/site-packages/patsy/build.py", line 63, in _eval_factor
    result = factor.eval(factor_info.state, data)

  File "/anaconda3/lib/python3.6/site-packages/patsy/eval.py", line 566, in eval
    data)

  File "/anaconda3/lib/python3.6/site-packages/patsy/eval.py", line 551, in _eval
    inner_namespace=inner_namespace)

  File "/anaconda3/lib/python3.6/site-packages/patsy/compat.py", line 43, in call_and_wrap_exc
    exec("raise new_exc from e")

  File "<string>", line 1, in <module>

PatsyError: Error evaluating factor: NameError: name 'cal' is not defined
    wt~cal
       ^^^`
```
Would be really thankful if u can help.

@hellsanjeev  You need to have columns in the dataframe with the same names as those used in the formula, in this case there is no "cal" column in the DataFrame used for prediction.

patsy uses the column names to identify which data corresponds to the terms in the formula both for creating the model and for prediction.
@josef-pkt Actually I have column in the dataframe with the name "cal". Previously, I had used **calo.columns = ['wt', 'cal']** but later changed the column name directly in the CSV just to be sure. 

But both didn't work! 

@josef-pkt There was a problem with my Python or Anaconda or both. Reinstalling help resolve it. Nevertheless, thank you for your help on this. 👍 

Resolving patch:
diff --git a/statsmodels/base/model.py b/statsmodels/base/model.py
--- a/statsmodels/base/model.py
+++ b/statsmodels/base/model.py
@@ -993,7 +993,15 @@ def predict(self, exog=None, transform=True, *args, **kwargs):
                     exog = pd.DataFrame(exog).T
             orig_exog_len = len(exog)
             is_dict = isinstance(exog, dict)
-            exog = dmatrix(design_info, exog, return_type="dataframe")
+            try:
+                exog = dmatrix(design_info, exog, return_type="dataframe")
+            except Exception as exc:
+                msg = ('predict requires that you use a DataFrame when '
+                       'predicting from a model\nthat was created using the '
+                       'formula api.'
+                       '\n\nThe original error message returned by patsy is:\n'
+                       '{0}'.format(str(str(exc))))
+                raise exc.__class__(msg)
             if orig_exog_len > len(exog) and not is_dict:
                 import warnings
                 if exog_index is None:


Test patch:
diff --git a/statsmodels/formula/tests/test_formula.py b/statsmodels/formula/tests/test_formula.py
--- a/statsmodels/formula/tests/test_formula.py
+++ b/statsmodels/formula/tests/test_formula.py
@@ -1,4 +1,4 @@
-from statsmodels.compat.python import iteritems, StringIO
+from statsmodels.compat.python import iteritems, StringIO, PY3
 import warnings
 
 from statsmodels.formula.api import ols
@@ -10,6 +10,8 @@
 import numpy.testing as npt
 from statsmodels.tools.testing import assert_equal
 import numpy as np
+import pandas as pd
+import patsy
 import pytest
 
 
@@ -210,3 +212,14 @@ def test_patsy_missing_data():
         assert 'nan values have been dropped' in repr(w[-1].message)
     # Frist record will be dropped in both cases
     assert_equal(res.fittedvalues, res2)
+
+
+def test_predict_nondataframe():
+    df = pd.DataFrame([[3, 0.030], [10, 0.060], [20, 0.120]],
+                      columns=['BSA', 'Absorbance'])
+
+    model = ols('Absorbance ~ BSA', data=df)
+    fit = model.fit()
+    error = patsy.PatsyError if PY3 else TypeError
+    with pytest.raises(error):
+        fit.predict([0.25])