Repository: statsmodels/statsmodels

Issue:
Odd way to get confidence and prediction intervals for new OLS prediction 
I've seen this question & answers [here](https://stackoverflow.com/questions/17559408/confidence-and-prediction-intervals-with-statsmodels), but nothing is helping... 

I found a way to get the confidence and prediction intervals around a prediction on a new data point, but it's very messy. Is there an easier way? Note, I am not trying to plot the confidence or prediction curves as in the stack answer linked above. I just want them for a single new prediction.

## generate dummy data and fit model:
```
import statsmodels.api as sm
import numpy as np

b0 = 0  # true intercept
b1 = 1  # true slope
e = np.random.randn(100)  # noise                                                                                                             

x = np.linspace(-3, 3, 100)
y = b0 + b1 * x + e                                                                                                                    

X = sm.add_constant(x)
model = sm.OLS(y, X)
results = model.fit()
```

## Make prediction of new observed X value "0":
```
prediction = results.get_prediction([1, 0])
prediction.summary_frame() # will throw error. but this line is needed for "table" to emerge                                                                           
prediction.table # final desired results are here. 
```

Odd that "table" is only available after `prediction.summary_frame()` is run? Unlike in the stack overflow [answer](https://stackoverflow.com/a/47191929/5392289), `prediction.summary_frame()` throws the error:

> TypeError: 'builtin_function_or_method' object is not iterable

Versions I'm running:
Darwin-16.7.0-x86_64-i386-64bit
('Python', '2.7.14 |Anaconda, Inc.| (default, Oct  5 2017, 02:28:52) \n[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]')
('NumPy', '1.13.3')
('SciPy', '1.0.0')
('statsmodels', '0.8.0')


Discussion:
Thanks for reporting

There must be a bug in the dataframe creation.  I will look it later today.

based on the example it requires a DataFrame as exog to get the index for the summary_frame

```
>>> import pandas as pd
>>> prediction = results.get_prediction(pd.DataFrame([[1, 0], [1, 0]]))
>>> prediction.summary_frame()
      mean   mean_se  mean_ci_lower  mean_ci_upper  obs_ci_lower  obs_ci_upper
0 -0.02566  0.094838      -0.213862       0.162543     -1.917071      1.865751
1 -0.02566  0.094838      -0.213862       0.162543     -1.917071      1.865751

>>> prediction = results.get_prediction(pd.DataFrame([[1, 0]]))
>>> prediction.summary_frame()
      mean   mean_se  mean_ci_lower  mean_ci_upper  obs_ci_lower  obs_ci_upper
0 -0.02566  0.094838      -0.213862       0.162543     -1.917071      1.865751
```

The bug is that there is no fallback for missing row_labels.  (I haven't checked yet why pandas doesn't use it's default index, when creating the summary frame.)

It works if row_labels are explicitly provided

```
>>> prediction = results.get_prediction([1, 0], row_labels=[0])
>>> prediction.summary_frame()
      mean   mean_se  mean_ci_lower  mean_ci_upper  obs_ci_lower  obs_ci_upper
0 -0.02566  0.094838      -0.213862       0.162543     -1.917071      1.865751
```

most likely the same problem is also in GLM get_prediction

Ok, the bug it `list.index` is not None. I just ran into this with another function or method.

```
            if hasattr(exog, 'index'):
                row_labels = exog.index
```

numpy arrays also works, and default row_labels creation works

```
>>> prediction = results.get_prediction([[1, 0]])
>>> prediction.row_labels
<built-in method index of list object at 0x000000000998D208>
>>> prediction.row_labels()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: index() takes at least 1 argument (0 given)

>>> prediction = results.get_prediction(np.array([1, 0]))
>>> prediction.row_labels
>>> 
>>> prediction.summary_frame()
      mean   mean_se  mean_ci_lower  mean_ci_upper  obs_ci_lower  obs_ci_upper
0 -0.02566  0.094838      -0.213862       0.162543     -1.917071      1.865751
```

i.e. using a list as exog is currently not supported, or anything that has an `index` attribute that is not a dataframe_like index.
fix is relatively easy using a `callable` check   
(There still might be other index ducks that don't quack in the right way, but I wanted to avoid isinstance checks for exog and index.)

test coverage for exog in `get_prediction` is almost non-existent. "statsmodels\regression\tests\test_predict.py" checks the computations only for the model.exog.
Also, we need to compare with `predict` coverage, where we had problems when switching to returning pandas Series instead of ndarray.

I will open a PR later today. I have the callable fix, but no unit tests yet.
this is an occasion to check again and also merge #3611
another issue that needs checking is the docstring and signature
`statsmodels.regression._prediction.get_prediction` doesn't list row_labels in the docstring.
`RegressionResults.get_prediction` uses/references that docstring

Do we need the `**kwargs` in RegressionResults._get_prediction?
Maybe not right now but subclasses might use it.

same list/callable and docstring problems in `statsmodels.genmod._prediction.get_prediction_glm`

Resolving patch:
diff --git a/statsmodels/genmod/_prediction.py b/statsmodels/genmod/_prediction.py
--- a/statsmodels/genmod/_prediction.py
+++ b/statsmodels/genmod/_prediction.py
@@ -186,9 +186,8 @@ def get_prediction_glm(self, exog=None, transform=True, weights=None,
 
     if exog is not None:
         if row_labels is None:
-            if hasattr(exog, 'index'):
-                row_labels = exog.index
-            else:
+            row_labels = getattr(exog, 'index', None)
+            if callable(row_labels):
                 row_labels = None
 
         exog = np.asarray(exog)
diff --git a/statsmodels/regression/_prediction.py b/statsmodels/regression/_prediction.py
--- a/statsmodels/regression/_prediction.py
+++ b/statsmodels/regression/_prediction.py
@@ -130,9 +130,8 @@ def get_prediction(self, exog=None, transform=True, weights=None,
 
     if exog is not None:
         if row_labels is None:
-            if hasattr(exog, 'index'):
-                row_labels = exog.index
-            else:
+            row_labels = getattr(exog, 'index', None)
+            if callable(row_labels):
                 row_labels = None
 
         exog = np.asarray(exog)


Test patch:
diff --git a/statsmodels/regression/tests/test_predict.py b/statsmodels/regression/tests/test_predict.py
--- a/statsmodels/regression/tests/test_predict.py
+++ b/statsmodels/regression/tests/test_predict.py
@@ -92,7 +92,6 @@ def test_predict_se():
     np.testing.assert_allclose(iv_l, res3.fittedvalues[:3] - ci_half[:3],
                                rtol=1e-12)
 
-
     #use wrong size for exog
     #prstd, iv_l, iv_u = wls_prediction_std(res3, x2[-1,0], weights=3.)
     np.testing.assert_raises(ValueError, wls_prediction_std, res3, x2[-1,0],
@@ -158,6 +157,27 @@ def test_ci(self):
         sf2 = pred_res2.summary_frame()
         assert_equal(sf2.columns.tolist(), col_names)
 
+        # check that list works, issue 4437
+        x = res_wls.model.exog.mean(0)
+        pred_res3 = res_wls.get_prediction(x)
+        ci3 = pred_res3.conf_int(obs=True)
+        pred_res3b = res_wls.get_prediction(x.tolist())
+        ci3b = pred_res3b.conf_int(obs=True)
+        assert_allclose(pred_res3b.se_obs, pred_res3.se_obs, rtol=1e-13)
+        assert_allclose(ci3b, ci3, rtol=1e-13)
+        res_df = pred_res3b.summary_frame()
+        assert_equal(res_df.index.values, [0])
+
+        x = res_wls.model.exog[-2:]
+        pred_res3 = res_wls.get_prediction(x)
+        ci3 = pred_res3.conf_int(obs=True)
+        pred_res3b = res_wls.get_prediction(x.tolist())
+        ci3b = pred_res3b.conf_int(obs=True)
+        assert_allclose(pred_res3b.se_obs, pred_res3.se_obs, rtol=1e-13)
+        assert_allclose(ci3b, ci3, rtol=1e-13)
+        res_df = pred_res3b.summary_frame()
+        assert_equal(res_df.index.values, [0, 1])
+
 
     def test_glm(self):
         # prelimnimary, getting started with basic test for GLM.get_prediction
@@ -217,3 +237,24 @@ def test_glm(self):
         # prediction with exog and no weights does not error
         res_glm = mod_glm.fit()
         pred_glm = res_glm.get_prediction(X)
+
+        # check that list works, issue 4437
+        x = res_glm.model.exog.mean(0)
+        pred_res3 = res_glm.get_prediction(x)
+        ci3 = pred_res3.conf_int()
+        pred_res3b = res_glm.get_prediction(x.tolist())
+        ci3b = pred_res3b.conf_int()
+        assert_allclose(pred_res3b.se_mean, pred_res3.se_mean, rtol=1e-13)
+        assert_allclose(ci3b, ci3, rtol=1e-13)
+        res_df = pred_res3b.summary_frame()
+        assert_equal(res_df.index.values, [0])
+
+        x = res_glm.model.exog[-2:]
+        pred_res3 = res_glm.get_prediction(x)
+        ci3 = pred_res3.conf_int()
+        pred_res3b = res_glm.get_prediction(x.tolist())
+        ci3b = pred_res3b.conf_int()
+        assert_allclose(pred_res3b.se_mean, pred_res3.se_mean, rtol=1e-13)
+        assert_allclose(ci3b, ci3, rtol=1e-13)
+        res_df = pred_res3b.summary_frame()
+        assert_equal(res_df.index.values, [0, 1])