Repository: statsmodels/statsmodels
Pull request: BUG: fixes #6553, sliced predicted values according to predicted index
- [x] closes #6553
- [x] tests added / passed. 
- [x] code/documentation is well formatted.  
- [x] properly formatted commit message. See 
      [NumPy's guide](https://docs.scipy.org/doc/numpy-1.15.1/dev/gitwash/development_workflow.html#writing-the-commit-message). 

Could you point me to the tests that I should change or where I should add a new test? 

Problem statement:
AutoReg predict start point bug
#### Describe the bug
AugoReg predict does not work in some cases. Seems to be related to start point. Works if I move start point to end of endog sample (i.e. in the example below, `start=100`) but not if the start is further ahead. 

#### Code Sample, a copy-pastable example if possible
```python
from statsmodels.tsa.ar_model import AutoReg
import pandas as pd
import numpy as np

y_train = pd.Series(np.random.normal(size=100))
m = AutoReg(y_train, lags=10, trend="c")
mf = m.fit()
mf.predict(start=101, end=110)
```

## Error traceback
> Traceback (most recent call last):
>   File "/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
>     exec(code_obj, self.user_global_ns, self.user_ns)
>   File "<ipython-input-2-98ce42ecfe13>", line 1, in <module>
>     runfile('/Users/mloning/Documents/Research/toolboxes/sktime/sktime/notes/playground.py', wdir='/Users/mloning/Documents/Research/toolboxes/sktime/sktime/notes')
>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
>     pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
>   File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
>     exec(compile(contents+"\n", file, 'exec'), glob, loc)
>   File "/Users/mloning/Documents/Research/toolboxes/sktime/sktime/notes/playground.py", line 8, in <module>
>     mf.predict(start=101, end=110)
>   File "/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/statsmodels/base/wrapper.py", line 115, in wrapper
>     obj = data.wrap_output(func(results, *args, **kwargs), how)
>   File "/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/statsmodels/tsa/ar_model.py", line 1981, in predict
>     exog_oos=exog_oos)
>   File "/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/statsmodels/tsa/ar_model.py", line 662, in predict
>     exog, exog_oos)
>   File "/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/statsmodels/tsa/ar_model.py", line 580, in _static_predict
>     return self._wrap_prediction(prediction, start, end + 1 + num_oos)
>   File "/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/statsmodels/tsa/ar_model.py", line 481, in _wrap_prediction
>     return pd.Series(prediction, index=index)
>   File "/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/pandas/core/series.py", line 299, in __init__
>     "index implies {ind}".format(val=len(data), ind=len(index))
> ValueError: Length of passed values is 11, index implies 10

</details>


If the issue has not been resolved, please file it in the issue tracker.

#### Expected Output
Return predictions for interval [start, end].

#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

[paste the output of ``import statsmodels.api as sm; sm.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
Python: 3.7.3.final.0
OS: Darwin 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64
byteorder: little
LC_ALL: None
LANG: None
statsmodels
===========
Installed: 0.11.1 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/statsmodels)
Required Dependencies
=====================
cython: 0.29.13 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/Cython)
numpy: 1.18.1 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/numpy)
scipy: 1.4.1 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/scipy)
pandas: 0.25.1 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.1 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/dateutil)
patsy: 0.5.1 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/patsy)
Optional Dependencies
=====================
matplotlib: 3.1.1 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/matplotlib)
    backend: module://backend_interagg 
cvxopt: Not installed
joblib: 0.14.1 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/joblib)
Developer Tools
================
IPython: 7.12.0 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/IPython)
    jinja2: 2.11.1 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/jinja2)
sphinx: 2.2.0 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/sphinx)
    pygments: 2.5.2 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/pygments)
pytest: 5.3.5 (/Users/mloning/.conda/envs/sktime-dev/lib/python3.7/site-packages/pytest)
virtualenv: Not installed

</details>

Hints from issue discussion:

Commit message: BUG: fixes #6553, sliced predicted values according to predicted index

Patch to review:
diff --git a/statsmodels/tsa/ar_model.py b/statsmodels/tsa/ar_model.py
index 3a6379e2cac..5d3a2305d28 100644
--- a/statsmodels/tsa/ar_model.py
+++ b/statsmodels/tsa/ar_model.py
@@ -468,8 +468,6 @@ def _setup_oos_forecast(self, add_forecasts, exog_oos):
         return x
 
     def _wrap_prediction(self, prediction, start, end):
-        if not isinstance(self.data.orig_endog, (pd.Series, pd.DataFrame)):
-            return prediction
         index = self.data.orig_endog.index
         if end > self.endog.shape[0]:
             freq = getattr(index, 'freq', None)
@@ -478,6 +476,9 @@ def _wrap_prediction(self, prediction, start, end):
             else:
                 index = pd.RangeIndex(end)
         index = index[start:end]
+        prediction = prediction[-len(index):]
+        if not isinstance(self.data.orig_endog, (pd.Series, pd.DataFrame)):
+            return prediction
         return pd.Series(prediction, index=index)
 
     def _dynamic_predict(self, params, start, end, dynamic, num_oos, exog,


Code review comments:
diff_hunk: @@ -478,6 +476,9 @@ def _wrap_prediction(self, prediction, start, end):
             else:
                 index = pd.RangeIndex(end)
         index = index[start:end]
+        prediction = prediction[-len(index):]
+        if not isinstance(self.data.orig_endog, (pd.Series, pd.DataFrame)):
path: statsmodels/tsa/ar_model.py
text: @user1:
You need to early exit in case of a NumPy array in the first line.  Is the function broken for numpy array inputs of just Series/DataFrame?

@author:
Yes, I realised that too late, didn't get the tests to run locally, not sure though if that should happen in this function or wherever the predictions are generated ...