Repository: statsmodels/statsmodels

Issue:
BUG: dataframe/series concatenation in statespace results append
So, here's my code:

```
df = pd.read_csv('spy1m.txt')
spy1m = pd.Series(list(df['return']), index=pd.to_datetime(df['date']), name='data').asfreq('M', 'pad')

nobs = len(spy1m)
ntrain = len(spy1m)//2
train = spy1m[:ntrain]
mod = tsa.SARIMAX(train, orders=(2, 0, 2))
res = mod.fit()
forecasts = []
forecasts.append(res.forecast(steps=1))

for i in range(ntrain, nobs):
    res = res.append(spy1m.iloc[i:i+1])
    fc = res.forecast(steps=1)
    forecasts.append(fc)
```

This code trains the model on the first half of the data and then tries to use it to predict the second half. Unfortunately, almost all the forecasts are zero:

```
pd.concat(forecasts).plot()
```
![graph1](https://user-images.githubusercontent.com/1126907/83058106-d706c280-a025-11ea-8cd9-d4811899a4cb.png)

On the other hand, if I use `extend` instead of `append`, I get something reasonable:

![graph2](https://user-images.githubusercontent.com/1126907/83058211-05849d80-a026-11ea-8366-49c9258107c6.png)

Also, if I try to access the endog data when using `append`, all the new data is null:

```
res.data.endog

array([-2.65330e-02, -1.42430e-02,  8.34100e-03,  8.08470e-02,
        5.53050e-02,  1.14780e-02,  1.76140e-02,  2.00140e-02,
       -1.15030e-02,  5.57790e-02,  1.02410e-02,  4.89820e-02,
        2.02250e-02,  1.44610e-02, -1.56690e-02, -1.55080e-02,
        1.42200e-02,  1.91330e-02, -3.30170e-02,  3.60600e-03,
        1.02160e-02,  1.46860e-02,  4.17860e-02,  3.26260e-02,
       -2.45360e-02,  2.08360e-02, -1.75610e-02, -1.90760e-02,
        3.24160e-02,  7.49000e-04,  3.77900e-02, -9.46700e-03,
        8.02400e-03, -1.74960e-02,  3.83590e-02, -8.88000e-04,
        2.71440e-02,  2.53400e-03,  1.35760e-02,  1.27350e-02,
       -2.92740e-02,  1.98000e-03,  5.35200e-03,  2.36510e-02,
        2.58550e-02,  3.22130e-02,  1.93820e-02,  1.30930e-02,
        1.63170e-02, -2.01580e-02,  1.12970e-02,  4.43190e-02,
        3.45420e-02, -1.72410e-02, -3.11180e-02,  1.43110e-02,
        3.88010e-02,  1.52830e-02, -4.10810e-02, -8.36500e-03,
       -6.20780e-02, -2.98590e-02, -4.77800e-03,  4.91830e-02,
        1.21400e-02, -8.30190e-02, -7.82400e-03,  1.45920e-02,
       -8.79520e-02, -1.67344e-01, -7.58540e-02,  1.30810e-02,
       -8.36740e-02, -1.05450e-01,  8.24680e-02,  9.98650e-02,
        5.95880e-02, -1.07700e-03,  7.38960e-02,  3.69280e-02,
        3.52540e-02, -1.85570e-02,  6.11710e-02,  1.88270e-02,
       -3.60660e-02,  3.09910e-02,  6.06990e-02,  1.57340e-02,
       -7.97780e-02, -5.25750e-02,  6.99260e-02, -4.45980e-02,
        8.83310e-02,  3.83020e-02, -1.61000e-04,  6.76320e-02,
        2.28150e-02,  3.47390e-02,  4.50000e-05,  2.93460e-02,
       -1.15080e-02, -1.67950e-02, -2.00830e-02,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan,
                nan,          nan,          nan,          nan])
```

It seems to me that statsmodels is somehow converting my new data to null each time I `append` it. What is going on? Is this a bug or am I doing something wrong?

Below is the data:

[spy1m.txt](https://github.com/statsmodels/statsmodels/files/4690970/spy1m.txt)



Discussion:
This is completely correct behavior. Multistep forecasts quickly converge to the long run value even when the AR parameter is moderately large. Extend us making a series of short term forecasts. 
Search the issues. This has come up many times. 
But I'm not doing multi-step forecasts, am I? My intention is to only do 1-step ahead forecasts but with new data.  Could someone explain the exact difference between append and extend to me?
I'm adding new data and then forecasting the next data point, why isn't new data being added to `res.data.exog`?
Thanks for reporting this, you're right it's a bug in how we're concatenating the new data in `append`.

In the meantime, a workaround is to cast `spy1m` to a dataframe, e.g.:

```python
df = pd.read_csv('spy1m.txt')
spy1m = pd.Series(list(df['return']), index=pd.to_datetime(df['date']), name='data').asfreq('M', 'pad')
spy1m = spy1m.to_frame()

# ...
```
Thanks, that worked! I know it's trivial, but would be interested in a pull request that simply calls `to_frame` inside `append`?

Resolving patch:
diff --git a/statsmodels/tsa/statespace/tools.py b/statsmodels/tsa/statespace/tools.py
--- a/statsmodels/tsa/statespace/tools.py
+++ b/statsmodels/tsa/statespace/tools.py
@@ -310,14 +310,43 @@ def concat(series, axis=0, allow_mix=False):
         objects.
     """
     is_pandas = np.r_[[_is_using_pandas(s, None) for s in series]]
+    ndim = np.r_[[np.ndim(s) for s in series]]
+    max_ndim = np.max(ndim)
+
+    if max_ndim > 2:
+        raise ValueError('`tools.concat` does not support arrays with 3 or'
+                         ' more dimensions.')
+
+    # Make sure the iterable is mutable
+    if isinstance(series, tuple):
+        series = list(series)
+
+    # Standardize ndim
+    for i in range(len(series)):
+        if ndim[i] == 0 and max_ndim == 1:
+            series[i] = np.atleast_1d(series[i])
+        elif ndim[i] == 0 and max_ndim == 2:
+            series[i] = np.atleast_2d(series[i])
+        elif ndim[i] == 1 and max_ndim == 2 and is_pandas[i]:
+            name = series[i].name
+            series[i] = series[i].to_frame()
+            series[i].columns = [name]
+        elif ndim[i] == 1 and max_ndim == 2 and not is_pandas[i]:
+            series[i] = np.atleast_2d(series[i]).T
 
     if np.all(is_pandas):
         if isinstance(series[0], pd.DataFrame):
             base_columns = series[0].columns
         else:
             base_columns = pd.Index([series[0].name])
-        for s in series[1:]:
+        for i in range(1, len(series)):
+            s = series[i]
+
             if isinstance(s, pd.DataFrame):
+                # Handle case where we were passed a dataframe and a series
+                # to concatenate, and the series did not have a name.
+                if s.columns.equals(pd.Index([None])):
+                    s.columns = base_columns[:1]
                 s_columns = s.columns
             else:
                 s_columns = pd.Index([s.name])


Test patch:
diff --git a/statsmodels/tsa/statespace/tests/test_forecasting.py b/statsmodels/tsa/statespace/tests/test_forecasting.py
new file mode 100644
--- /dev/null
+++ b/statsmodels/tsa/statespace/tests/test_forecasting.py
@@ -0,0 +1,49 @@
+r"""
+Tests for forecasting-related features not tested elsewhere
+"""
+
+import numpy as np
+import pandas as pd
+
+import pytest
+from numpy.testing import assert_allclose
+
+from statsmodels.tsa.statespace import sarimax
+
+
+@pytest.mark.parametrize('data', ['list', 'numpy', 'range', 'date', 'period'])
+def test_append_multistep(data):
+    # Test that `MLEResults.append` works when called repeatedly
+    endog = [1., 0.5, 1.5, 0.9, 0.2, 0.34]
+    if data == 'numpy':
+        endog = np.array(endog)
+    elif data == 'range':
+        endog = pd.Series(endog)
+    elif data == 'date':
+        index = pd.date_range(start='2000-01-01', periods=6, freq='MS')
+        endog = pd.Series(endog, index=index)
+    elif data == 'period':
+        index = pd.period_range(start='2000-01', periods=6, freq='M')
+        endog = pd.Series(endog, index=index)
+
+    # Base model fitting
+    mod = sarimax.SARIMAX(endog[:2], order=(1, 0, 0))
+    res = mod.smooth([0.5, 1.0])
+    assert_allclose(res.model.endog[:, 0], [1., 0.5])
+    assert_allclose(res.forecast(1), 0.25)
+
+    # First append
+    res1 = res.append(endog[2:3])
+    assert_allclose(res1.model.endog[:, 0], [1., 0.5, 1.5])
+    assert_allclose(res1.forecast(1), 0.75)
+
+    # Second append
+    res2 = res1.append(endog[3:5])
+    assert_allclose(res2.model.endog[:, 0], [1., 0.5, 1.5, 0.9, 0.2])
+    assert_allclose(res2.forecast(1), 0.1)
+
+    # Third append
+    res3 = res2.append(endog[5:6])
+    print(res3.model.endog)
+    assert_allclose(res3.model.endog[:, 0], [1., 0.5, 1.5, 0.9, 0.2, 0.34])
+    assert_allclose(res3.forecast(1), 0.17)