Repository: statsmodels/statsmodels

Issue:
BUG: error in append for ARIMA model with trend
#### Describe the bug

ARIMA strange behavior depending on order of d. Am I completely wrong that it should work in both cases ? It casts origin_exog  to another size so size of feature does not fit anymore.
 

#### Code Sample, a copy-pastable example if possible

<details>

import pandas as pd
import statsmodels.api as sm
from pandas._libs.tslibs.timestamps import Timestamp


def forecast_with_arima(ar=1, d=0, ma=0):
    train = pd.DataFrame({'to_forecast': {Timestamp('2014-01-01 02:00:00'): 10.0, Timestamp('2014-01-02 02:00:00'): 5.8,
                                          Timestamp('2014-01-03 02:00:00'): 7.8,
                                          Timestamp('2014-01-04 02:00:00'): 8.8}})
    test = pd.DataFrame({'to_forecast': {Timestamp('2014-01-05 02:00:00'): 5.7, Timestamp('2014-01-06 02:00:00'): 7.9,
                                         Timestamp('2014-01-07 02:00:00'): 9.9, Timestamp('2014-01-08 02:00:00'): 8.8}})

    train.index = pd.DatetimeIndex(train.index.values, freq=train.index.inferred_freq)
    test.index = pd.DatetimeIndex(test.index.values, freq=test.index.inferred_freq)

    train.loc[(train.index >= '1-01-2013 02:00:00'), 'feature'] = 1
    test.loc[(test.index >= '1-01-2013 02:00:00'), 'feature'] = 1


    results = sm.tsa.arima.ARIMA(
        train['to_forecast'],
        exog=train[['feature']],
        order=(ar, i, ma)).fit()

    forecast = results.append(test['to_forecast'], exog=test[['feature']], refit=False)
    test['hat'] = forecast.fittedvalues



if __name__ == '__main__':
    print("Start forecast with i=1")
    # this works
    forecast_with_arima(d=1)
    print("forecast with i=1 done")

    print("Start forecast with i=0")
    # this not
    forecast_with_arima(d=0)
    print("forecast with i=0 done")

</details>



#### Expected Output

Works with d = 1 but not with d = 0.
My expectation would be it should work in both cases.

#### Real Output


Start forecast with d=1
forecast with d=1 done
Start forecast with d=0
Traceback (most recent call last):
  File "/home/user/Dokumente/folder/bug_statsmodels/statsmodels_bug.py", line 39, in <module>
    forecast_with_arima(d=0)
  File "/home/user/Dokumente/folder/bug_statsmodels/statsmodels_bug.py", line 25, in forecast_with_arima
    forecast = results.append(test['to_forecast'], exog=test[['feature']], refit=False)
  File "/home/user/anaconda3/lib/python3.7/site-packages/statsmodels/tsa/statespace/mlemodel.py", line 3665, in append
    allow_mix=True)
  File "/home/user/anaconda3/lib/python3.7/site-packages/statsmodels/tsa/statespace/tools.py", line 319, in concat
    raise ValueError('Columns must match to concatenate along'
ValueError: Columns must match to concatenate along rows.





#### Output of ``import statsmodels.api as sm; sm.show_versions()``

<details>

INSTALLED VERSIONS
------------------
Python: 3.7.6.final.0
OS: Linux 4.15.0-106-generic #107-Ubuntu SMP Thu Jun 4 11:27:52 UTC 2020 x86_64
byteorder: little
LC_ALL: None
LANG: de_DE.UTF-8

statsmodels
===========

Installed: 0.11.1 (/home/user/anaconda3/lib/python3.7/site-packages/statsmodels)

Required Dependencies
=====================

cython: 0.29.20 (/home/user/anaconda3/lib/python3.7/site-packages/Cython)
numpy: 1.18.5 (/home/user/anaconda3/lib/python3.7/site-packages/numpy)
scipy: 1.4.1 (/home/user/anaconda3/lib/python3.7/site-packages/scipy)
pandas: 1.0.1 (/home/user/anaconda3/lib/python3.7/site-packages/pandas)
    dateutil: 2.8.1 (/home/user/anaconda3/lib/python3.7/site-packages/dateutil)
patsy: 0.5.1 (/home/user/anaconda3/lib/python3.7/site-packages/patsy)

Optional Dependencies
=====================

matplotlib: 3.2.2 (/home/user/anaconda3/lib/python3.7/site-packages/matplotlib)
    backend: Qt5Agg 
cvxopt: Not installed
joblib: 0.15.1 (/home/user/anaconda3/lib/python3.7/site-packages/joblib)

Developer Tools
================

IPython: 7.15.0 (/home/user/anaconda3/lib/python3.7/site-packages/IPython)
    jinja2: 2.11.2 (/home/user/anaconda3/lib/python3.7/site-packages/jinja2)
sphinx: 3.1.1 (/home/user/anaconda3/lib/python3.7/site-packages/sphinx)
    pygments: 2.6.1 (/home/user/anaconda3/lib/python3.7/site-packages/pygments)
pytest: 5.4.3 (/home/user/anaconda3/lib/python3.7/site-packages/pytest)
virtualenv: Not installed

</details>



Discussion:
`exog` is not transformed and so must have the correct size when differencing (d=1) and not (d=0).  You are most likely 1 observation short.

Thank you for the quick answer, but i don't understand your point. Why the columns do not fit. I thought isn't about the numbers of observations it's about the columns which are not identical.
![screenshot](https://user-images.githubusercontent.com/25321403/85425610-ae68de80-b579-11ea-8531-09aaa4ef75f6.png). Here the new exog is compared with the old one of the model and it has an 'const' column now. Where does it come from? The error is raised in the line below.


const comes from `trend="c"` which is the default. You are passing `exog=train[['feature']]` in your constructor. 
Can you post a complete, runnable exam that reproduces the issue? Feel free to simulate your training data.
```python
import pandas as pd
import statsmodels.api as sm
from pandas._libs.tslibs.timestamps import Timestamp


def forecast_with_arima(ar=1, d=0, ma=0):

    train = pd.DataFrame({'to_forecast': {Timestamp('2014-01-01 02:00:00'): 10.0, Timestamp('2014-01-02 02:00:00'): 5.8,
                                          Timestamp('2014-01-03 02:00:00'): 7.8,
                                          Timestamp('2014-01-04 02:00:00'): 8.8}})

    test = pd.DataFrame({'to_forecast': {Timestamp('2014-01-05 02:00:00'): 5.7, Timestamp('2014-01-06 02:00:00'): 7.9,
                                         Timestamp('2014-01-07 02:00:00'): 9.9, Timestamp('2014-01-08 02:00:00'): 8.8}})

    train.index = pd.DatetimeIndex(train.index.values, freq=train.index.inferred_freq)
    test.index = pd.DatetimeIndex(test.index.values, freq=test.index.inferred_freq)

    train.loc[(train.index >= '1-01-2013 02:00:00'), 'feature'] = 1
    test.loc[(test.index >= '1-01-2013 02:00:00'), 'feature'] = 1


    results = sm.tsa.arima.ARIMA(
        train['to_forecast'],
        exog=train[['feature']],
        order=(ar, d, ma)).fit()

    forecast = results.append(test['to_forecast'], exog=test[['feature']], refit=False)
    test['hat'] = forecast.fittedvalues


if __name__ == '__main__':
    print("Start forecast with d=1")
    # this works
    forecast_with_arima(d=1)
    print("forecast with d=1 done")

    print("Start forecast with d=0")
    # this not
    forecast_with_arima(d=0)
    print("forecast with d=0 done")



```

this should be completly runnable, it gives the same error as shown in first comment. The training and test data are already randomly chosen and reproduce the error as well as the dummy feature.
Thanks @bashtage and @Beich2010 for looking into this!  It looks like for the `ARIMA` model, the `append` procedure has mixed up the input `exog` array that the user gave with the augmented `exog` array that includes the `trend` term.

This only shows up here for the d=0 case because the default is `trend='c'` for d=0, and `trend='n'` for d>0.


Resolving patch:
diff --git a/statsmodels/tsa/arima/model.py b/statsmodels/tsa/arima/model.py
--- a/statsmodels/tsa/arima/model.py
+++ b/statsmodels/tsa/arima/model.py
@@ -8,6 +8,7 @@
 
 import warnings
 
+from statsmodels.tools.data import _is_using_pandas
 from statsmodels.tsa.statespace import sarimax
 from statsmodels.tsa.statespace.kalman_filter import MEMORY_CONSERVE
 from statsmodels.tsa.statespace.tools import diff
@@ -133,6 +134,14 @@ def __init__(self, endog, exog=None, order=(0, 0, 0),
             dates=dates, freq=freq, missing=missing)
         exog = self._spec_arima._model.data.orig_exog
 
+        # Keep the given `exog` by removing the prepended trend variables
+        input_exog = None
+        if exog is not None:
+            if _is_using_pandas(exog, None):
+                input_exog = exog.iloc[:, self._spec_arima.k_trend:]
+            else:
+                input_exog = exog[:, self._spec_arima.k_trend:]
+
         # Initialize the base SARIMAX class
         # Note: we don't pass in a trend value to the base class, since ARIMA
         # standardizes the trend to always be part of exog, while the base
@@ -146,6 +155,14 @@ def __init__(self, endog, exog=None, order=(0, 0, 0),
             missing=missing)
         self.trend = trend
 
+        # Save the input exog and input exog names, so that we can refer to
+        # them later (see especially `ARIMAResults.append`)
+        self._input_exog = input_exog
+        if exog is not None:
+            self._input_exog_names = self.exog_names[self._spec_arima.k_trend:]
+        else:
+            self._input_exog_names = None
+
         # Override the public attributes for k_exog and k_trend to reflect the
         # distinction here (for the purpose of the superclass, these are both
         # combined as `k_exog`)
@@ -425,7 +442,30 @@ def fit(self, start_params=None, transformed=True, includes_fixed=False,
 
 @Appender(sarimax.SARIMAXResults.__doc__)
 class ARIMAResults(sarimax.SARIMAXResults):
-    pass
+
+    @Appender(sarimax.SARIMAXResults.append.__doc__)
+    def append(self, endog, exog=None, refit=False, fit_kwargs=None, **kwargs):
+        # MLEResults.append will concatenate the given `exog` here with
+        # `data.orig_exog`. However, `data.orig_exog` already has had any
+        # trend variables prepended to it, while the `exog` given here should
+        # not. Instead, we need to temporarily replace `orig_exog` and
+        # `exog_names` with the ones that correspond to those that were input
+        # by the user.
+        if exog is not None:
+            orig_exog = self.model.data.orig_exog
+            exog_names = self.model.exog_names
+            self.model.data.orig_exog = self.model._input_exog
+            self.model.exog_names = self.model._input_exog_names
+
+        # Perform the appending procedure
+        out = super().append(endog, exog=exog, refit=refit,
+                             fit_kwargs=fit_kwargs, **kwargs)
+
+        # Now we reverse the temporary change made above
+        if exog is not None:
+            self.model.data.orig_exog = orig_exog
+            self.model.exog_names = exog_names
+        return out
 
 
 class ARIMAResultsWrapper(sarimax.SARIMAXResultsWrapper):
diff --git a/statsmodels/tsa/arima/specification.py b/statsmodels/tsa/arima/specification.py
--- a/statsmodels/tsa/arima/specification.py
+++ b/statsmodels/tsa/arima/specification.py
@@ -370,7 +370,26 @@ def __init__(self, endog=None, exog=None, order=None,
                              % duplicate_ma_lags)
 
         # Handle trend
+        self.trend = trend
         self.trend_poly, _ = prepare_trend_spec(trend)
+
+        # Check for a constant column in the provided exog
+        exog_is_pandas = _is_using_pandas(exog, None)
+        if (exog is not None and len(self.trend_poly) > 0 and
+                self.trend_poly[0] == 1):
+            # Figure out if we have any constant columns
+            x = np.asanyarray(exog)
+            ptp0 = np.ptp(x, axis=0)
+            col_is_const = ptp0 == 0
+            nz_const = col_is_const & (x[0] != 0)
+            col_const = nz_const
+
+            # If we already have a constant column, raise an error
+            if np.any(col_const):
+                raise ValueError('A constant trend was included in the model'
+                                 ' specification, but the `exog` data already'
+                                 ' contains a column of constants.')
+
         # This contains the included exponents of the trend polynomial,
         # where e.g. the constant term has exponent 0, a linear trend has
         # exponent 1, etc.
@@ -404,10 +423,11 @@ def __init__(self, endog=None, exog=None, order=None,
         # Add trend data into exog
         nobs = len(endog) if exog is None else len(exog)
         if self.trend_order is not None:
+            # Add in the data
             trend_data = self.construct_trend_data(nobs, trend_offset)
             if exog is None:
                 exog = trend_data
-            elif _is_using_pandas(exog, None):
+            elif exog_is_pandas:
                 trend_data = pd.DataFrame(trend_data, index=exog.index,
                                           columns=self.construct_trend_names())
                 exog = pd.concat([trend_data, exog], axis=1)
diff --git a/statsmodels/tsa/statespace/mlemodel.py b/statsmodels/tsa/statespace/mlemodel.py
--- a/statsmodels/tsa/statespace/mlemodel.py
+++ b/statsmodels/tsa/statespace/mlemodel.py
@@ -3911,11 +3911,22 @@ def append(self, endog, exog=None, refit=False, fit_kwargs=None, **kwargs):
         new_endog = concat([self.model.data.orig_endog, endog], axis=0,
                            allow_mix=True)
 
+        # Handle `exog`
+        if exog is not None:
+            _, exog = prepare_exog(exog)
+            _check_index(append_ix, exog, '`exog`')
+
+            new_exog = concat([self.model.data.orig_exog, exog], axis=0,
+                              allow_mix=True)
+        else:
+            new_exog = None
+
         # Create a continuous index for the combined data
         if isinstance(self.model.data, PandasData):
             start = 0
             end = len(new_endog) - 1
             _, _, _, new_index = self.model._get_prediction_index(start, end)
+
             # Standardize `endog` to have the right index and columns
             columns = self.model.endog_names
             if not isinstance(columns, list):
@@ -3923,15 +3934,10 @@ def append(self, endog, exog=None, refit=False, fit_kwargs=None, **kwargs):
             new_endog = pd.DataFrame(new_endog, index=new_index,
                                      columns=columns)
 
-        # Handle `exog`
-        if exog is not None:
-            _, exog = prepare_exog(exog)
-            _check_index(append_ix, exog, '`exog`')
-
-            new_exog = concat([self.model.data.orig_exog, exog], axis=0,
-                              allow_mix=True)
-        else:
-            new_exog = None
+            # Standardize `exog` to have the right index
+            if new_exog is not None:
+                new_exog = pd.DataFrame(new_exog, index=new_index,
+                                        columns=self.model.exog_names)
 
         mod = self.model.clone(new_endog, exog=new_exog, **kwargs)
         res = self._apply(mod, refit=refit, fit_kwargs=fit_kwargs, **kwargs)


Test patch:
diff --git a/statsmodels/tsa/arima/tests/test_model.py b/statsmodels/tsa/arima/tests/test_model.py
--- a/statsmodels/tsa/arima/tests/test_model.py
+++ b/statsmodels/tsa/arima/tests/test_model.py
@@ -249,6 +249,32 @@ def test_append():
     assert_allclose(res2.llf, res_e.llf)
 
 
+def test_append_with_exog():
+    # Numpy
+    endog = dta['infl'].iloc[:100].values
+    exog = np.arange(len(endog))
+    mod = ARIMA(endog[:50], exog=exog[:50], trend='c')
+    res = mod.fit()
+    res_e = res.append(endog[50:], exog=exog[50:])
+    mod2 = ARIMA(endog, exog=exog, trend='c')
+    res2 = mod2.filter(res_e.params)
+
+    assert_allclose(res2.llf, res_e.llf)
+
+
+def test_append_with_exog_pandas():
+    # Pandas
+    endog = dta['infl'].iloc[:100]
+    exog = pd.Series(np.arange(len(endog)), index=endog.index)
+    mod = ARIMA(endog.iloc[:50], exog=exog.iloc[:50], trend='c')
+    res = mod.fit()
+    res_e = res.append(endog.iloc[50:], exog=exog.iloc[50:])
+    mod2 = ARIMA(endog, exog=exog, trend='c')
+    res2 = mod2.filter(res_e.params)
+
+    assert_allclose(res2.llf, res_e.llf)
+
+
 def test_cov_type_none():
     endog = dta['infl'].iloc[:100].values
     mod = ARIMA(endog[:50], trend='c')
diff --git a/statsmodels/tsa/statespace/tests/test_concentrated.py b/statsmodels/tsa/statespace/tests/test_concentrated.py
--- a/statsmodels/tsa/statespace/tests/test_concentrated.py
+++ b/statsmodels/tsa/statespace/tests/test_concentrated.py
@@ -86,7 +86,7 @@ def test_concentrated_loglike_sarimax():
     # Add loglikelihood burn, trend, and exog
     kwargs['loglikelihood_burn'] = 5
     kwargs['trend'] = 'c'
-    kwargs['exog'] = np.ones(nobs)
+    kwargs['exog'] = np.arange(nobs)
     out = get_sarimax_models(endog, **kwargs)
     assert_allclose(out.res_conc.llf, out.res_orig.llf)
     assert_allclose(out.res_conc.llf_obs[2:], out.res_orig.llf_obs[2:])
@@ -120,7 +120,7 @@ def test_fixed_scale_sarimax():
     kwargs = {
         'seasonal_order': (1, 1, 1, 2),
         'trend': 'ct',
-        'exog': np.ones(nobs)
+        'exog': np.sin(np.arange(nobs))
     }
 
     # Construct a concentrated version of the given SARIMAX model