Repository: statsmodels/statsmodels

Issue:
ENH: Check trend when d > 0 in ARIMA-type models
HI All,

I am new to python so forgive me if I did not explain properly.

I am reading a python time series forecast book.  In it they talk of the Arima model and show the code to estimate the model.

The code starts with 
from statsmodels.tsa.arima_model import ARIMA

to specify the model:

model = ARIMA(series, order=(5,1,0))
model_fit = model.fit(disp=0)

When the results are printed out for the model(print(model_fit.summary()  I get the following warning:
![Capture2](https://user-images.githubusercontent.com/75435215/101043455-c7010a00-3597-11eb-878a-a2e5ec1b054b.PNG)

The results of the model are as follows:
![Capture3](https://user-images.githubusercontent.com/75435215/101043607-edbf4080-3597-11eb-8516-3d076b531f28.PNG)

So when I go ahead and change the library above to read:
from statsmodels.tsa.arima.model import ARIMA
 and run the arima model again, I get the following results:
![Capture4](https://user-images.githubusercontent.com/75435215/101043834-2a8b3780-3598-11eb-84e5-54fc7671f823.PNG)

As you can see, the second results have no relation to what was obtained before. 

Does anyone have any idea why the new ARIMA is different than the old and how I can get the new one to give me the same results as the older version?

Many thanks to you all.



Discussion:
The old method seems to have found a better set of parameters. Looks at the log likelihood, it is a little bit higher.  You could try your own starting values and see what happens.  I would guess the fits are pretty similar despite the differences in parameter values. 
Another thing is that the old `ARIMA` model always included a constant by default.  In the new model, a constant is only included by default in stationary models (i.e. models in which `d = 0`, while you have `d=1`).  To replicate the old behavior, you can do:

```python
model = ARIMA(series, order=(5,1,0), trend='c')
```
Closing since I think this is answered, but feel free to follow up if you have questions.
Thank you for the replies. I tried adding a constant as advised but things did not change much in terms of estimated coefficients. For example, when you calculate the average of the residuals in the older model it was -5, but the new model (even with the constant) is +22. 
Can you share your data?
Also, what do you get when you run

```
AutoReg(Sales.diff().dropna(),5).fit().params
```

which uses OLS to estimate the model parameters. Assumes `Sales` is a `pd.Series`.
> 
> 
> Can you share your data?
Thanks a bunch Kevin. would love to.

"Yr-Month","Sales"
"1-01",266.0
"1-02",145.9
"1-03",183.1
"1-04",119.3
"1-05",180.3
"1-06",168.5
"1-07",231.8
"1-08",224.5
"1-09",192.8
"1-10",122.9
"1-11",336.5
"1-12",185.9
"2-01",194.3
"2-02",149.5
"2-03",210.1
"2-04",273.3
"2-05",191.4
"2-06",287.0
"2-07",226.0
"2-08",303.6
"2-09",289.9
"2-10",421.6
"2-11",264.5
"2-12",342.3
"3-01",339.7
"3-02",440.4
"3-03",315.9
"3-04",439.3
"3-05",401.3
"3-06",437.4
"3-07",575.5
"3-08",407.6
"3-09",682.0
"3-10",475.3
"3-11",581.3
"3-12",646.9
The book is a great time series book. My problem is that if parts of the model is deprecated, then I need to make sure I can still replicate the results of the book with the 'new'  and updated model in python.
> 
> 
> Also, what do you get when you run
> 
> ```
> AutoReg(Sales.diff().dropna(),5).fit().params
> ```
> 
> which uses OLS to estimate the model parameters. Assumes `Sales` is a `pd.Series`.

intercept    49.014374
Sales.L1     -1.201207
Sales.L2     -0.763247
Sales.L3     -0.462942
Sales.L4     -0.190935
Sales.L5      0.136667
dtype: float64

This seems to give me closer coefficient estimates with the original deprecated model 
With the new model:

```
ARIMA(Sales.diff().dropna(),order=(5,0,0),trend="c").fit().params

Out[127]: 
const       12.065186
ar.L1       -1.108131
ar.L2       -0.620273
ar.L3       -0.360547
ar.L4       -0.125167
ar.L5        0.128924
sigma2    4126.983200
dtype: float64
```
The devil is in the details.

You have to set the trend correctly:

```
ARIMA(Sales,order=(5,1,0),trend="t").fit().params

Out[138]: 
x1          12.066901
ar.L1       -1.108919
ar.L2       -0.621157
ar.L3       -0.361282
ar.L4       -0.125640
ar.L5        0.128800
sigma2    4127.586987
dtype: float64
```

@ChadFulton Sorry to bring you back in.  This seems a little surprising to me. Is this expected -- that the included strand is difference.  I noticed that the intercept when trend="c" was completely unstable which suggested to be that is the effectively being dropped. Is this the expected behavior?
@bashtage thanks for bringing this up! I had forgotten about this difference.

This is expected in the sense that `SARIMAX` (and so also `arima.ARIMA`) models the `trend` term relative to the original dataset. So you're right, for an integrated model, the constant term isn't identified, which is why it is not included by default.

As one point of support for what we do: this behavior matches how R interprets trend terms. e.g. the base `arima` function ignores the `include.mean` argument if the model has integration, and the `forecast` package's `Arima` function does what `SARIMAX` does, meaning that to match the results here, you would need to include a drift term and not a constant term:

```R
Arima(Sales, order=c(5, 1, 0), include.drift=TRUE)
```
Should it raise on `trend="c"` or `trend="ct"` when d=1?
> Should it raise on trend="c" or trend="ct" when d=1?

I agree, raising in these cases is probably the best idea.
Thank you all for your support and patience. You are invaluable.
Hello all. I have run into a similar issue as the original post. I am very new to all of this, so forgive me if the solution is obvious and has been covered above. I was also using the original ARIMA model `old_model = ARIMA(ts_log, order = (2,1,2))` with the understanding that **d = 1** would basically difference **t** and **t - 1**. 
This worked really well, but when I used the new model `new_model = ARIMA(ts_log, order = (2,1,2))`, the results were vastly different. I tried to add in `trend = c` or `trend = ct` etc. but none of those seemed to work.
Should I be using **ts_log_diff** as input to `new_model` instead? **ts_log_diff** is just a first order difference of the original series.  
I am happy to provide more details/data. Again, total newbie here, so please forgive me if this is already covered above. I was just unsure of the changes and why the old version worked but the new one doesn't. Thank you! 
Can you post a minimal example that shows the difference between the models?
Below is the code:
```
from statsmodels.tsa.arima.model import ARIMA
from statsmodels.tsa.arima_model import ARIMA as arima

ts_log_diff = ts_log - ts_log.shift()

old_model = arima(ts_log, order = (2,1,2))
new_model = ARIMA(ts_log, order =(2,1,2))

old_results = old_model.fit()
new_results = new_model.fit()

plt.plot(ts_log_diff)
plt.plot(old_results.fittedvalues, color='green')
plt.plot(new_results.fittedvalues, color='red')
```

The code given above shows both the old and new version. I will also include the plots generated by both (separately just for clarity):

OLD_MODEL:
<img width="416" alt="Screen Shot 2021-01-05 at 5 33 52 PM" src="https://user-images.githubusercontent.com/21435381/103711358-1c03b780-4f6c-11eb-9fd8-6c35484a8b0e.png">

NEW_MODEL:
<img width="396" alt="Screen Shot 2021-01-05 at 5 30 37 PM" src="https://user-images.githubusercontent.com/21435381/103711393-30e04b00-4f6c-11eb-9bf9-a9e88eebe340.png">

Not sure why the results are so different since I am passing in the same data to both. Thanks again for all the help! 
Try trend="t" in the new model.

On Tue, Jan 5, 2021, 23:40 Kausthub Poondi <notifications@github.com> wrote:

> Below is the code:
>
> from statsmodels.tsa.arima.model import ARIMA
> from statsmodels.tsa.arima_model import ARIMA as arima
>
> ts_log_diff = ts_log - ts_log.shift()
>
> old_model = arima(ts_log, order = (2,1,2))
> new_model = ARIMA(ts_log, order =(2,1,2))
>
> old_results = old_model.fit()
> new_results = new_model.fit()
>
> plt.plot(ts_log_diff)
> plt.plot(old_results.fittedvalues, color='green')
> plt.plot(new_results.fittedvalues, color='red')
>
> The code given above shows both the old and new version. I will also
> include the plots generated by both (separately just for clarity):
>
> OLD_MODEL:
> [image: Screen Shot 2021-01-05 at 5 33 52 PM]
> <https://user-images.githubusercontent.com/21435381/103711358-1c03b780-4f6c-11eb-9fd8-6c35484a8b0e.png>
>
> NEW_MODEL:
> [image: Screen Shot 2021-01-05 at 5 30 37 PM]
> <https://user-images.githubusercontent.com/21435381/103711393-30e04b00-4f6c-11eb-9bf9-a9e88eebe340.png>
>
> Not sure why the results are so different since I am passing in the same
> data to both. Thanks again for all the help!
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/7191#issuecomment-754967386>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABKTSRL524TDZVE2ER5RWWLSYOPPHANCNFSM4UMCU35A>
> .
>

```new_model = ARIMA(ts_log, order =(2,1,2), trend = "t")```
With this, I still get the same plot as above (the one in red). 
The forecasts of the old model were by default in the differenced scale, which is why they look similar when plotted against `ts_log_diff`. The new model forecasts in the scale of the original data, so it should be compared to `ts_log`.

To check that the models are producing similar results, you can do:

```python
plt.plot(ts_log)
plt.plot(old_results.predict(typ='levels'), color='green')
plt.plot(new_results.fittedvalues, color='red')
```
**Note**: I made a small change to the previous code sample.
Looks like they are producing pretty similar results, as seen below. One last thing, is there a way to get rid of the initial spike in the red plot? Not sure why it appears, but ```.fit(disp = -1)``` doesn't work anymore because it says ```disp``` isn't recognized. Thank you so so much for your help! Really appreciate it. 
<img width="397" alt="Screen Shot 2021-01-06 at 10 00 34 AM" src="https://user-images.githubusercontent.com/21435381/103790011-84e34200-4ff5-11eb-9a65-772184204589.png">

> One last thing, is there a way to get rid of the initial spike in the red plot?

The solution is to just not plot it (which is essentially what the old model does).  In the future we might want to change cases like this to set the first period to NaN. While the current output of 0 is not incorrect (it is the model's prediction before seeing any data), it is not particularly useful.

Ah ok got it. I was able to just remove the first term with a ```.drop(0)``` and that did the trick. However, I noticed that even after I solved all these issues and call the ```.predict()``` function. 

```
num_months_ahead = 20
total_months = product_df.shape[0] + num_months_ahead

##OLD
old_predictions = old_results.predict(1, total_months)
predictions_ARIMA_diff_cumsum = old_predictions.cumsum()
predictions_ARIMA_log = pd.Series([ts_log.iloc[0] for i in range(0, total_months +  1)])
predictions_ARIMA_log = predictions_ARIMA_log.add(predictions_ARIMA_diff_cumsum, fill_value=0)
predictions_ARIMA = np.exp(predictions_ARIMA_log)
print(predictions_ARIMA)

##NEW
new_predictions = new_results.predict(1, total_months)
predictions_ARIMA_new = np.exp(new_predictions)
print(predictions_ARIMA_new)
```
The first section uses the old model (which is based on a tutorial I followed). The second section uses the new model and since it isn't differencing anymore, I just basically recover the original values by exponentiating. They both predict meaningful values for in-sample predictions, but _only the old model makes sense for out of sample predictions_. The new model just eventually predicts the same value (or within a very small range). Everything else between the two looks pretty much the same now (in fact the newer model is even better than the old one in terms of accuracy!) Would you happen to know why this occurs?
Glad to hear things are working better. For this specific issue, we'd probably need an example with data that we could replicate to see what's going on. 
Sure thing! So what is being passed into the model is 
```
ts_log = np.log(ts)
plt.plot(ts_log)
test_stationarity(ts_log)
```
where ```ts``` is: 
```
0     10723
1     12818
2     15794
3     13689
4     12742
5     13261
6     12888
7     12300
8     12161
9     12482
10    12899
11    12389
12    14225
13    15800
14    15700
15    14700
16    13800
17    14300
18    14649
19    15598
20    14960
21    15152
22    15655
23    15650
24    16976
25    18228
```

With the old model, when I ask it to predict 20 months ahead, the following is returned (the output of ```print(predictions_ARIMA)``` from above): 
```
0     10723.000000
1     10929.722897
2     11430.980098
3     11441.502969
4     10553.236469
.
.
.
24    16026.353775
25    16826.569731
26    17766.065388
27    17455.037459
28    17482.395567
29    18529.172392
30    18905.241692
31    18607.908678
32    19197.703624
33    20092.013451
34    20062.835356
35    20129.758870
36    21017.305525
37    21530.181996
38    21442.901850
39    21947.612715
40    22798.706006
41    22990.065845
42    23135.591426
43    23920.244224
44    24528.551695
45    24632.661054
46    25112.486049
```

However, with the new model, when I ask it to predict 20 months ahead, the following is returned (the output of ```print(predictions_ARIMA_new)``` from above): 
```
1     10723.000075
2     13063.122445
3     14894.213399
4     12054.779638
.
.
.
24    15593.157414
25    17094.670068
26    17528.850149
27    16699.588033
28    16739.071521
29    16937.395044
30    16956.253895
31    16911.831179
32    16901.098899
33    16910.048401
34    16913.853066
35    16912.278589
36    16911.156961
37    16911.369620
38    16911.664617
39    16911.656240
40    16911.585397
41    16911.577328
42    16911.592911
43    16911.597026
44    16911.593930
45    16911.592520
46    16911.593050
```

As you can see, with the new model, it stabilizes around 16911 and just stays there forever (even if I predict 100 months ahead, the issue is the same). The old model however makes a lot more sense and predictions as we would expect. Really appreciate the help thus far :)  

Resolving patch:
diff --git a/statsmodels/tsa/arima/model.py b/statsmodels/tsa/arima/model.py
--- a/statsmodels/tsa/arima/model.py
+++ b/statsmodels/tsa/arima/model.py
@@ -8,6 +8,8 @@
 
 import warnings
 
+import numpy as np
+
 from statsmodels.tools.data import _is_using_pandas
 from statsmodels.tsa.statespace import sarimax
 from statsmodels.tsa.statespace.kalman_filter import MEMORY_CONSERVE
@@ -135,6 +137,22 @@ def __init__(self, endog, exog=None, order=(0, 0, 0),
             validate_specification=validate_specification)
         exog = self._spec_arima._model.data.orig_exog
 
+        # Raise an error if we have a constant in an integrated model
+
+        has_trend = len(self._spec_arima.trend_terms) > 0
+        if has_trend:
+            lowest_trend = np.min(self._spec_arima.trend_terms)
+            if lowest_trend < order[1] + seasonal_order[1]:
+                raise ValueError(
+                    'In models with integration (`d > 0`) or seasonal'
+                    ' integration (`D > 0`), trend terms of lower order than'
+                    ' `d + D` cannot be (as they would be eliminated due to'
+                    ' the differencing operation). For example, a constant'
+                    ' cannot be included in an ARIMA(1, 1, 1) model, but'
+                    ' including a linear trend, which would have the same'
+                    ' effect as fitting a constant to the differenced data,'
+                    ' is allowed.')
+
         # Keep the given `exog` by removing the prepended trend variables
         input_exog = None
         if exog is not None:
diff --git a/statsmodels/tsa/statespace/sarimax.py b/statsmodels/tsa/statespace/sarimax.py
--- a/statsmodels/tsa/statespace/sarimax.py
+++ b/statsmodels/tsa/statespace/sarimax.py
@@ -1732,7 +1732,7 @@ def _get_extension_time_varying_matrices(
         # Create extended model
         if extend_kwargs is None:
             extend_kwargs = {}
-        if not self.simple_differencing and self._k_trend > 0:
+        if not self.simple_differencing and self.k_trend > 0:
             extend_kwargs.setdefault(
                 'trend_offset', self.trend_offset + self.nobs)
         extend_kwargs.setdefault('validate_specification', False)


Test patch:
diff --git a/statsmodels/tsa/arima/tests/test_model.py b/statsmodels/tsa/arima/tests/test_model.py
--- a/statsmodels/tsa/arima/tests/test_model.py
+++ b/statsmodels/tsa/arima/tests/test_model.py
@@ -239,10 +239,59 @@ def test_clone():
     endog = dta['realgdp'].iloc[:100]
     exog = np.arange(endog.shape[0])
     check_cloned(ARIMA(endog, order=(2, 1, 1), seasonal_order=(1, 1, 2, 4),
-                       exog=exog, trend='c', concentrate_scale=True),
+                       exog=exog, trend=[0, 0, 1], concentrate_scale=True),
                  endog, exog=exog)
 
 
+def test_constant_integrated_model_error():
+    with pytest.raises(ValueError, match="In models with integration"):
+        ARIMA(np.ones(100), order=(1, 1, 0), trend='c')
+
+    with pytest.raises(ValueError, match="In models with integration"):
+        ARIMA(np.ones(100), order=(1, 0, 0), seasonal_order=(1, 1, 0, 6),
+              trend='c')
+
+    with pytest.raises(ValueError, match="In models with integration"):
+        ARIMA(np.ones(100), order=(1, 2, 0), trend='t')
+
+    with pytest.raises(ValueError, match="In models with integration"):
+        ARIMA(np.ones(100), order=(1, 1, 0), seasonal_order=(1, 1, 0, 6),
+              trend='t')
+
+
+def test_forecast():
+    # Numpy
+    endog = dta['infl'].iloc[:100].values
+
+    mod = ARIMA(endog[:50], order=(1, 1, 0), trend='t')
+    res = mod.filter([0.2, 0.3, 1.0])
+
+    endog2 = endog.copy()
+    endog2[50:] = np.nan
+    mod2 = mod.clone(endog2)
+    res2 = mod2.filter(res.params)
+
+    assert_allclose(res.forecast(50), res2.fittedvalues[-50:])
+
+
+def test_forecast_with_exog():
+    # Numpy
+    endog = dta['infl'].iloc[:100].values
+    exog = np.arange(len(endog))**2
+
+    mod = ARIMA(endog[:50], order=(1, 1, 0), exog=exog[:50], trend='t')
+    res = mod.filter([0.2, 0.05, 0.3, 1.0])
+
+    endog2 = endog.copy()
+    endog2[50:] = np.nan
+    mod2 = mod.clone(endog2, exog=exog)
+    print(mod.param_names)
+    print(mod2.param_names)
+    res2 = mod2.filter(res.params)
+
+    assert_allclose(res.forecast(50, exog=exog[50:]), res2.fittedvalues[-50:])
+
+
 def test_append():
     endog = dta['infl'].iloc[:100].values
     mod = ARIMA(endog[:50], trend='c')
@@ -267,6 +316,19 @@ def test_append_with_exog():
     assert_allclose(res2.llf, res_e.llf)
 
 
+def test_append_with_exog_and_trend():
+    # Numpy
+    endog = dta['infl'].iloc[:100].values
+    exog = np.arange(len(endog))**2
+    mod = ARIMA(endog[:50], exog=exog[:50], trend='ct')
+    res = mod.fit()
+    res_e = res.append(endog[50:], exog=exog[50:])
+    mod2 = ARIMA(endog, exog=exog, trend='ct')
+    res2 = mod2.filter(res_e.params)
+
+    assert_allclose(res2.llf, res_e.llf)
+
+
 def test_append_with_exog_pandas():
     # Pandas
     endog = dta['infl'].iloc[:100]