Repository: statsmodels/statsmodels

Issue:
Erroe: fcast = mu + np.dot(arparams, endog[i:i+p]) 
from statsmodels.tsa.ar_model import AR
import numpy as np

sample = np.array([0.46341460943222046, 0.46341460943222046, 0.39024388790130615, 0.4146341383457184, 0.4146341383457184, 0.4146341383457184, 0.3414634168148041, 0.4390243887901306, 0.46341460943222046, 0.4390243887901306])
model = AR(sample)
model_fit = model.fit(disp=0) 
output = model_fit.predict(start=10,end=12)
print(output)

The python code is as above, When I run this code, I got following error, 
  File "AR_test.py", line 8, in <module>
    output = model_fit.predict(start=10,end=12)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\statsmodels\base\wrapper.py", line 95, in wrapper
    obj = data.wrap_output(func(results, *args, **kwargs), how)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\statsmodels\tsa\ar_model.py", line 795, in predict
    predictedvalues = self.model.predict(params, start, end, dynamic)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\statsmodels\tsa\ar_model.py", line 237, in predict
    out_of_sample)
  File "D:\anaconda\envs\tensorflow\lib\site-packages\statsmodels\tsa\ar_model.py", line 49, in _ar_predict_out_of_sample
    fcast = mu + np.dot(arparams, endog[i:i+p])
ValueError: shapes (6,) and (7,) not aligned: 6 (dim 0) != 7 (dim 0)

Could anyone suggest how to fix this issue? Thanks in advance.



Discussion:
My initial hypothesis about the maxlag default value was incorrect so I edited it with my new findings, although you can return the sample model prediction if you set maxlag to 6 or lower in the `.fit()` statement:

I believe this has to do with the `add_trend` that is getting called during `_xstackX` of the `.fit()` call.  It may have to do with the repetitive nature of the sample data (the three repeating 0.414...) as it seems to think a constant is already added, and does not add one.  Therefore the number of parameters is misshapen.  
```python
from statsmodels.tsa.ar_model import AR
import numpy as np

sample = np.array([0.46341460943222046, 0.46341460943222046, 0.39024388790130615, 
                   0.4146341383457184, 0.4146341383457184, 0.4146341383457184, 
                   0.3414634168148041, 0.4390243887901306, 0.46341460943222046, 
                   0.4390243887901306])
model1 = AR(sample)
model_fit1 = model1.fit()
len(model_fit1.params) #returns a length of 7


#Changing the data slightly
sample = np.array([0.461460943222046, 0.46341460943222046, 0.39024388790130615, 
                 0.5146341383457184, 0.2146341383457184, 0.4146341383457184, 
                 0.3414634168148041, 0.4390243887901306, 0.46341460943222046, 
                 0.4390243887901306])
model2 = AR(sample)
model_fit2 = model2.fit()
len(model_fit2.params) #returns a length of 8
```
This is the piece in `statsmodels.tsa.tsatools.add_trend` that is evaluating for the already existing constant column and by default skips if it is present.
```python
np.logical_and(np.any(np.ptp(np.asanyarray(x), axis=0) == 0, axis=0),
                                       np.all(x != 0.0, axis=0))
```
Where x is the array of lags.  For model 1 it returns true, and skips the adding constant value which leaves it one short when it does the OLS fitting which produces the model that is one value short.
```python
from statsmodels.tsa.tsatools import lagmat
x = lagmat(model1.endog, model1.k_ar, trim='both')
np.logical_and(np.any(np.ptp(np.asanyarray(x), axis=0) == 0, axis=0),
                                       np.all(x != 0.0, axis=0))
```
![image](https://user-images.githubusercontent.com/31365056/46900799-d4198700-ce6d-11e8-97c2-cf8fe1698c52.png)

Not sure if that is a bug or not, or if this is just a unique circumstance with the sample values, I also tried replacing those repeating 0.41463... with other repeating numbers and the error occurred.

Hope that helps to tease out the issue a little more clearly, and what I have found so far.

This got lost without label.
temporarily prio-high to figure out what's going on.
@jtweeder is right.  Before the call to `add_trend` column 3 is constant, so `add_trend` silently changes trend from "c" to "nc", but that information doesn't make it back to the `AR` object, which still has `k_trend` set to 1.

If instead of calling `res = model.fit()` we do:

```
res = model.fit(trend="nc")
output = res.predict(start=10, end=12)

>>> print(output)
[0.44687422 0.45608137 0.47046381]
```

Note that `res.params` here matches `model_fit.params` the OP would have gotten.  The only difference is that `model.k_trend` (the fact that this is a model attribute is a different can of worms) attribute is correctly set to 0.

Two options for how to fix this:
- in `_stackX` pass `has_constant="raise"` to `add_trend`, and if it raises then tell the user to try passing `trend="nc"`
- have `add_trend` return an updated `k_trend`

Resolving patch:
diff --git a/statsmodels/tsa/ar_model.py b/statsmodels/tsa/ar_model.py
--- a/statsmodels/tsa/ar_model.py
+++ b/statsmodels/tsa/ar_model.py
@@ -32,22 +32,22 @@ def _check_ar_start(start, k_ar, method, dynamic):
                          "or dynamic forecast. Got %d" % start)
 
 
-def _ar_predict_out_of_sample(y, params, p, k_trend, steps, start=0):
+def _ar_predict_out_of_sample(y, params, k_ar, k_trend, steps, start=0):
     mu = params[:k_trend] if k_trend else 0  # only have to worry constant
     arparams = params[k_trend:][::-1]  # reverse for dot
 
     # dynamic endogenous variable
-    endog = np.zeros(p + steps)  # this is one too big but doesn't matter
+    endog = np.zeros(k_ar + steps)  # this is one too big but doesn't matter
     if start:
-        endog[:p] = y[start-p:start]
+        endog[:k_ar] = y[start-k_ar:start]
     else:
-        endog[:p] = y[-p:]
+        endog[:k_ar] = y[-k_ar:]
 
     forecast = np.zeros(steps)
     for i in range(steps):
-        fcast = mu + np.dot(arparams, endog[i:i+p])
+        fcast = mu + np.dot(arparams, endog[i:i+k_ar])
         forecast[i] = fcast
-        endog[i + p] = fcast
+        endog[i + k_ar] = fcast
 
     return forecast
 
@@ -223,7 +223,7 @@ def predict(self, params, start=None, end=None, dynamic=False):
             return predictedvalues
 
         # just do the whole thing and truncate
-        fittedvalues = dot(self.X, params)
+        fittedvalues = np.dot(self.X, params)
 
         pv_start = max(k_ar - start, 0)
         fv_start = max(start - k_ar, 0)
@@ -388,7 +388,7 @@ def _stackX(self, k_ar, trend):
         X = lagmat(endog, maxlag=k_ar, trim='both')
         k_trend = util.get_trendorder(trend)
         if k_trend:
-            X = add_trend(X, prepend=True, trend=trend)
+            X = add_trend(X, prepend=True, trend=trend, has_constant="raise")
         self.k_trend = k_trend
         return X
 
@@ -430,7 +430,7 @@ def select_order(self, maxlag, ic, trend='c', method='mle'):
                 fit = AR(endog_tmp).fit(maxlag=lag, method=method,
                                         full_output=0, trend=trend,
                                         maxiter=100, disp=0)
-                results[lag] = eval('fit.'+ic)
+                results[lag] = getattr(fit, ic)
             bestic, bestlag = min((res, k) for k, res in iteritems(results))
 
         else:  # choose by last t-stat.
diff --git a/statsmodels/tsa/tsatools.py b/statsmodels/tsa/tsatools.py
--- a/statsmodels/tsa/tsatools.py
+++ b/statsmodels/tsa/tsatools.py
@@ -96,8 +96,11 @@ def safe_is_const(s):
                     return False
             col_const = x.apply(safe_is_const, 0)
         else:
-            col_const = np.logical_and(np.any(np.ptp(np.asanyarray(x), axis=0) == 0, axis=0),
-                                       np.all(x != 0.0, axis=0))
+            ptp0 = np.ptp(np.asanyarray(x), axis=0)
+            col_is_const = ptp0 == 0
+            nz_const = col_is_const & (x[0] != 0)
+            col_const = nz_const
+
         if np.any(col_const):
             if has_constant == 'raise':
                 raise ValueError("x already contains a constant")


Test patch:
diff --git a/statsmodels/tsa/tests/test_ar.py b/statsmodels/tsa/tests/test_ar.py
--- a/statsmodels/tsa/tests/test_ar.py
+++ b/statsmodels/tsa/tests/test_ar.py
@@ -337,6 +337,33 @@ def test_ar_select_order_tstat():
     assert_equal(res, 0)
 
 
+def test_constant_column_trend():
+    # GH#5258, after calling lagmat, the sample below has a constant column,
+    #  which used to cause the result.k_trend attribute to be set incorrectly
+    # See also GH#5538
+
+    sample = np.array([
+        0.46341460943222046, 0.46341460943222046, 0.39024388790130615,
+        0.4146341383457184, 0.4146341383457184, 0.4146341383457184,
+        0.3414634168148041, 0.4390243887901306, 0.46341460943222046,
+        0.4390243887901306])
+    model = AR(sample)
+
+    # Fitting with a constant and maxlag=7 raises because of regressor
+    #  collinearity.
+    with pytest.raises(ValueError, match="already contains"):
+        model.fit(trend="c")
+
+    res = model.fit(trend="nc")
+    assert res.k_trend == 0
+    assert res.k_ar == 7
+    assert len(res.params) == 7
+    pred = res.predict(start=10, end=12)
+    # expected numbers are regression-test
+    expected = np.array([0.44687422, 0.45608137, 0.47046381])
+    assert_allclose(pred, expected)
+
+
 #TODO: likelihood for ARX model?
 #class TestAutolagARX(object):
 #    def setup(self):