Repository: statsmodels/statsmodels

Issue:
IRF in VAR model with no trend
I was following this tutorial http://statsmodels.sourceforge.net/devel/vector_ar.html#class-reference This is the code

``` python
import statsmodels.api as sm
import pandas as pd
import numpy as np

mdata = sm.datasets.macrodata.load_pandas().data
dates = mdata[['year', 'quarter']].astype(int).astype(str)
quarterly = dates["year"] + "Q" + dates["quarter"]
from statsmodels.tsa.base.datetools import dates_from_str
quarterly = dates_from_str(quarterly)

mdata = mdata[['realgdp','realcons','realinv']]
mdata.index = pd.DatetimeIndex(quarterly)
mdata = np.log(mdata).diff().dropna()
```

If I now run

``` python
model = sm.tsa.VAR(mdata)
results = model.fit(4, trend = 'nc')
irf = results.irf(10)
irf.plot()
```

I get an error. However the following code runs fine

``` python
model = sm.tsa.VAR(mdata)
results = model.fit(4)
irf = results.irf(10)
irf.plot()
```

Is this a bug? Am I doing something wrong? I am using Python 2.7.6 and statsmodels 0.5.0, and this is the error

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/statsmodels/tsa/vector_ar/irf.py", line 138, in plot
    stderr = self.cov(orth=orth)
  File "/usr/lib/python2.7/dist-packages/statsmodels/tsa/vector_ar/irf.py", line 264, in cov
    covs[i] = chain_dot(Gi, self.cov_a, Gi.T)
  File "/usr/lib/python2.7/dist-packages/statsmodels/tools/tools.py", line 460, in chain_dot
    return reduce(lambda x, y: np.dot(y, x), arrs[::-1])
  File "/usr/lib/python2.7/dist-packages/statsmodels/tools/tools.py", line 460, in <lambda>
    return reduce(lambda x, y: np.dot(y, x), arrs[::-1])
ValueError: matrices are not aligned
```



Discussion:
I have problems already with pandas in the example

```
Traceback (most recent call last):
  File "E:\Josef\eclipsegworkspace\statsmodels-git\local_scripts\local_scripts\try_var_irf_issue1636.py", line 19, in <module>
    mdata.index = pd.DatetimeIndex(quarterly)
  File "C:\Programs\Python27\lib\site-packages\pandas-0.12.0-py2.7-win32.egg\pandas\tseries\index.py", line 199, in __new__
    data = tools.to_datetime(data)
  File "C:\Programs\Python27\lib\site-packages\pandas-0.12.0-py2.7-win32.egg\pandas\tseries\tools.py", line 127, in to_datetime
    return _convert_listlike(arg, box=box)
  File "C:\Programs\Python27\lib\site-packages\pandas-0.12.0-py2.7-win32.egg\pandas\tseries\tools.py", line 117, in _convert_listlike
    raise e
ValueError: Out of bounds nanosecond timestamp: 1-03-31 00:00:00
>>> pd.__version__
'0.12.0'
```

looks like a bug to me

cov_a might be cutting off a constant, even though there was None

```
>>> irf.cov_a.shape
(33, 33)
```

```
>>> irf._orth_cov().shape
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new2_py27\statsmodels\statsmodels\tsa\vector_ar\irf.py", line 571, in _orth_cov
    apiece = chain_dot(Ci, self.cov_a, Ci.T)
...
ValueError: matrices are not aligned
```

debugging with introspection, doesn't help much because I have no idea what this things mean

```
>>> dir(irf)
['G', 'H', 'P', 'T', '_A'
```

(still without looking at the code)

somewhere cov_a is created from cov_params, and truncates independent of `trend`

```
>>> irf.model.cov_params.shape
(36, 36)
```

without `nc`

```
>>> irf_c.model.cov_params.shape
(39, 39)
```

code search
`self.cov_a = model._cov_alpha`
now what's cov_alpha
`def _cov_alpha(self): self.cov_params[self.neqs:, self.neqs:]`
this cuts off the constant (one for each equation)
neqs = 3 in both cases

**fix**: don't cut off if `trend='nc'`
and I guess it's also incorrect for any other trend than 'c'

PR welcome

and my guess is that `cov_alpha` or `cov_a` is actually `cov_endog` the covariance of the parameters of the lagged endogenous variables. 
(endogs that are on the right hand side and part of exog in OLS :) (my initial proposal for exog was `predetermined`)

sorry for the "language" detour.

bump this should be prio-high

Very new to statsmodels, and I don't have Lutkepohl handy, but shouldn't it just be:

```
    @property
    def _cov_alpha(self):
        """
        Estimated covariance matrix of model coefficients ex intercept
        """
        # drop intercept
        return self.cov_params[self.k_trend*self.neqs:, self.k_trend*self.neqs:]
```

@egh   Yes, it looks like that's all there is.
cov_alpha is only used by `irf` so it won't affect any other code.

Can you open a PR, and also check the case when there is a linear trend?
It would be good to add unittests that check that `irf` works for k_trend = 0, 1 or 2.

I'm fairly certain this is fixed in #815.

How do I pull this? (I'm sorry I'm not too adept with git). But this error persists in the master branch.

``` python
import statsmodels
statsmodels.__version__
```

gives `'0.6.0.dev-2fcc5ae'` If I then run my example I get a similar error.

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "statsmodels/tsa/vector_ar/irf.py", line 138, in plot
    stderr = self.cov(orth=orth)
  File "statsmodels/tsa/vector_ar/irf.py", line 264, in cov
    covs[i] = chain_dot(Gi, self.cov_a, Gi.T)
  File "statsmodels/tools/tools.py", line 504, in chain_dot
    return reduce(lambda x, y: np.dot(y, x), arrs[::-1])
  File "statsmodels/tools/tools.py", line 504, in <lambda>
    return reduce(lambda x, y: np.dot(y, x), arrs[::-1])
ValueError: shapes (33,33) and (36,9) not aligned: 33 (dim 1) != 36 (dim 0)
```

I'll see how much work it would be to pull out the fixes from that PR.

This isn't yet fixed in that PR.

To be honest, I'm not going to be able to fix this for 0.6. It would take me more time than I have to get back into this.

This code really needs a champion, so if you're interested in working through this, I'd continue the work started in my branch and see the TODOs on #815. 


Resolving patch:
diff --git a/statsmodels/tsa/vector_ar/var_model.py b/statsmodels/tsa/vector_ar/var_model.py
--- a/statsmodels/tsa/vector_ar/var_model.py
+++ b/statsmodels/tsa/vector_ar/var_model.py
@@ -970,8 +970,8 @@ def _cov_alpha(self):
         """
         Estimated covariance matrix of model coefficients ex intercept
         """
-        # drop intercept
-        return self.cov_params[self.neqs:, self.neqs:]
+        # drop intercept and trend
+        return self.cov_params[self.k_trend*self.neqs:, self.k_trend*self.neqs:]
 
     @cache_readonly
     def _cov_sigma(self):


Test patch:
diff --git a/statsmodels/tsa/vector_ar/tests/test_var.py b/statsmodels/tsa/vector_ar/tests/test_var.py
--- a/statsmodels/tsa/vector_ar/tests/test_var.py
+++ b/statsmodels/tsa/vector_ar/tests/test_var.py
@@ -18,7 +18,8 @@
 from statsmodels.tsa.vector_ar.var_model import VAR
 
 
-from numpy.testing import assert_almost_equal, assert_equal, assert_
+from numpy.testing import (assert_almost_equal, assert_equal, assert_,
+                           assert_allclose)
 
 DECIMAL_12 = 12
 DECIMAL_6 = 6
@@ -576,6 +577,36 @@ def test_var_constant():
     model = VAR(data)
     assert_raises(ValueError, model.fit, 1)
 
+def test_irf_trend():
+    # test for irf with different trend see #1636
+    # this is a rough comparison by adding trend or subtracting mean to data
+    # to get similar AR coefficients and IRF
+    data = get_macrodata().view((float,3))
+
+    model = sm.tsa.VAR(data)
+    results = model.fit(4) #, trend = 'c')
+    irf = results.irf(10)
+
+
+    data_nc = data - data.mean(0)
+    model_nc = sm.tsa.VAR(data_nc)
+    results_nc = model_nc.fit(4, trend = 'nc')
+    irf_nc = results_nc.irf(10)
+
+    assert_allclose(irf_nc.stderr()[1:4], irf.stderr()[1:4], rtol=0.01)
+
+    trend = 1e-3 * np.arange(len(data)) / (len(data) - 1)
+    # for pandas version, currently not used, if data is a pd.DataFrame
+    #data_t = pd.DataFrame(data.values + trend[:,None], index=data.index, columns=data.columns)
+    data_t = data + trend[:,None]
+
+    model_t = sm.tsa.VAR(data_t)
+    results_t = model_t.fit(4, trend = 'ct')
+    irf_t = results_t.irf(10)
+
+    assert_allclose(irf_t.stderr()[1:4], irf.stderr()[1:4], rtol=0.03)
+
+
 if __name__ == '__main__':
     import nose
     nose.runmodule(argv=[__file__,'-vvs','-x','--pdb', '--pdb-failure'],