Repository: statsmodels/statsmodels

Issue:
BUG: tsa.vector_ar.var_model.VARProcess.acf gives wrong results
version: 0.6.1
1) `acf()` returns (p+1) x k x k ndarray instead of p x k x k claimed by the [documentation](http://statsmodels.sourceforge.net/stable/generated/statsmodels.tsa.vector_ar.var_model.VARProcess.acf.html#statsmodels.tsa.vector_ar.var_model.VARProcess.acf).
2) Denote the ndarray returned by `acf()` as Gamma(m, i, j). Since the code cites Lutkepohl p.28-29, I assume the results returned follow the convention of Lutkepohl 2005, but the code doesn't. In Lutkepohl, Gamma(m, i, j) is the covariance E[(X_{i}(t)-\mu_i)(X_{j}(t-m)-\mu_j)] but in the code Gamma(m, j, j) is actually E[(X_{i}(t-m)-\mu_i)(X_{j}(t)-\mu_j)].
3) Perhaps because of 2), Gamma(m, i, j) with m>p-1 are all wrong.
To prove 3):

```
from statsmodels.tsa.vector_ar.var_model import VARProcess, VAR
from statsmodels.tsa.vector_ar.util import varsim
import numpy as np

coefs = np.zeros([2, 3, 3])

coefs[0, :, :] = np.array([[0.8, 0., 0.4], [0, 0.9, 0], [0., 0.5, 0.5]])
coefs[1, :, :] = np.array([[-0.5, 0., 0.], [0., -0.8, 0.], [0., 0., -0.2]])
sigma_u = np.array([[0.3, 0., 0.], [0., 1., 0.], [0., 0., 0.2]])

model = VARProcess(coefs, np.zeros([3]), sigma_u)

ts = varsim(coefs, np.zeros([3]), sigma_u, steps=10000)

vec = np.vstack((ts[2:, 0], ts[:-2, 1]))

print(vec.shape)

print("autocovariance matrix:\n", model.acf())

print("Gamma(1, 0, 1) calculated by simulation: \n", cov(vec))
```

will give you

```
(2, 9998)
autocovariance matrix:
 [[[ 2.02348458 -1.50968049  0.16377068]
  [-1.50968049  3.7037037   0.55664169]
  [ 0.16377068  0.55664169  2.07268082]]

 [[ 1.12286396 -0.95107044 -0.96108075]
  [-0.50955249  1.85185185  2.41663951]
  [ 1.44062925 -1.43233409  1.09555104]]

 [[ 0.46280058 -0.57894975 -0.41252953]
  [ 0.74914715 -1.2962963   1.72966221]
  [ 0.43278424  0.09843054  1.34155911]]]
Gamma(1, 0, 1) calculated by simulation: 
 [[ 1.94745349  1.28036354]
 [ 1.28036354  3.61850361]]
```

Notice that the cross term in the last matrix does not match any entry in the last covariance matrix returned by `acf()`. This is wrong.



Discussion:
hstack concatenates 1-D arrays

```
>>> np.hstack((ts[1:, 0], ts[:-1, 1])).shape
(19998,)
>>> np.var(np.hstack((ts[1:, 0], ts[:-1, 1])))
1.1572533339212125
```

I get simulated acf and theoretical acf that are pretty close to each other

```
>>> print("autocovariance matrix:\n", model.acf())
autocovariance matrix:
 [[[ 1.0564832   0.12110905]
  [ 0.12110905  1.26906823]]

 [[ 0.08142651 -0.24170274]
  [ 0.26850134 -0.47129458]]]
>>> cov_emp = np.cov(np.hstack((ts[1:, :], ts[:-1, :])), rowvar=False)
>>> print(cov_emp)
[[ 1.03801628  0.0960058   0.06798033 -0.24436204]
 [ 0.0960058   1.27678491  0.26213411 -0.48283047]
 [ 0.06798033  0.26213411  1.03781303  0.09589149]
 [-0.24436204 -0.48283047  0.09589149  1.27672062]]
>>> cov_emp[:2,:2]
array([[ 1.03801628,  0.0960058 ],
       [ 0.0960058 ,  1.27678491]])
>>> cov_emp[:2,2:]
array([[ 0.06798033, -0.24436204],
       [ 0.26213411, -0.48283047]])
```

Note: in your

> Gamma(m, i, j) is the covariance E[(X_{i}(t)-\mu_i)(X_{j}(t-m)-\mu_j)] but in the code Gamma(m, j, j) is actually E[(X_{i}(t+m)-\mu_i)(X_{j}(t)-\mu_j)]

the two expectations are the same; it is just a redefinition of the index `t' = t+m`

for reference, I didn't see the function and class in the `api`

```
import numpy as np
from statsmodels.tsa.vector_ar.var_model import VARProcess
from statsmodels.tsa.vector_ar.util import varsim
```

@josef-pkt, really sorry for the mistakes. I updated the MWE (and notations) and now it contains the VAR(2) model (with 3 variables) that prompted me to submit the bug (to be confirmed) report. It seems that there is indeed no problem with the VAR(1) model, but it seems to have problem with the above-mentioned VAR(2) model.

@samuelandjw   The results for the new example look wrong to me too.
I think your updated expectations in the description are still equivalent.

If my interpretation of the shapes is correct without having checked the source, then acf at lags zero and one are correct and incorrect at two, which is case 3) in the description

```
>>> vec = np.column_stack((ts[2:, :], ts[:-2, :]))
>>> vec1 = np.column_stack((ts[1:, :], ts[:-1, :]))
>>> acf2 = model.acf()

>>> acf2[0]
array([[ 2.02348458, -1.50968049,  0.16377068],
       [-1.50968049,  3.7037037 ,  0.55664169],
       [ 0.16377068,  0.55664169,  2.07268082]])
>>> np.cov(vec1, rowvar=False)[:3,:3].T
array([[ 2.12017079, -1.5512455 ,  0.17070833],
       [-1.5512455 ,  3.78126427,  0.59532264],
       [ 0.17070833,  0.59532264,  2.13795861]])

>>> acf2[1]
array([[ 1.12286396, -0.95107044, -0.96108075],
       [-0.50955249,  1.85185185,  2.41663951],
       [ 1.44062925, -1.43233409,  1.09555104]])
>>> np.cov(vec1, rowvar=False)[:3,3:].T
array([[ 1.18003461, -0.99110389, -1.00545952],
       [-0.5048489 ,  1.89375382,  2.48913515],
       [ 1.50692884, -1.45015582,  1.13289189]])

>>> acf2[2]
array([[ 0.46280058, -0.57894975, -0.41252953],
       [ 0.74914715, -1.2962963 ,  1.72966221],
       [ 0.43278424,  0.09843054,  1.34155911]])
>>> np.cov(vec, rowvar=False)[:3,3:].T
array([[-0.52356233,  0.35343962, -1.03803674],
       [ 1.37426177, -1.31311011,  2.0671985 ],
       [ 1.5826549 , -1.77660981, -0.59947324]])

>>> model.is_stable()
True
```

As far as I can see, the problem is a missing transpose, i.e. swap the last two axis in the acf initialization

https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/vector_ar/var_model.py#L132
`result[:p] = _var_acf(coefs, sig_u).swapaxes(2,1)`

Because it's a missing/incorrect transposition, this bug doesn't show up in independent or symmetric VARs.
Needs a asymmetric test case. Currently acf, acorr only have a smoke and a shape test, AFAICS.

(terminology I thought acf is autocorrelation)

@samuelandjw   Even after fixing the calculation there is the question of what is in the rows and what's in columns.

I think I understand now part (2) better: 
For given lag `m`, `i` is the row index and `j` is the column index
`Gamma(m, i, j)` is the covariance`E[(X_{i}(t)-\mu_i)(X_{j}(t-m)-\mu_j)]`
after adding the swapaxis to `_var_acf`

The docstring doesn't say anything about which k in (p,k,k) is which, and I haven't looked at Lutkepohl.

Can you check if this is correct then.

@josef-pkt Now that the two axes are swapped, everything is fine. The entries in the matrices follow the convention of Lutkepohl 2005. 
By the way, the method name `acf()` is indeed weird for autocovariance. `acv()` might be more intuitive.


Resolving patch:
diff --git a/statsmodels/graphics/regressionplots.py b/statsmodels/graphics/regressionplots.py
--- a/statsmodels/graphics/regressionplots.py
+++ b/statsmodels/graphics/regressionplots.py
@@ -12,6 +12,7 @@
 '''
 from statsmodels.compat.python import lrange, string_types, lzip, range
 import numpy as np
+import pandas as pd
 from patsy import dmatrix
 
 from statsmodels.regression.linear_model import OLS, GLS, WLS
@@ -339,22 +340,37 @@ def plot_partregress(endog, exog_i, exog_others, data=None,
         RHS = dmatrix(RHS, data)
     else:
         RHS = exog_others
-
+    RHS_isemtpy = False
+    if isinstance(RHS, np.ndarray) and RHS.size==0:
+        RHS_isemtpy = True
+    elif isinstance(RHS, pd.DataFrame) and RHS.empty:
+        RHS_isemtpy = True
     if isinstance(exog_i, string_types):
         exog_i = dmatrix(exog_i + "-1", data)
 
     # all arrays or pandas-like
-    res_yaxis = OLS(endog, RHS).fit()
-    res_xaxis = OLS(exog_i, RHS).fit()
 
-    ax.plot(res_xaxis.resid, res_yaxis.resid, 'o', **kwargs)
-    fitted_line = OLS(res_yaxis.resid, res_xaxis.resid).fit()
+    if RHS_isemtpy:
+        ax.plot(endog, exog_i, 'o', **kwargs)
+        fitted_line = OLS(endog, exog_i).fit()
+        x_axis_endog_name = 'x' if isinstance(exog_i, np.ndarray) else exog_i.name
+        y_axis_endog_name = 'y' if isinstance(endog, np.ndarray) else endog.design_info.column_names[0]
+    else:
+        res_yaxis = OLS(endog, RHS).fit()
+        res_xaxis = OLS(exog_i, RHS).fit()
+        xaxis_resid = res_xaxis.resid
+        yaxis_resid = res_yaxis.resid
+        x_axis_endog_name = res_xaxis.model.endog_names
+        y_axis_endog_name = res_yaxis.model.endog_names
+        ax.plot(xaxis_resid, yaxis_resid, 'o', **kwargs)
+        fitted_line = OLS(yaxis_resid, xaxis_resid).fit()
+
     fig = abline_plot(0, fitted_line.params[0], color='k', ax=ax)
-    x_axis_endog_name = res_xaxis.model.endog_names
+
     if x_axis_endog_name == 'y':  # for no names regression will just get a y
         x_axis_endog_name = 'x'  # this is misleading, so use x
     ax.set_xlabel("e(%s | X)" % x_axis_endog_name)
-    ax.set_ylabel("e(%s | X)" % res_yaxis.model.endog_names)
+    ax.set_ylabel("e(%s | X)" % y_axis_endog_name)
     ax.set_title('Partial Regression Plot', **title_kwargs)
 
     #NOTE: if we want to get super fancy, we could annotate if a point is


Test patch:
diff --git a/statsmodels/graphics/tests/test_regressionplots.py b/statsmodels/graphics/tests/test_regressionplots.py
--- a/statsmodels/graphics/tests/test_regressionplots.py
+++ b/statsmodels/graphics/tests/test_regressionplots.py
@@ -1,11 +1,7 @@
-'''Tests for regressionplots, entire module is skipped
-
-'''
-
 import numpy as np
-import nose
 
 import statsmodels.api as sm
+from numpy.testing import dec
 from statsmodels.graphics.regressionplots import (plot_fit, plot_ccpr,
                   plot_partregress, plot_regress_exog, abline_plot,
                   plot_partregress_grid, plot_ccpr_grid, add_lowess,
@@ -27,15 +23,12 @@
 else:
     pdf = None
 
-def setup():
-    if not have_matplotlib:
-        raise nose.SkipTest('No tests here')
-
 def close_or_save(pdf, fig):
     if pdf_output:
         pdf.savefig(fig)
     plt.close(fig)
 
+@dec.skipif(not have_matplotlib)
 def teardown_module():
     plt.close('all')
     if pdf_output:
@@ -60,6 +53,7 @@ def setup(self):
 
         self.res = res
 
+    @dec.skipif(not have_matplotlib)
     def test_plot_fit(self):
         res = self.res
 
@@ -79,11 +73,10 @@ def test_plot_fit(self):
 
         close_or_save(pdf, fig)
 
+    @dec.skipif(not have_matplotlib)
     def test_plot_oth(self):
         #just test that they run
         res = self.res
-        endog = res.model.endog
-        exog = res.model.exog
 
         plot_fit(res, 0, y_true=None)
         plot_partregress_grid(res, exog_idx=[0,1])
@@ -111,6 +104,19 @@ def setup(self):
         y = Series(y, name="outcome")
         res = sm.OLS(y, exog0).fit()
         self.res = res
+        data = DataFrame(exog0, columns=["const", "var1", "var2"])
+        data['y'] = y
+        self.data = data
+
+class TestPlotFormula(TestPlotPandas):
+    @dec.skipif(not have_matplotlib)
+    def test_one_column_exog(self):
+        from statsmodels.formula.api import ols
+        res = ols("y~var1-1", data=self.data).fit()
+        plot_regress_exog(res, "var1")
+        res = ols("y~var1", data=self.data).fit()
+        plot_regress_exog(res, "var1")
+
 
 class TestABLine(object):
 
@@ -124,12 +130,14 @@ def setupClass(cls):
         cls.y = y
         cls.mod = mod
 
+    @dec.skipif(not have_matplotlib)
     def test_abline_model(self):
         fig = abline_plot(model_results=self.mod)
         ax = fig.axes[0]
         ax.scatter(self.X[:,1], self.y)
         close_or_save(pdf, fig)
 
+    @dec.skipif(not have_matplotlib)
     def test_abline_model_ax(self):
         fig = plt.figure()
         ax = fig.add_subplot(111)
@@ -137,12 +145,14 @@ def test_abline_model_ax(self):
         fig = abline_plot(model_results=self.mod, ax=ax)
         close_or_save(pdf, fig)
 
+    @dec.skipif(not have_matplotlib)
     def test_abline_ab(self):
         mod = self.mod
         intercept, slope = mod.params
         fig = abline_plot(intercept=intercept, slope=slope)
         close_or_save(pdf, fig)
 
+    @dec.skipif(not have_matplotlib)
     def test_abline_ab_ax(self):
         mod = self.mod
         intercept, slope = mod.params
@@ -167,6 +177,7 @@ def setupClass(cls):
 
 class TestAddedVariablePlot(object):
 
+    @dec.skipif(not have_matplotlib)
     def test_added_variable_poisson(self):
 
         np.random.seed(3446)
@@ -215,6 +226,7 @@ def test_added_variable_poisson(self):
 
 class TestPartialResidualPlot(object):
 
+    @dec.skipif(not have_matplotlib)
     def test_partial_residual_poisson(self):
 
         np.random.seed(3446)
@@ -250,6 +262,7 @@ def test_partial_residual_poisson(self):
 
 class TestCERESPlot(object):
 
+    @dec.skipif(not have_matplotlib)
     def test_ceres_poisson(self):
 
         np.random.seed(3446)
@@ -282,3 +295,7 @@ def test_ceres_poisson(self):
                 ax.set_title(ti + "\nPoisson regression\n" +
                              effect_str)
                 close_or_save(pdf, fig)
+
+if __name__ == "__main__":
+    import nose
+    nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb'], exit=False)