Repository: statsmodels/statsmodels

Issue:
Johansen coint eigenvector switches signs
I'm performing a rolling Johansen test for cointegration across three time series in order to obtain an online/evolving eigentvector at each new timestep. I'm using the .coint_johansen() implementation from statsmodels.tsa.vector_ar.vecm. 
My issue is that the eigenvector sometimes switches signs? The rolling window lookback for the Johansen calculation is 20 000 minutes and the absolute amount of the element values evolve ever so slowly. Hence it would be extremely unlikely that the eigenvectors would "turn backwards" from one minute to the next. The graph in the link shows the first element of the rolling eigenvector. The Johansen algorithm should output continuous values without the occasional sign reversals.

```
2018-10-31 17:20:00 [-0.01669008  0.02542677 -0.09270737]
2018-10-31 17:21:00 [-0.01677716  0.02543248 -0.09265081]
2018-10-31 17:22:00 [ 0.01684433 -0.02543571  0.09261844]
2018-10-31 17:23:00 [-0.01675599  0.02543448 -0.09266106]
2018-10-31 17:24:00 [-0.01676379  0.02543654 -0.09265483]
2018-10-31 17:25:00 [ 0.01689747 -0.02544224  0.09258631]
2018-10-31 17:26:00 [ 0.01697933 -0.02544516  0.0925532 ]
```
Refer to this post for a visual impression.

https://stackoverflow.com/questions/54945971/johansen-cointegration-test-vector-directions


Discussion:
AFAIR, without having looked at the computation again:

My guess is that this is most likely a consequence of the linear algebra. The sign of an svd singular vectors or eigenvectors is "arbitrary".
We normalize this in the estimation of the VECM model, but AFAIK there is no normalization in coint_johansen. If eigenvectors are useful information in coint_johansen, then we need to add a normalization option there. The default in VECM is to set the first submatrix to the identity matrix, which, I think, would mean that we divide the eigen vector by the first value if there is only one eigenvector.

As workaround, you could normalize/divide it in your code.

(I'm pretty sure this is correct, but this needs to be confirmed.)

For a single vector, there is also the standalone test function for cointegration, that takes one of the variables as endog in the auxiliary regression and therefore normalizes it to one.



To clarify, the underlying numpy or scipy linalg functions do not return a consistent sign. In other cases, I just normalize the sign by making the first entry positive.
In this particular case, it is quite tricky to verify a consistent sign in arrears. Simply ensuring that e.g. the first eigenvector component is always positive does not solve the problem as we're rolling along a time series. It may be the case that the first component actually approaches and crosses zero as time evolves. In this particular application (asset portfolio mean reversion) it is crucial to understand whether a component actually swung into negative or whether it bounced back into the positive range.
If you could add an option to "consistently orient" the eigenvector (which is not the same as normalizing it) it would be great but I'm not sure this is even possible. Normally you wouldn't experience this unless you put the calculations into a time sequence. Without the rolling time window context, it's not possible for the eigenvector calculation to know which orientation is "correct". 
I don't know your application, but the eigenvectors just define a subspace.
I think the only normalization that the linalg libraries impose is that the sum of squared entries is one. The rest depends on the internal implementation details of the linalg libraries.

So, I don't see how we can say whether we should move forward or backward on a line embedded in a larger space.

What we can say is whether the other entries in the eigen vector switch signs, after we have imposed an additional normalization that, for example, sets the first element to one.


Just as example case for the ambiguity:
Assume one of the values in the eigenvector goes to zero. After that, there is no way to tell what the sign of the values should be, did we "cross the zero" or did we "return to the same side".
It's not part of the definition of the eigenvectors, so I don't know how we could "fix" the sign in this case.

It might be possible to use an algorithm that has deterministic signs, i.e. sign stays the same in the windows because the algorithm "fixes" it in a consistent way.
However, AFAIK, BLAS/LAPACK don't provide anything like that, and I have never seen anything about this.

I think we agree. This issue should be dropped.
We should still add the normalization that is used by VECM.
I guess it will be relatively cheap in terms of performance, and avoids the sign ambiguity.

Actually, I'm not sure anymore. If we have two or more eigen vectors, then I think the normalization also rotates and doesn't just fix the sign and scaling, but that would be irrelevant in the one cointegrating vector case.

Resolving patch:
diff --git a/statsmodels/tsa/vector_ar/vecm.py b/statsmodels/tsa/vector_ar/vecm.py
--- a/statsmodels/tsa/vector_ar/vecm.py
+++ b/statsmodels/tsa/vector_ar/vecm.py
@@ -596,7 +596,8 @@ def coint_johansen(endog, det_order, k_ar_diff):
 
     References
     ----------
-    .. [1] Lütkepohl, H. 2005. *New Introduction to Multiple Time Series Analysis*. Springer.
+    .. [1] Lütkepohl, H. 2005. New Introduction to Multiple Time Series
+        Analysis. Springer.
     """
     import warnings
     if det_order not in [-1, 0, 1]:
@@ -662,6 +663,11 @@ def resid(y, x):
     aind = np.flipud(auind)
     a = au[aind]
     d = dt[:, aind]
+    # Normalize by first non-zero element of d, usually [0, 0]
+    # GH 5517
+    non_zero_d = d.flat != 0
+    if np.any(non_zero_d):
+        d *= np.sign(d.flat[non_zero_d][0])
 
     #  Compute the trace and max eigenvalue statistics
     lr1 = np.zeros(neqs)


Test patch:
diff --git a/statsmodels/tsa/vector_ar/tests/test_coint.py b/statsmodels/tsa/vector_ar/tests/test_coint.py
--- a/statsmodels/tsa/vector_ar/tests/test_coint.py
+++ b/statsmodels/tsa/vector_ar/tests/test_coint.py
@@ -38,6 +38,12 @@ def test_table_maxeval(self):
         assert_almost_equal(table2,
                             self.res2_m.reshape(table2.shape, order='F'))
 
+    def test_normalization(self):
+        # GH 5517
+        evec = self.res.evec
+        non_zero = evec.flat != 0
+        assert evec.flat[non_zero][0] > 0
+
 
 class TestCointJoh12(CheckCointJoh):