Repository: statsmodels/statsmodels

Issue:
arma singular matrix
Original Launchpad bug 800012: https://bugs.launchpad.net/statsmodels/+bug/800012
Reported by: josef-pktd (joep).

import numpy as np
import scikits.statsmodels.api as sm

singular matrix with zero autocorrelation timeseries in ARMA ?

exog = sm.add_constant(np.random.randn(100, 2), prepend=True)
endog = exog.sum(1) + 0.2 \* np.random.randn(100)

modarma = sm.tsa.ARMA(endog)
resarma = modarma.fit(order=(1,1))

Optimization terminated successfully.
         Current function value: 6.341930
         Iterations 12
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "sm_overview.py", line 36, in <module>
    resarma = modarma.fit(order=(1,1))
  File "E:\Josef\eclipsegworkspace\statsmodels-git\statsmodels-josef\scikits\statsmodels\tsa\arima_model.py", line 375, in fit
    bounds=bounds, iprint=disp)
  File "C:\Python26\lib\site-packages\scipy\optimize\lbfgsb.py", line 196, in fmin_l_bfgs_b
    f, g = func_and_grad(x)
  File "C:\Python26\lib\site-packages\scipy\optimize\lbfgsb.py", line 142, in func_and_grad
    f = func(x, _args)
  File "E:\Josef\eclipsegworkspace\statsmodels-git\statsmodels-josef\scikits\statsmodels\tsa\arima_model.py", line 343, in <lambda>
    loglike = lambda params: -self.loglike_kalman(params)
  File "E:\Josef\eclipsegworkspace\statsmodels-git\statsmodels-josef\scikits\statsmodels\tsa\arima_model.py", line 203, in loglike_kalman
    return KalmanFilter.loglike(params, self)
  File "E:\Josef\eclipsegworkspace\statsmodels-git\statsmodels-josef\scikits\statsmodels\tsa\kalmanf\kalmanfilter.py", line 632, in loglike
    Q_0 = dot(inv(identity(m_*2)-kron(T_mat,T_mat)),
  File "C:\Python26\lib\site-packages\numpy\linalg\linalg.py", line 445, in inv
    return wrap(solve(a, identity(a.shape[0], dtype=a.dtype)))
  File "C:\Python26\lib\site-packages\numpy\linalg\linalg.py", line 328, in solve
    raise LinAlgError, 'Singular matrix'
numpy.linalg.linalg.LinAlgError: Singular matrix



Discussion:
I still get the exception, the example is random, I didn't get it with order (1,1), but with (2,2) 

(2011-10-06 after pandas-integration merge)

I can reproduce with this data on my windows machine, python 2.7, 0.4.0 rc1, numpy 1.6.1, scipy 0.10.0b1.

http://www.pastie.org/3718289

``` python
import pandas
import statsmodels.api as sm
import dateutil

y = pandas.read_clipboard(names=['dates','y'], delimiter=",")
dates = map(dateutil.parser.parse, y.dates)
y.index = dates
del y['dates']

mod = sm.tsa.ARMA(y, freq='M').fit(order=(2,2), trend='nc')
```

There are nans in start_params.

[Edit: This is reproducible across platforms and numpy/scipy versions] 

There is a problem with the Hannan-Rissanen starting parameters. They are not necessarily stationary and invertible, yet calling inverse ar/ma assumes they were and can lead to NaNs. (Also I'm not sure the parameters I gave are stationary...but this doesn't seem to be the root of the problem). If you give [0,0,0,0] it converges but far away from "true" parameters. If you give "true" parameters, it's a bit closer.

I'm not sure what to do here yet. Replacing NaNs with zeros _works_ (but converges to a non-optimal solution). Come back to this.

``` python
start_params = mod._fit_start_params_hr((2,2,0))
from statsmodels.tsa.tsatools import _ar_invtransparams, _ma_invtransparams
k,k_ar, k_ma = 0,2,2
arcoefs = newparams[k:k+k_ar]
macoefs = newparams[k+k_ar:]
_ar_invtransparams(arcoefs)
_ma_invtransparams(macoefs)
```

For the record, gretl and stata also choke on this data, though there's nothing really to indicate the solution is fishy, except the MA roots in gretl. So I'm not _as_ worried about this.

If you ask gretl to be noisy (show iteration details), you can see that all the MA parameters it tries are "out of bounds" until it gets to an MA solution with a root on the unit circle and it stops.

I didn't explore too much in Stata but it arrives at the same "solution" we do if you set start_params = [0,0,0,0].

So the question is, fail early and loudly, or not at all? I'm inclined to raise a warning that there were NaNs in the starting values and that we are reseting them all to zero. But the user should try this with different start values and reach their own conclusions. Maybe even put a note about this in the docs.

Thoughts?

I was also trying out different starting values. I'm using a different example, and I also have problems if the mean is misspecified. But I haven't tried yet with a case where the model is correctly specified.

I would switch to different starting values automatically in this case, but I'm not sure yet if or how we want to issue a warning.  My impression is that a specification search, e.g. minimize BIC, should reject these cases.

Also besides the roots we might want to have other checks on the estimated parameters, but haven't thought much about it yet. regression diagnostics for ARMA ?

Here's another one that fails, simpler test case

``` python
inv = sm.datasets.macrodata.load().data['realinv']
arima_mod = sm.tsa.ARIMA(np.log(inv), (1,1,2)).fit()
```


Resolving patch:
diff --git a/statsmodels/tsa/arima_model.py b/statsmodels/tsa/arima_model.py
--- a/statsmodels/tsa/arima_model.py
+++ b/statsmodels/tsa/arima_model.py
@@ -416,9 +416,26 @@ def _fit_start_params_hr(self, order):
                 start_params[k:k+p+q] = coefs
             else:
                 start_params[k+p:k+p+q] = yule_walker(endog, order=q)[0]
-        if q==0 and p != 0:
+        if q == 0 and p != 0:
             arcoefs = yule_walker(endog, order=p)[0]
             start_params[k:k+p] = arcoefs
+
+        # check AR coefficients
+        if p and not np.all(np.abs(np.roots(np.r_[1,
+                                        -start_params[k:k+p]])) < 1):
+            raise ValueError("The computed initial AR coefficients are not "
+                             "stationary\nYou should induce stationarity, "
+                             "choose a different model order, or you can\n"
+                             "pass your own start_params.")
+        # check MA coefficients
+        elif q and not np.all(np.abs(np.roots(np.r_[1,
+                                            start_params[k+p:]])) < 1):
+            raise ValueError("The computed initial MA coefficients are not "
+                             "invertible\nYou should induce invertibility, "
+                             "choose a different model order, or you can\n"
+                             "pass your own start_params.")
+
+        # check MA coefficients
         return start_params
 
     def _fit_start_params(self, order, method):


Test patch:
diff --git a/statsmodels/tsa/tests/test_arima.py b/statsmodels/tsa/tests/test_arima.py
--- a/statsmodels/tsa/tests/test_arima.py
+++ b/statsmodels/tsa/tests/test_arima.py
@@ -1773,6 +1773,27 @@ def test_arimax():
 
     assert_almost_equal(res.params.values, params, 3)
 
+def test_bad_start_params():
+    endog = np.array([820.69093, 781.0103028, 785.8786988, 767.64282267,
+         778.9837648 ,   824.6595702 ,   813.01877867,   751.65598567,
+         753.431091  ,   746.920813  ,   795.6201904 ,   772.65732833,
+         793.4486454 ,   868.8457766 ,   823.07226547,   783.09067747,
+         791.50723847,   770.93086347,   835.34157333,   810.64147947,
+         738.36071367,   776.49038513,   822.93272333,   815.26461227,
+         773.70552987,   777.3726522 ,   811.83444853,   840.95489133,
+         777.51031933,   745.90077307,   806.95113093,   805.77521973,
+         756.70927733,   749.89091773,  1694.2266924 ,  2398.4802244 ,
+        1434.6728516 ,   909.73940427,   929.01291907,   769.07561453,
+         801.1112548 ,   796.16163313,   817.2496376 ,   857.73046447,
+         838.849345  ,   761.92338873,   731.7842242 ,   770.4641844 ])
+    mod = ARMA(endog, (15, 0))
+    assert_raises(ValueError, mod.fit)
+
+    from statsmodels.datasets.macrodata import load
+    inv = load().data['realinv']
+    arima_mod = ARIMA(np.log(inv), (1,1,2))
+    assert_raises(ValueError, mod.fit)
+
 if __name__ == "__main__":
     import nose
     nose.runmodule(argv=[__file__, '-vvs', '-x', '--pdb'], exit=False)