Repository: statsmodels/statsmodels

Issue:
Highly persistent test case for arma_acovf
arma_covf has special handling for highly persistent cases, which does not currently have test coverage.  This PR adds a test for such a case.  I believe this test uncovers a bug in this special case code.  Specifically, in the highly-persistent case `acovf` gets defined as:

```
acovf = np.array([np.dot(ir[:lags-t], ir[t:lags]) for t in range(lags)])
```

in order to match the manually computed answer (and the answer produced by np.correlate without the special-case handling) this will need to be changed to 

```
acovf = np.array([np.dot(ir[:-lags-t], ir[t:-lags]) for t in range(lags)])
```

I'll add that as a separate commit after confirming that the test fails.


Discussion:
That looks likely to be a bug.

however, you need to watch out for `-0` as endpoint in the slice
`np.arange(5)[:-0]`

Good catch.  We can short-circuit most of the function and just return `np.array([])` in that case.
What I meant was that you need to use len() to get the full series in the slice
`x[:len(x) - lags]`
so that lags=0  uses x[:]

(a favorite bug of mine in initial versions)
Yes.  I can check for that in the next commit.  What I meant is that particular case can be avoided entirely.  The method's last line is `return acovf[:lags]`.  So if for some reason a user passes `lags=0`, we can skip a bunch of steps.
```
FAIL: statsmodels.tsa.tests.test_arima_process.test_arma_acovf_persistent
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python2.7/site-packages/statsmodels-0.8.0-py2.7-linux-x86_64.egg/statsmodels/tsa/tests/test_arima_process.py", line 61, in test_arma_acovf_persistent
    assert_almost_equal(res, expected, 6)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python2.7/site-packages/numpy/testing/utils.py", line 523, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python2.7/site-packages/numpy/testing/utils.py", line 918, in assert_array_almost_equal
    precision=decimal)
  File "/home/travis/miniconda2/envs/statsmodels-test/lib/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 6 decimals

(mismatch 100.0%)
 x: array([ 9.955131,  8.959611,  7.964093,  6.968577,  5.973063,  4.97755 ,
        3.982038,  2.986528,  1.991018,  0.995509])
 y: array([ 1000.250063,   999.749937,   999.250063,   998.750437,
         998.251062,   997.751937,   997.253061,   996.754434,
         996.256057,   995.757929])
```
Wow, resolving conflicts with GH's editor is a breeze.

[![Coverage Status](https://coveralls.io/builds/11527811/badge)](https://coveralls.io/builds/11527811)

Coverage increased (+0.004%) to 90.437% when pulling **b24d29139eba2dcfd492138719b57a74dd28db5f on jbrockmendel:patch-3** into **90e2af8be80114179deb6b0ca3ac71975c7f9064 on statsmodels:master**.

needs interactive rebase, has a merge of master into this branch, and maybe some squashing would be useful.

@jbrockmendel  we never merge master into a feature branch, we rebase on master to update the feature branch.

Resolving patch:
diff --git a/statsmodels/tsa/arima_process.py b/statsmodels/tsa/arima_process.py
--- a/statsmodels/tsa/arima_process.py
+++ b/statsmodels/tsa/arima_process.py
@@ -127,7 +127,9 @@ def arma_acovf(ar, ma, nobs=10):
         ir = arma_impulse_response(ar, ma, leads=nobs_ir)
     # again no idea where the speed break points are:
     if nobs_ir > 50000 and nobs < 1001:
-        acovf = np.array([np.dot(ir[:nobs - t], ir[t:nobs])
+        end = len(ir)
+        # Explitly slice from the end to avoid foo[:-0] returning an empty slice
+        acovf = np.array([np.dot(ir[:end-nobs-t], ir[t:end-nobs])
                           for t in range(nobs)])
     else:
         acovf = np.correlate(ir, ir, 'full')[len(ir) - 1:]


Test patch:
diff --git a/statsmodels/tsa/tests/test_arima_process.py b/statsmodels/tsa/tests/test_arima_process.py
--- a/statsmodels/tsa/tests/test_arima_process.py
+++ b/statsmodels/tsa/tests/test_arima_process.py
@@ -7,6 +7,7 @@
 
 import numpy as np
 from numpy.testing import (assert_array_almost_equal, assert_almost_equal,
+                           assert_allclose,
                            assert_equal, assert_raises, assert_, dec)
 
 from statsmodels.tsa.arima_process import (arma_generate_sample, arma_acovf,
@@ -41,6 +42,24 @@ def test_arma_acovf():
     rep2 = [1. * sigma * phi ** i / (1 - phi ** 2) for i in range(N)]
     assert_almost_equal(rep1, rep2, 7)  # 7 is max precision here
 
+def test_arma_acovf_persistent():
+    # Test arma_acovf in case where there is a near-unit root.
+    # .999 is high enough to trigger the "while ir[-1] > 5*1e-5:" clause,
+    # but not high enough to trigger the "nobs_ir > 50000" clause.
+    ar = np.array([1, -.9995])
+    ma = np.array([1])
+    process = ArmaProcess(ar, ma)
+    res = process.acovf(10)
+
+    # Theoretical variance sig2 given by:
+    # sig2 = .9995**2 * sig2 + 1
+    sig2 = 1/(1-.9995**2)
+
+    corrs = .9995**np.arange(10)
+    expected = sig2*corrs
+    assert_equal(res.ndim, 1)
+    assert_allclose(res, expected, atol=1e-6)
+    # atol=7 breaks at .999, worked at .995
 
 def test_arma_acf():
     # Check for specific AR(1)