Repository: statsmodels/statsmodels

Issue:
ENH: pacf needs to limit nlag to ensure that it is well defined.
```
plot_pacf(np.array([ -6.,   8.,   4.,  -2.,   4.,   1.,  -2.,  -3.,  -5.,   2.,   4.,
        12.,  24., -17.,  -3.,  -4.,  -8.,  -2.,   4.,  -1.,   3.,   5.,
         7.,  -6., -26.,  17.,  -4.,  -5.,   8.,   4., -10.,  -3.,  -1.,
        -2.,  -7.,  18.,  -6.,  14.,   6.,  -4.,  13.,  -9.,  11.,   2.,
         0.,  -3.,  10., -17.,  -4.,  -6.,   3.,  -3.,  -7.,  13.,  -7.,
        -5.,   0.,  -5.,  -5.,  -6.,  25., -16.,  -1.,   9., -12.,   7.,
         2.,   2.,  -1.,  -2.,  -4.,   4., -23.,  16.,  -1.,   4.,   6.,
        -1.,   6.,   0.,  -7.,  -4.,  10.,  -7.,   1.,   5.,  -3.,  -4.,
        -1.,   4.,   1.,  -4.,   9.,  -3.,  -4.,  37.,  -5.,   4.,   1.,
         2.,  -2.,  -4.,   1.,  -1.,   3.,  -1.,  -3., -27.,  21., -13.,
        -8.,  -3.,   3.,   3.,  -4.,  -2.,  -9.,  -7.,   2., -10., -21.,
        19.,   5.,  -2.,  -7.,  -2.,  -5.,  -7.,   6.,  13.,   3.,  10.,
        -2.,   1.,  -2.,   8.,  11.,   4.,   3.,   1., -10.,  -8.,  -2.,
        -5.,  -2.,  -1.,  -2.,  -4.,  -7.,  -2.,   3.,  11.,  13.,   9.,
         6.])) 
```

returns

> .../python3.6/site-packages/statsmodels/regression/linear_model.py:1283: RuntimeWarning: invalid value encountered in sqrt
>   return rho, np.sqrt(sigmasq)

`plot_acf(np.array(...))` - on the same data - works as expected.
Digging into the source code, it appears that `statsmodels.tsa.stattools.pacf(np.array(...))` works for the default `nlags=40` but results in the same error for nlags returned by  `_prepare_data_corr_plot(x, lags, zero)` in statsmodels.graphics.tsaplots




Discussion:
@elivil Same Issue!
Specifying the `lags` will solve the problem! 

lags : int or array_like, optional
    int or Array of lag values, used on horizontal axis. Uses
    np.arange(lags) when lags is an int.  If not provided,
    ``lags=np.arange(len(corr))`` is used.
Same here!
I think we should change the default lags

lags= len(x) - 2 works, but for pacf the lags towards the end are very noisy, and it's not so cheap to compute all pacf.

maybe `lags=max(len(x) // 3, 100)` just as a rough idea, I have no idea or don't remember what other packages are doing. 

after trying a lot of ways, finally i find the solution, you should point out the method='ywm' or 'ywmle' in function plot_pacf()
Hi

I'm afraid this issue still happens, not with `plot_pacf` but with `statsmodels/tsa/stattools/pacf `
I'm using 0.10.1 version on python 3.7.4 under Linux

Indeed setting the `nlags` parameter those solve the problem, but the default value yields this problem.
I traced the problem back to the function `statsmodels/regression/linear_model/yule_walker` on the `np.sqrt(sigmasq) `on the the return line.

Is there any further information I can give you?

Cheers
Could you try 0.11.0?

On Thu, Jan 23, 2020, 17:29 LacerdaPT <notifications@github.com> wrote:

> Hi
>
> I'm afraid this issue still happens, not with plot_pacf but with statsmodels/tsa/stattools/pacf
>
> I'm using 0.10.1 version on python 3.7.4 under Linux
>
> Indeed setting the nlags parameter those solve the problem, but the
> default value yields this problem.
> I traced the problem back to the function
> statsmodels/regression/linear_model/yule_walker on the np.sqrt(sigmasq) on
> the the return line.
>
> Is there any further information I can give you?
>
> Cheers
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/statsmodels/statsmodels/issues/4663?email_source=notifications&email_token=ABKTSROVU4EYKL5LXWMRM4TQ7HHYBA5CNFSM4FAX4YI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJYE66I#issuecomment-577785721>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABKTSROGJ6W4GMKBUADD253Q7HHYBANCNFSM4FAX4YIQ>
> .
>

hm, it's worse. Now the workaround is not working anymore and it used to happen only on one of the line where I use pacf, now it happens where it didn't :|


The example above looks good:

![image](https://user-images.githubusercontent.com/5585221/73034707-93186c80-3e3d-11ea-8d45-9ff54db874ed.png)

Can you share the problematic data?
This is the data. The values are larger but they are float64.

`[59559.421351925914, 59779.939465085154, 60018.764029857295, 60039.73016102302, 60259.72675813857, 60439.623743545635, 60194.90200135414, 60749.20427945856, 60927.90161732956, 61226.407870750474, 61410.041024872335, 61554.591920929684, 61583.614900313005, 61829.939465085154, 62241.43069652397, 61695.85919328109, 61763.72675813857, 61791.559227416605, 61397.805227160585, 61928.84713660141, 62180.482262490856, 61750.74120408381, 62309.395863582016, 62194.591920929684, 62341.03425515172, 62845.10075540774, 62547.09736319063, 62849.40758037786, 62967.39342480524, 62898.97858225532, 63167.16006587027, 63518.48999374427, 62952.417746361825, 63396.407870750474, 63712.62167003363, 63574.591920929684, 63562.0019970872, 63918.003981214184, 63757.43069652397, 64038.76241908754, 63956.0600914719, 64108.01084031983, 64928.12780780575, 62727.0983681285, 63964.35323023279, 64884.74120408381, 64362.94425067879, 64608.591920929684, 64690.38909386139, 64002.84269089161, 64053.09736319063, 64257.14951586173, 64278.39342480523, 63643.17213064241, 63288.45038845091, 63567.77570802998, 62714.03064958763, 62532.074537417146, 62118.42812164653, 62099.591920929684, 62062.0019970872, 61746.71365863354, 61814.764029857295, 61717.79467715206, 61535.0600914719, 62008.97858225532, 62034.25684006381, 62613.48999374427, 62430.482262490856]`

I'm using the pacf function, not the plot_pacf. I say again because I noticed that, as far as I could tell, all the commits to fix the issue on the OP were all done on the plot_pacf. And I'm using version 0.11

The exact code is: `partial_ac = pacf(series, alpha=0.05, nlags=max(series.count(), 60))`
series is a pd.Series with freq='MS' with 69 values so nlags = 69

The problem is the fact that sigmasq becomes negative on the function yule_walker when order=42 and k=42.

Thanks in advance
The issue here is that you are trying to some something that is impossible.  You cannot compute partial ac's for lags larger than nobs // 2 (or perhaps (nobs-1)//2). The is because computing the ith partial autocorrelation requires inverting an (i-1 by i-1) matrix*.

In practice, you shouldn't estimate autocorrelations or partial autocorrelations for lags larger than about sqrt(nobs).

What is missing here is an error message telling users that this is impossible. 

\* This isn't done in the code, but this matrix must be invertible for the ac/pac to be well defined. 
That is happens at 42 is just a numerical issue when it becomes numerically 0, which then creates a nan, and the warning (note that partial acf value 42 is -1.19, which makes no sense either). 
Ah, great! 
Thank you!

Maybe apply that rule as the default value for nlags :)

Again thanks a lot!
Cheers

Resolving patch:
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -924,8 +924,10 @@ def pacf(x, nlags=None, method="ywadjusted", alpha=None):
     ----------
     x : array_like
         Observations of time series for which pacf is calculated.
-    nlags : int, default 40
-        The largest lag for which the pacf is returned.
+    nlags : int
+        The largest lag for which the pacf is returned. The default
+        is currently 40, but will change to
+        min(int(10 * np.log10(nobs)), nobs // 2 - 1) in the future
     method : str, default "ywunbiased"
         Specifies which method for the calculations to use.
 
@@ -1011,19 +1013,25 @@ def pacf(x, nlags=None, method="ywadjusted", alpha=None):
         "ldbiased",
         "ld_biased",
     )
+    x = array_like(x, "x", maxdim=1)
     method = string_like(method, "method", options=methods)
-
     alpha = float_like(alpha, "alpha", optional=True)
 
     if nlags is None:
         warnings.warn(
             "The default number of lags is changing from 40 to"
-            "min(int(10 * np.log10(nobs)), nobs - 1) after 0.12"
+            "min(int(10 * np.log10(nobs)), nobs // 2 - 1) after 0.12"
             "is released. Set the number of lags to an integer to "
             " silence this warning.",
             FutureWarning,
         )
         nlags = 40
+    if nlags >= x.shape[0] // 2:
+        raise ValueError(
+            "Can only compute partial correlations for lags up to 50% of the "
+            f"sample size. The requested nlags {nlags} must be < "
+            f"{x.shape[0] // 2}."
+        )
 
     if method in ("ols", "ols-inefficient", "ols-adjusted"):
         efficient = "inefficient" not in method


Test patch:
diff --git a/statsmodels/tsa/tests/test_stattools.py b/statsmodels/tsa/tests/test_stattools.py
--- a/statsmodels/tsa/tests/test_stattools.py
+++ b/statsmodels/tsa/tests/test_stattools.py
@@ -1048,3 +1048,9 @@ def test_acf_conservate_nanops(reset_randomstate):
     for i in range(1, 10+1):
         expected[i] = np.nansum(resid[i:] * resid[:nobs-i]) / gamma0
     assert_allclose(result, expected, rtol=1e-4, atol=1e-4)
+
+
+def test_pacf_nlags_error(reset_randomstate):
+    e = np.random.standard_normal(100)
+    with pytest.raises(ValueError, match="Can only compute partial"):
+        pacf(e, 50)