Repository: statsmodels/statsmodels

Issue:
PACF does not support explicit lag arrays
The documentation for `tsaplots.plot_pacf` allows the `lags` option to be array-like. In reality, passing an explicit array causes an error. The problem in `plot_pacf` is that if `lags` is not `None`, then the function uses `np.arange(lags+1)`, assuming that `lags` is an integer. This can be fixed in plot_pacf pretty easily with an extra check for `hasattr(lags, "__iter__")`.

This only fixes the superficial problem, however. The real problem is that the data for `plot_pacf` comes from `statstools.pacf`, which does _not_ allow an array to be passed as an argument. It expects an integer or nothing. Furthermore, `pacf` is implemented a couple of different ways, so I don't immediately have a fix for converting all of them to allow arrays to be passed to their lags. Also, the interface for `pacf` and friends expects an `nlags` argument, not an integer `lags` argument, so the "Right" solution may end up not being backwards compatible. :(

For large datasets with sparse lags, this can make the difference in computational tractability. In my case, for instance, I had millions of points, and I wanted only power-of-2 lags (which was a known feature of the data source).

Thanks!



Discussion:
Is it just the size of the pacf array, or the actual calculations that you would like to reduce?
How many lags (in power of 2) do you want to use? How large is your largest lag?

The problem as I see it, is that the standard definition and calculation of pacf requires that we calculate all lags up to the maximum nlags. Maybe:
- exact: I think we can get away with calculating only selective lags but we would still have to condition on a very large number of regressors, pacf(lag_i) requires conditioning on all lags up to `lag_i - 1`
- assuming pacf has many zeros and only conditioning on non-redundant information given this assumption. I don't know if any of the methods for the pacf calculation can be adjusted to do this efficiently.

(I'm just "making this up". I don't know if there is a literature for this, a very quick google search didn't show anything directly.)

What king of process has only power of 2 pacf lags?

---

related for acf: in large sample the fastest is fft, but that requires a fixed, constant interval grid.

Hi Josef,

I should probably insert a disclaimer: I'm exploring, and I really only know enough about PACF to be dangerous to those around me. So it's entirely possible that either (a) I'm looking for something that doesn't make statistical sense, and/or (b) I'm looking at the problem the wrong way. The real reason that I submitted a bug was that there's inconsistency in the interfaces and docs. The `plot_pacf` function is advertising features it doesn't (and possibly can't) support. If the end result of this is simply that the docs get updated, then that's that.

Anyways, regarding the problem, it's the computations that I'm looking to reduce, because I'm looking at all powers of 2 up to 2^15 for now, and probably 2^30 down the road. And it's entirely possible that non-powers-of-2 are useful things to look at, but the point is that the span of the lags is really high, and the computation is untenable to do it by using consecutive lags. So I guess a better term would be "sparse lags" instead of powers-of-2 lags.

The problem I'm looking at is program analysis and characterization. Many programs exhibit a lot of regularity, and I'm trying to identify the right mathematical and computational tools to pull out those features and present them clearly. I can grab a variety of features traces, and for some programs, several of them have regularities at powers of 2.

I think I understand what you're saying. I think my implicit assumption (which is an approximation) was that analyzing sparse lags would simply assume that the autocorrelation at the omitted lags are zero. I think that implies their PACF will be zero, and if they actually _were_ zero, that their contribution to the PACF of the desired strides would be zero. What worries me (and I believe this is the question you're driving at), is if the omitted lags have nonzero autocorrelation, then what happens to the value of the PACF after a string of omitted nonzeros?

I guess there are two questions then:
1) Is there actually a way to compute this without getting all of the omitted-but-nonzero terms lumped into the next highest non-omitted lag? (doesn't seem particularly likely)
2) Even if there is, is this a common/useful enough scenario that it makes sense to roll something in to statsmodels?

I'm happy to help try to provide information, but I'll defer to your stats background and opinions on statsmodels.

I think it's possible that it makes sense, but we would need to assume that left out lags have negligible effect.
just as clarification: 
The autocorrelation can still be dense, only the direct effect of a lagged observation reflected in the partial autocorrelation needs to be sparse. In analogy to OLS, we have 6 variables that are all correlated, but only the first and the 6th have an effect, 2 to 5 are correlated with the dependent variable only because they are correlated with 1 and 6. In a large sample OLS would estimate parameters for 2 to 5 close to zero, and the outcome would be almost the same as dropping 2 to 5.

I think the same should happen in a seasonal model, where, for example, what we do today depends on what we did 1 year ago, but not on what we did 200 or 400 days ago.
We have arima process classes and functions that could be used to try out some (smaller) examples, but I haven't used those in a long time. (Everybody reads in the morning what happened this day in history, which affects the rest of the day.:)

power of 2 lags up to 2^15 or 2^30 would mean only 15 to 30 values for the pacf, does it?
That wouldn't need to have heavily optimized code.

Do you have enough observations to estimate a pacf for such high lags? 
Not completely clear to me: if you don't get the "seasonality" exactly right (is the lage 2^10 or 2^10 - 5 or 2^10 + 15), then there should still be enough correlation to get a reasonable estimate if there is a large enough short-run effect, i.e. large pacf for small lags.

About statsmodels: the power of 2 case is uncommon, but for various time series analysis functionality we where discussing and have the requests for seasonal lag model. Those are very common and also have overall a sparse structure, i.e. many zeros in between the seasonal lags.

With the non-linear (2^j - s^{j-1}) spacing of the lags, yule walker equations don't have the Toeplitz structure anymore, so my guess is that it won't fit the current structure or pattern for the pacf calculations.
my guess is OLS and the non-toeplitz yule-walker analog should work

```
>>> lags = 2**np.arange(5)
>>> lags[:,None] - lags
array([[  0,  -1,  -3,  -7, -15],
       [  1,   0,  -2,  -6, -14],
       [  3,   2,   0,  -4, -12],
       [  7,   6,   4,   0,  -8],
       [ 15,  14,  12,   8,   0]])
```


Resolving patch:
diff --git a/statsmodels/graphics/tsaplots.py b/statsmodels/graphics/tsaplots.py
--- a/statsmodels/graphics/tsaplots.py
+++ b/statsmodels/graphics/tsaplots.py
@@ -7,8 +7,49 @@
 from statsmodels.graphics import utils
 from statsmodels.tsa.stattools import acf, pacf
 
+
+def _prepare_data_corr_plot(x, lags, zero):
+    zero = bool(zero)
+    irregular = False if zero else True
+    if lags is None:
+        lags = np.arange(not zero, len(x))
+    elif np.isscalar(lags):
+        lags = np.arange(not zero, int(lags) + 1)  # +1 for zero lag
+    else:
+        irregular = True
+        lags = np.asanyarray(lags).astype(np.int)
+    nlags = lags.max(0)
+
+    return lags, nlags, irregular
+
+
+def _plot_corr(ax, title, acf_x, confint, lags, irregular, use_vlines, **kwargs):
+    if irregular:
+        acf_x = acf_x[lags]
+        if confint is not None:
+            confint = confint[lags]
+
+    if use_vlines:
+        ax.vlines(lags, [0], acf_x, **kwargs)
+        ax.axhline(**kwargs)
+
+    kwargs.setdefault('marker', 'o')
+    kwargs.setdefault('markersize', 5)
+    kwargs.setdefault('linestyle', 'None')
+    ax.margins(.05)
+    ax.plot(lags, acf_x, **kwargs)
+    ax.set_title(title)
+
+    if confint is not None:
+        if lags[0] == 0:
+            lags = lags[1:]
+            confint = confint[1:]
+            acf_x = acf_x[1:]
+        ax.fill_between(lags, confint[:, 0] - acf_x, confint[:, 1] - acf_x, alpha=.25)
+
+
 def plot_acf(x, ax=None, lags=None, alpha=.05, use_vlines=True, unbiased=False,
-            fft=False, **kwargs):
+             fft=False, title='Autocorrelation', zero=True, **kwargs):
     """Plot the autocorrelation function
 
     Plots lags on the horizontal and the correlations on vertical axis.
@@ -20,9 +61,10 @@ def plot_acf(x, ax=None, lags=None, alpha=.05, use_vlines=True, unbiased=False,
     ax : Matplotlib AxesSubplot instance, optional
         If given, this subplot is used to plot in instead of a new figure being
         created.
-    lags : array_like, optional
-        Array of lag values, used on horizontal axis.
-        If not given, ``lags=np.arange(len(corr))`` is used.
+    lags : int or array_like, optional
+        int or Array of lag values, used on horizontal axis. Uses
+        np.arange(lags) when lags is an int.  If not provided,
+        ``lags=np.arange(len(corr))`` is used.
     alpha : scalar, optional
         If a number is given, the confidence intervals for the given level are
         returned. For instance if alpha=.05, 95 % confidence intervals are
@@ -33,9 +75,14 @@ def plot_acf(x, ax=None, lags=None, alpha=.05, use_vlines=True, unbiased=False,
         If False, only markers are plotted.  The default marker is 'o'; it can
         be overridden with a ``marker`` kwarg.
     unbiased : bool
-       If True, then denominators for autocovariance are n-k, otherwise n
+        If True, then denominators for autocovariance are n-k, otherwise n
     fft : bool, optional
         If True, computes the ACF via FFT.
+    title : str, optional
+        Title to place on plot.  Default is 'Autocorrelation'
+    zero : bool, optional
+        Flag indicating whether to include the 0-lag autocorrelation.
+        Default is True.
     **kwargs : kwargs, optional
         Optional keyword arguments that are directly passed on to the
         Matplotlib ``plot`` and ``axhline`` functions.
@@ -61,12 +108,7 @@ def plot_acf(x, ax=None, lags=None, alpha=.05, use_vlines=True, unbiased=False,
     """
     fig, ax = utils.create_mpl_ax(ax)
 
-    if lags is None:
-        lags = np.arange(len(x))
-        nlags = len(lags) - 1
-    else:
-        nlags = lags
-        lags = np.arange(lags + 1) # +1 for zero lag
+    lags, nlags, irregular = _prepare_data_corr_plot(x, lags, zero)
 
     confint = None
     # acf has different return type based on alpha
@@ -77,25 +119,13 @@ def plot_acf(x, ax=None, lags=None, alpha=.05, use_vlines=True, unbiased=False,
         acf_x, confint = acf(x, nlags=nlags, alpha=alpha, fft=fft,
                              unbiased=unbiased)
 
-    if use_vlines:
-        ax.vlines(lags, [0], acf_x, **kwargs)
-        ax.axhline(**kwargs)
-
-    kwargs.setdefault('marker', 'o')
-    kwargs.setdefault('markersize', 5)
-    kwargs.setdefault('linestyle', 'None')
-    ax.margins(.05)
-    ax.plot(lags, acf_x, **kwargs)
-    ax.set_title("Autocorrelation")
-
-    if confint is not None:
-        # center the confidence interval TODO: do in acf?
-        ax.fill_between(lags, confint[:,0] - acf_x, confint[:,1] - acf_x, alpha=.25)
+    _plot_corr(ax, title, acf_x, confint, lags, irregular, use_vlines, **kwargs)
 
     return fig
 
-def plot_pacf(x, ax=None, lags=None, alpha=.05, method='ywm',
-                use_vlines=True, **kwargs):
+
+def plot_pacf(x, ax=None, lags=None, alpha=.05, method='ywm', use_vlines=True,
+              title='Partial Autocorrelation', zero=True, **kwargs):
     """Plot the partial autocorrelation function
 
     Plots lags on the horizontal and the correlations on vertical axis.
@@ -107,9 +137,10 @@ def plot_pacf(x, ax=None, lags=None, alpha=.05, method='ywm',
     ax : Matplotlib AxesSubplot instance, optional
         If given, this subplot is used to plot in instead of a new figure being
         created.
-    lags : array_like, optional
-        Array of lag values, used on horizontal axis.
-        If not given, ``lags=np.arange(len(corr))`` is used.
+    lags : int or array_like, optional
+        int or Array of lag values, used on horizontal axis. Uses
+        np.arange(lags) when lags is an int.  If not provided,
+        ``lags=np.arange(len(corr))`` is used.
     alpha : scalar, optional
         If a number is given, the confidence intervals for the given level are
         returned. For instance if alpha=.05, 95 % confidence intervals are
@@ -124,11 +155,15 @@ def plot_pacf(x, ax=None, lags=None, alpha=.05, method='ywm',
         - ols - regression of time series on lags of it and on constant
         - ld or ldunbiased : Levinson-Durbin recursion with bias correction
         - ldb or ldbiased : Levinson-Durbin recursion without bias correction
-
     use_vlines : bool, optional
         If True, vertical lines and markers are plotted.
         If False, only markers are plotted.  The default marker is 'o'; it can
         be overridden with a ``marker`` kwarg.
+    title : str, optional
+        Title to place on plot.  Default is 'Partial Autocorrelation'
+    zero : bool, optional
+        Flag indicating whether to include the 0-lag autocorrelation.
+        Default is True.
     **kwargs : kwargs, optional
         Optional keyword arguments that are directly passed on to the
         Matplotlib ``plot`` and ``axhline`` functions.
@@ -154,37 +189,19 @@ def plot_pacf(x, ax=None, lags=None, alpha=.05, method='ywm',
     """
     fig, ax = utils.create_mpl_ax(ax)
 
-    if lags is None:
-        lags = np.arange(len(x))
-        nlags = len(lags) - 1
-    else:
-        nlags = lags
-        lags = np.arange(lags + 1) # +1 for zero lag
+    lags, nlags, irregular = _prepare_data_corr_plot(x, lags, zero)
 
     confint = None
-    if alpha is  None:
-        acf_x  = pacf(x, nlags=nlags, alpha=alpha, method=method)
+    if alpha is None:
+        acf_x = pacf(x, nlags=nlags, alpha=alpha, method=method)
     else:
         acf_x, confint = pacf(x, nlags=nlags, alpha=alpha, method=method)
 
-    if use_vlines:
-        ax.vlines(lags, [0], acf_x, **kwargs)
-        ax.axhline(**kwargs)
-
-    # center the confidence interval TODO: do in acf?
-    kwargs.setdefault('marker', 'o')
-    kwargs.setdefault('markersize', 5)
-    kwargs.setdefault('linestyle', 'None')
-    ax.margins(.05)
-    ax.plot(lags, acf_x, **kwargs)
-    ax.set_title("Partial Autocorrelation")
-
-    if confint is not None:
-        # center the confidence interval TODO: do in acf?
-        ax.fill_between(lags, confint[:,0] - acf_x, confint[:,1] - acf_x, alpha=.25)
+    _plot_corr(ax, title, acf_x, confint, lags, irregular, use_vlines, **kwargs)
 
     return fig
 
+
 def seasonal_plot(grouped_x, xticklabels, ylabel=None, ax=None):
     """
     Consider using one of month_plot or quarter_plot unless you need
@@ -299,22 +316,3 @@ def quarter_plot(x, dates=None, ylabel=None, ax=None):
     xticklabels = ['q1', 'q2', 'q3', 'q4']
     return seasonal_plot(x.groupby(lambda y : y.quarter), xticklabels,
                          ylabel=ylabel, ax=ax)
-
-
-if __name__ == "__main__":
-    import pandas as pd
-
-    #R code to run to load that dataset in this directory
-    #data(co2)
-    #library(zoo)
-    #write.csv(as.data.frame(list(date=as.Date(co2), co2=coredata(co2))), "co2.csv", row.names=FALSE)
-    co2 = pd.read_csv("co2.csv", index_col=0, parse_dates=True)
-    month_plot(co2.co2)
-
-    #will work when dates are sorted
-    #co2 = sm.datasets.get_rdataset("co2", cache=True)
-
-    x = pd.Series(np.arange(20),
-                  index=pd.PeriodIndex(start='1/1/1990', periods=20, freq='Q'))
-    quarter_plot(x)
-


Test patch:
diff --git a/statsmodels/graphics/tests/test_tsaplots.py b/statsmodels/graphics/tests/test_tsaplots.py
--- a/statsmodels/graphics/tests/test_tsaplots.py
+++ b/statsmodels/graphics/tests/test_tsaplots.py
@@ -22,14 +22,35 @@ def test_plot_acf():
     ax = fig.add_subplot(111)
 
     ar = np.r_[1., -0.9]
-    ma = np.r_[1.,  0.9]
+    ma = np.r_[1., 0.9]
     armaprocess = tsp.ArmaProcess(ar, ma)
-    acf = armaprocess.acf(20)[:20]
+    rs = np.random.RandomState(1234)
+    acf = armaprocess.generate_sample(100, distrvs=rs.standard_normal)
+    plot_acf(acf, ax=ax, lags=10)
     plot_acf(acf, ax=ax)
     plot_acf(acf, ax=ax, alpha=None)
 
     plt.close(fig)
 
+
+@dec.skipif(not have_matplotlib)
+def test_plot_acf_irregular():
+    # Just test that it runs.
+    fig = plt.figure()
+    ax = fig.add_subplot(111)
+
+    ar = np.r_[1., -0.9]
+    ma = np.r_[1., 0.9]
+    armaprocess = tsp.ArmaProcess(ar, ma)
+    rs = np.random.RandomState(1234)
+    acf = armaprocess.generate_sample(100, distrvs=rs.standard_normal)
+    plot_acf(acf, ax=ax, lags=np.arange(1, 11))
+    plot_acf(acf, ax=ax, lags=10, zero=False)
+    plot_acf(acf, ax=ax, alpha=None, zero=False)
+
+    plt.close(fig)
+
+
 @dec.skipif(not have_matplotlib)
 def test_plot_pacf():
     # Just test that it runs.
@@ -37,15 +58,33 @@ def test_plot_pacf():
     ax = fig.add_subplot(111)
 
     ar = np.r_[1., -0.9]
-    ma = np.r_[1.,  0.9]
+    ma = np.r_[1., 0.9]
     armaprocess = tsp.ArmaProcess(ar, ma)
-    pacf = armaprocess.pacf(20)[:20]
+    rs = np.random.RandomState(1234)
+    pacf = armaprocess.generate_sample(100, distrvs=rs.standard_normal)
     plot_pacf(pacf, ax=ax)
     plot_pacf(pacf, ax=ax, alpha=None)
 
     plt.close(fig)
 
 
+@dec.skipif(not have_matplotlib)
+def test_plot_pacf_irregular():
+    # Just test that it runs.
+    fig = plt.figure()
+    ax = fig.add_subplot(111)
+
+    ar = np.r_[1., -0.9]
+    ma = np.r_[1., 0.9]
+    armaprocess = tsp.ArmaProcess(ar, ma)
+    rs = np.random.RandomState(1234)
+    pacf = armaprocess.generate_sample(100, distrvs=rs.standard_normal)
+    plot_pacf(pacf, ax=ax, lags=np.arange(1, 11))
+    plot_pacf(pacf, ax=ax, lags=10, zero=False)
+    plot_pacf(pacf, ax=ax, alpha=None, zero=False)
+
+    plt.close(fig)
+
 @dec.skipif(not have_matplotlib)
 def test_plot_month():
     dta = sm.datasets.elnino.load_pandas().data