Repository: statsmodels/statsmodels

Issue:
DESIGN: Speed up innovations algorithm / filter with Cython or Numba?
The innovation algorithm added by @bashtage in #5042 can give us a way to compute the exact log-likelihood function of an ARMA process (see https://github.com/statsmodels/statsmodels/pull/5325/files#diff-8d4eb599f6121222f9a183e3a7205f9bR794).

I'm not sure, but it seems as though this approach could be a lot faster for ARMA models since it avoids a lot of the unnecessary operations implicit in the matrix-based state space models approach. This is especially true for SARIMA models with long seasonal effects, which we have gotten several questions about.

I know @bashtage mentioned that his existing functions can be trivially sped up by numba, which I think we're planning to make an optional dependency?

I can also easily write a Cython version, but I don't know if we want to head more into the numba direction, since the code is easier for most people to read / edit / debug?


Discussion:
I suppose one has to do the experiment to see if it really is faster.  It will probably have other warts when processes highly persistent or anti-persistent.
In general I think cython is more stable then numba, i.e. once the bugs are fixed, it's just C and we don't need to worry what the current version of JIT does. 
numba is more useful for developers that don't know cython well enough, and maybe in cases where we need more flexibility.

I think with both of you, Chad and Kevin, using cython in tsa, there is no problem putting additional filters into cython.

When I get around to Poisson time series processes, then I'd rather use numba than cython because it looks less painful for me.


@ChadFulton Can this be closed now?
I guess so - where we are right now, we have a Cythonized version of the innovations algorithm that works only for specified ARMA processes, in e.g. `innovations.arma_innovations`. It can not be used with an estimated autocovariance function like `stattools.innovations_algo` can, because it takes advantage of the specific form of the autocovariance matrix for stationary ARMA processes.

I think you mentioned in your original PR for `innovations_algo` and `innovations_filter` that Numba yielded significant performance improvements, and that still seems to me like a worthwhile improvement, but could be a different issue / PR.
Sounds like this stays open. 
PERF tag?

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -144,9 +144,10 @@
 
 exts = dict(
     _exponential_smoothers={'source': 'statsmodels/tsa/_exponential_smoothers.pyx'},  # noqa: E501
+    _innovations={'source': 'statsmodels/tsa/_innovations.pyx'},
     _hamilton_filter={'source': 'statsmodels/tsa/regime_switching/_hamilton_filter.pyx.in'},  # noqa: E501
     _kim_smoother={'source': 'statsmodels/tsa/regime_switching/_kim_smoother.pyx.in'},  # noqa: E501
-    _innovations={'source': 'statsmodels/tsa/innovations/_arma_innovations.pyx.in'},  # noqa: E501
+    _arma_innovations={'source': 'statsmodels/tsa/innovations/_arma_innovations.pyx.in'},  # noqa: E501
     linbin={'source': 'statsmodels/nonparametric/linbin.pyx'},
     _smoothers_lowess={'source': 'statsmodels/nonparametric/_smoothers_lowess.pyx'},  # noqa: E501
     kalman_loglike={'source': 'statsmodels/tsa/kalmanf/kalman_loglike.pyx',
diff --git a/statsmodels/tsa/stattools.py b/statsmodels/tsa/stattools.py
--- a/statsmodels/tsa/stattools.py
+++ b/statsmodels/tsa/stattools.py
@@ -8,7 +8,6 @@
 from statsmodels.compat.scipy import _next_regular
 
 import numpy as np
-import pandas as pd
 from numpy.linalg import LinAlgError
 from scipy import stats
 
@@ -18,6 +17,7 @@
                                              CollinearityWarning)
 from statsmodels.tools.tools import add_constant, Bunch
 from statsmodels.tsa._bds import bds
+from statsmodels.tsa._innovations import innovations_filter, innovations_algo
 from statsmodels.tsa.adfvalues import mackinnonp, mackinnoncrit
 from statsmodels.tsa.arima_model import ARMA
 from statsmodels.tsa.tsatools import lagmat, lagmat2ds, add_trend
@@ -1034,152 +1034,6 @@ def levinson_durbin_pacf(pacf, nlags=None):
     return arcoefs, acf
 
 
-def innovations_algo(acov, nobs=None, rtol=None):
-    """
-    Innovations algorithm to convert autocovariances to MA parameters
-
-    Parameters
-    ----------
-    acov : array_like
-        Array containing autocovariances including lag 0
-    nobs : int, optional
-        Number of periods to run the algorithm.  If not provided, nobs is
-        equal to the length of acovf
-    rtol : float, optional
-        Tolerance used to check for convergence. Default value is 0 which will
-        never prematurely end the algorithm. Checks after 10 iterations and
-        stops if sigma2[i] - sigma2[i - 10] < rtol * sigma2[0]. When the
-        stopping condition is met, the remaining values in theta and sigma2
-        are forward filled using the value of the final iteration.
-
-    Returns
-    -------
-    theta : ndarray
-        Innovation coefficients of MA representation. Array is (nobs, q) where
-        q is the largest index of a non-zero autocovariance. theta
-        corresponds to the first q columns of the coefficient matrix in the
-        common description of the innovation algorithm.
-    sigma2 : ndarray
-        The prediction error variance (nobs,).
-
-    Examples
-    --------
-    >>> import statsmodels.api as sm
-    >>> data = sm.datasets.macrodata.load_pandas()
-    >>> rgdpg = data.data['realgdp'].pct_change().dropna()
-    >>> acov = sm.tsa.acovf(rgdpg)
-    >>> nobs = activity.shape[0]
-    >>> theta, sigma2  = innovations_algo(acov[:4], nobs=nobs)
-
-    See Also
-    --------
-    innovations_filter
-
-    References
-    ----------
-    .. [*] Brockwell, P.J. and Davis, R.A., 2016. Introduction to time series
-        and forecasting. Springer.
-    """
-    acov = np.squeeze(np.asarray(acov))
-    if acov.ndim != 1:
-        raise ValueError('acov must be 1-d or squeezable to 1-d.')
-    rtol = 0.0 if rtol is None else rtol
-    if not isinstance(rtol, float):
-        raise ValueError('rtol must be a non-negative float or None.')
-    if nobs is not None and (nobs != int(nobs) or nobs < 1):
-        raise ValueError('nobs must be a positive integer')
-    n = acov.shape[0] if nobs is None else int(nobs)
-    max_lag = int(np.max(np.argwhere(acov != 0)))
-
-    v = np.zeros(n + 1)
-    v[0] = acov[0]
-    # Retain only the relevant columns of theta
-    theta = np.zeros((n + 1, max_lag + 1))
-    for i in range(1, n):
-        for k in range(max(i - max_lag, 0), i):
-            sub = 0
-            for j in range(max(i - max_lag, 0), k):
-                sub += theta[k, k - j] * theta[i, i - j] * v[j]
-            theta[i, i - k] = 1. / v[k] * (acov[i - k] - sub)
-        v[i] = acov[0]
-        for j in range(max(i - max_lag, 0), i):
-            v[i] -= theta[i, i - j] ** 2 * v[j]
-        # Break if v has converged
-        if i >= 10:
-            if v[i - 10] - v[i] < v[0] * rtol:
-                # Forward fill all remaining values
-                v[i + 1:] = v[i]
-                theta[i + 1:] = theta[i]
-                break
-
-    theta = theta[:-1, 1:]
-    v = v[:-1]
-    return theta, v
-
-
-def innovations_filter(endog, theta):
-    """
-    Filter observations using the innovations algorithm
-
-    Parameters
-    ----------
-    endog : array_like
-        The time series to filter (nobs,). Should be demeaned if not mean 0.
-    theta : ndarray
-        Innovation coefficients of MA representation. Array must be (nobs, q)
-        where q order of the MA.
-
-    Returns
-    -------
-    resid : ndarray
-        Array of filtered innovations
-
-    Examples
-    --------
-    >>> import statsmodels.api as sm
-    >>> data = sm.datasets.macrodata.load_pandas()
-    >>> rgdpg = data.data['realgdp'].pct_change().dropna()
-    >>> acov = sm.tsa.acovf(rgdpg)
-    >>> nobs = activity.shape[0]
-    >>> theta, sigma2  = innovations_algo(acov[:4], nobs=nobs)
-    >>> resid = innovations_filter(rgdpg, theta)
-
-    See Also
-    --------
-    innovations_algo
-
-    References
-    ----------
-    .. [*] Brockwell, P.J. and Davis, R.A., 2016. Introduction to time series
-        and forecasting. Springer.
-    """
-    orig_endog = endog
-    endog = np.squeeze(np.asarray(endog))
-    if endog.ndim != 1:
-        raise ValueError('endog must be 1-d or squeezable to 1-d.')
-    nobs = endog.shape[0]
-    n_theta, k = theta.shape
-    if nobs != n_theta:
-        raise ValueError('theta must be (nobs, q) where q is the moder order')
-    is_pandas = isinstance(orig_endog, (pd.DataFrame, pd.Series))
-    if is_pandas:
-        if len(orig_endog.index) != nobs:
-            msg = 'If endog is a Series or DataFrame, the index must ' \
-                  'correspond to the number of time series observations.'
-            raise ValueError(msg)
-    u = np.empty(nobs)
-    u[0] = endog[0]
-    for i in range(1, nobs):
-        if i < k:
-            hat = (theta[i, :i] * u[:i][::-1]).sum()
-        else:
-            hat = (theta[i] * u[i - k:i][::-1]).sum()
-        u[i] = endog[i] - hat
-    if is_pandas:
-        u = pd.Series(u, index=orig_endog.index.copy())
-    return u
-
-
 def grangercausalitytests(x, maxlag, addconst=True, verbose=True):
     """four tests for granger non causality of 2 timeseries
 


Test patch: