Repository: statsmodels/statsmodels

Issue:
ENH: Migrate pandas MovingOLS
Part of moving the pandas stats stuff to statsmodels to support.

see also #2745



Discussion:
Link to code:

https://github.com/pydata/pandas/blob/1d94dd2ada7d0af57c245b083d26d60b8649a04e/pandas/stats/ols.py#L601

See also #2203 

https://github.com/pydata/pandas/blob/master/pandas/stats/ols.py#L819
pandas moving window is using cumulative moment matrices, as discussed many years ago.

There are several alternatives for the linear algebra with trade-offs for speed and numerical stability.
expanding window, i.e. recursive OLS uses now the kalman filter.

Compared to recursive estimation, we have the problem of downdating the observation that we drop as the window moves.
Another alternative (just an idea right now while looking the the recursive estimation formulas):
We can store the updating change in the inverse moment matrix and remove it again when we move the window, i.e. instead of calculating a downdating step, we just keep the updating step around for reuse. 
(In the SUR (e.g. VAR or contemporaneous correlation) case we can also recursively update the cov_resid matrix. formula in PcGive documentation 2 chapter 11 p.142 for system estimation. )

another stackoverflow question and link to code in old version https://stackoverflow.com/questions/44380068/pandas-rolling-regression-alternatives-to-looping

and another look at the code:
It looks like pandas is doing three loops through the period/nobs to get the basic results
- params: based on cumulative x'x 
- rolling_rank to get the `df`
- `_var_beta_raw` to get the cov_params, using optionally HAC/Newey-West

some of the other results can be computed in a vectorized way from above, but there are additional results that use again a loop

- _resid_stats uses a loop to compute sum of squares, ssr, sst used for example for rsquared

Going 3, 4 or more times through the nobs loop might not save much compared to doing the full OLS loop, assuming we want to get more results than the minimal estimate.

One possibility might then be to do a hybrid
To a plain OLS loop to store some basic statistics, then use the existing methods from pandas to compute the results that can be computed in the vectorized form based on those statistics.

(With cov_type nonrobust we should be able to compute more from the (cumulative) cross-products x'x for each observation observation, but with robust cov_type, including newey-west, we need the extra computation based on each window, and we need to get the residuals for each window.
I just hit this when porting some code, and it's going to be more of an issue down the road as codebases gradually start migrating up to modern pandas.  Is there anything project outsiders can do to help?
(I still haven't used pandas moving ols, but have a better idea now after browsing the code.)

The relatively cheap way would be to take the entire module or class and replace extra dependencies (like pandas.math with more generic numpy or statsmodels functions but keep to looping logic without essential changes, plus adjust some API conventions/namings to statsmodels.

The larger solution would be to refactor and reimplement some things the avoids repeated loops, possibly op for the user (e.g. the user needs to decide in advance if robust standard errors like HAC/Newey-West are needed). This would most likely require checking the timing to see what is really worth it. In this case the overall windowing logic could be kept from the pandas version, but with fewer loops for extra results.

An alternative would be to write a single pass version where we compute an OLS for each window, but the user has to decide in advance which results should be kept. I think this would look more like the recipes/discussions on stackoverflow to reuse statsmodels OLS. 

The first solution should work as a relatively quick replacement for what pandas had.


@josef-pkt I just did the "relatively cheap way" while working on #3866, and would be happy to submit a PR while I get started on actually refactoring it, if you think it's worth the trouble.  What sort of API/naming changes would I have to make for this to be more statsmodels-like?
@jengelman   Based on what I remember from looking at it in summer, I would think that the initial port could leave all the structure as is. The API would be better if it were consistent with our naming conventions, e.g. in OLS and RegressionResults.
Also unit tests against an explicit loop over OLS could verify that we have the same definitions and defaults in the moving version as in the simple OLS version. (I remember seeing some pandas issues for HAC/newey_west but never looked at those.) Unit tests to cross-check the two models would be relatively easy to write.

The overall pattern and usecase is quite different from our main models which only produce a single estimation result, so the structure will not be close to them, but it would be useful for consistency and lower learning cost for users if it would "feel" like a vectorized version of statsmodels.OLS.

An issue for future development, extensions and refactorings is to see whether it is possible to isolate the API a bit from the internal structure and computation, or to make the model/windowing structure general or modular enough to make code reuse for extensions easier. (As example, in our current models it was and is relatively easy to add new fit methods like fit_regularized or fit_constrained with relatively little extra effort and code.)

That's mostly pretty vague, but I would have to look at specific details to know more.


Resolving patch:
diff --git a/statsmodels/regression/rolling.py b/statsmodels/regression/rolling.py
new file mode 100644
--- /dev/null
+++ b/statsmodels/regression/rolling.py
@@ -0,0 +1,733 @@
+"""
+Rolling OLS and WLS
+
+Implements an efficient rolling estimator that avoids repeated matrix
+multiplication.
+
+Copyright (c) 2019 Kevin Sheppard
+License: 3-clause BSD
+"""
+from statsmodels.compat.numpy import lstsq
+
+from collections import namedtuple
+
+import numpy as np
+from pandas import Series, DataFrame, MultiIndex
+from pandas.util._decorators import Appender, Substitution
+from scipy import stats
+
+from statsmodels.base import model
+from statsmodels.base.model import LikelihoodModelResults, Model
+from statsmodels.compat.pandas import cache_readonly
+from statsmodels.regression.linear_model import RegressionResults, \
+    RegressionModel
+from statsmodels.tools.validation import array_like, int_like, string_like
+
+
+def strip4(line):
+    if line.startswith(' '):
+        return line[4:]
+    return line
+
+
+RollingStore = namedtuple('RollingStore', ['params', 'ssr', 'llf', 'nobs',
+                                           's2', 'xpxi', 'xeex',
+                                           'centered_tss', 'uncentered_tss'])
+
+common_params = '\n'.join(map(strip4, model._model_params_doc.split('\n')))
+window_parameters = """\
+window : int
+    Length of the rolling window. Must be strictly larger than the number
+    of variables in the model.
+"""
+
+weight_parameters = """
+weights : array_like, optional
+    A 1d array of weights.  If you supply 1/W then the variables are
+    pre- multiplied by 1/sqrt(W).  If no weights are supplied the
+    default value is 1 and WLS results are the same as OLS.
+"""
+
+_missing_param_doc = """\
+min_nobs : {int, None}
+    Minimum number of observations required to estimate a model when
+    data are missing.  If None, the minimum depends on the number of
+    regressors in the model. Must be smaller than window.
+missing : str
+    Available options are 'drop', 'skip' and 'raise'. If 'drop', any
+    observations with nans are dropped and the estimates are computed using
+    only the non-missing values in each window. If 'skip' blocks containing
+    missing values are skipped and the corresponding results contains NaN.
+    If 'raise', an error is raised. Default is 'drop'."""
+
+
+extra_base = _missing_param_doc
+extra_parameters = window_parameters + weight_parameters + extra_base
+
+_doc = """
+Rolling %(model)s regression
+
+%(parameters)s
+%(extra_parameters)s
+
+See Also
+--------
+statsmodels.regression.linear_model.%(model)s
+
+Notes
+-----
+Tested against %(model)s for accuracy.
+
+Results may differ from %(model)s applied to windows of data if this
+model contains an implicit constant (i.e., includes dummies for all
+categories) rather than an explicit constant (e.g., a column of 1s).
+
+Examples
+--------
+>>> from statsmodels.regression.rolling import Rolling%(model)s
+>>> from statsmodels.datasets import longley
+>>> data = longley.load(as_pandas=False)
+>>> exog = add_constant(data.exog, prepend=False)
+>>> mod = Rolling%(model)s(data.endog, exog)
+>>> rolling_res = mod.fit(reset=50)
+
+Use params_only to skip all calculations except parameter estimation
+
+>>> rolling_params = mod.fit(params_only=True)
+"""
+
+
+@Substitution(model='WLS',
+              parameters=common_params,
+              extra_parameters=extra_parameters)
+@Appender(_doc)
+class RollingWLS(object):
+
+    def __init__(self, endog, exog, window=None, weights=None, min_nobs=None,
+                 missing='drop'):
+        # Call Model.__init__ twice to use const detection in first pass
+        # But to not drop in the second pass
+        missing = string_like(missing, 'missing',
+                              options=('drop', 'raise', 'skip'))
+        temp_msng = 'drop' if missing != 'raise' else 'raise'
+        Model.__init__(self, endog, exog, missing=temp_msng, hasconst=None)
+        k_const = self.k_constant
+        const_idx = self.data.const_idx
+        Model.__init__(self, endog, exog, missing='none', hasconst=False)
+        self.k_constant = k_const
+        self.data.const_idx = const_idx
+        self._y = array_like(endog, 'endog')
+        nobs = self._y.shape[0]
+        self._x = array_like(exog, 'endog', ndim=2, shape=(nobs, None))
+        window = int_like(window, 'window', optional=True)
+        weights = array_like(weights, 'weights', optional=True, shape=(nobs,))
+        self._window = window if window is not None else self._y.shape[0]
+        self._weighted = weights is not None
+        self._weights = np.ones(nobs) if weights is None else weights
+        w12 = np.sqrt(self._weights)
+        self._wy = w12 * self._y
+        self._wx = w12[:, None] * self._x
+        self._is_nan = np.zeros_like(self._y, dtype=np.bool)
+        self._has_nan = self._find_nans()
+        self.const_idx = self.data.const_idx
+        self._skip_missing = missing == 'skip'
+        min_nobs = int_like(min_nobs, 'min_nobs', optional=True)
+        self._min_nobs = min_nobs if min_nobs is not None else self._x.shape[1]
+        if self._min_nobs < self._x.shape[1] or self._min_nobs > self._window:
+            raise ValueError('min_nobs must be larger than the number of '
+                             'regressors in the model and less than window')
+
+    def _handle_data(self, endog, exog, missing, hasconst, **kwargs):
+        return Model._handle_data(self, endog, exog, missing, hasconst,
+                                  **kwargs)
+
+    def _find_nans(self):
+        nans = np.isnan(self._y)
+        nans |= np.any(np.isnan(self._x), axis=1)
+        nans |= np.isnan(self._weights)
+        self._is_nan[:] = nans
+        has_nan = np.cumsum(nans)
+        w = self._window
+        has_nan[w - 1:] = has_nan[w - 1:] - has_nan[:-(w - 1)]
+        has_nan[:w - 1] = False
+
+        return has_nan.astype(np.bool)
+
+    def _get_data(self, idx):
+        window = self._window
+        # TODO: Make a helper function
+        y = self._y[idx - window:idx]
+        wy = self._wy[idx - window:idx]
+        wx = self._wx[idx - window:idx]
+        weights = self._weights[idx - window:idx]
+        missing = self._is_nan[idx - window:idx]
+        not_missing = ~missing
+        if np.any(missing):
+            y = y[not_missing]
+            wy = wy[not_missing]
+            wx = wx[not_missing]
+            weights = weights[not_missing]
+        return y, wy, wx, weights, not_missing
+
+    def _fit_single(self, idx, wxpwx, wxpwy, nobs, store, params_only, method):
+        if nobs < self._min_nobs:
+            return
+        try:
+            wxpwxi = np.linalg.inv(wxpwx)
+            if method == 'inv':
+                params = wxpwxi @ wxpwy
+            else:
+                _, wy, wx, _, _ = self._get_data(idx)
+                if method == 'lstsq':
+                    params = lstsq(wx, wy)[0]
+                else:  # 'pinv'
+                    wxpwxiwxp = np.linalg.pinv(wx)
+                    params = wxpwxiwxp @ wy
+
+        except np.linalg.LinAlgError:
+            return
+        store.params[idx - 1] = params
+        if params_only:
+            return
+        y, wy, wx, weights, _ = self._get_data(idx)
+
+        wresid, ssr, llf = self._loglike(params, wy, wx, weights, nobs)
+        wxwresid = wx * wresid[:, None]
+        wxepwxe = wxwresid.T @ wxwresid
+        tot_params = wx.shape[1]
+        s2 = ssr / (nobs - tot_params)
+
+        centered_tss, uncentered_tss = self._sum_of_squares(y, wy, weights)
+
+        store.ssr[idx - 1] = ssr
+        store.llf[idx - 1] = llf
+        store.nobs[idx - 1] = nobs
+        store.s2[idx - 1] = s2
+        store.xpxi[idx - 1] = wxpwxi
+        store.xeex[idx - 1] = wxepwxe
+        store.centered_tss[idx - 1] = centered_tss
+        store.uncentered_tss[idx - 1] = uncentered_tss
+
+    def _loglike(self, params, wy, wx, weights, nobs):
+        nobs2 = nobs / 2.0
+        wresid = wy - wx @ params
+        ssr = np.sum(wresid ** 2, axis=0)
+        llf = -np.log(ssr) * nobs2  # concentrated likelihood
+        llf -= (1 + np.log(np.pi / nobs2)) * nobs2  # with constant
+        llf += 0.5 * np.sum(np.log(weights))
+        return wresid, ssr, llf
+
+    def _sum_of_squares(self, y, wy, weights):
+        mean = np.average(y, weights=weights)
+        centered_tss = np.sum(weights * (y - mean) ** 2)
+        uncentered_tss = np.dot(wy, wy)
+        return centered_tss, uncentered_tss
+
+    def _reset(self, idx):
+        """Compute xpx and xpy using a single dot product"""
+        _, wy, wx, _, not_missing = self._get_data(idx)
+        nobs = not_missing.sum()
+        xpx = wx.T @ wx
+        xpy = wx.T @ wy
+        return xpx, xpy, nobs
+
+    def fit(self, method='inv', cov_type='nonrobust', cov_kwds=None,
+            reset=None, use_t=False, params_only=False):
+        """
+        Estimate model parameters.
+
+        Parameters
+        ----------
+        method : {'inv', 'lstsq', 'pinv'}
+            Method to use when computing the the model parameters.
+
+            * 'inv' - use moving windows inner-products and matrix inversion.
+               This method is the fastest, but may be less accurate than the
+               other methods.
+            * 'lstsq' - Use numpy.linalg.lstsq
+            * 'pinv' - Use numpy.linalg.pinv. This method matches the default
+               estimator in non-moving regression estimators.
+        cov_type : {'nonrobust', 'HCCM', 'HC0'}
+            Covariance estimator:
+
+            * nonrobust - The classic OLS covariance estimator
+            * HCCM, HC0 - White heteroskedasticity robust covariance
+        cov_kwds : dict
+            Unused
+        reset : int, optional
+            Interval to recompute the moving window inner products used to
+            estimate the model parameters. Smaller values improve accuracy,
+            although in practice this setting is not required to be set.
+        use_t : bool, optional
+            Flag indicating to use the Student's t distribution when computing
+            p-values.
+        params_only : bool, optional
+            Flag indicating that only parameters should be computed. Avoids
+            calculating all other statistics or performing inference.
+
+        Returns
+        -------
+        RollingRegressionResults
+            Estimation results where all pre-sample values are nan-filled.
+        """
+        method = string_like(method, 'method', options=('inv', 'lstsq',
+                                                        'pinv'))
+        reset = int_like(reset, 'reset', optional=True)
+        reset = self._y.shape[0] if reset is None else reset
+        if reset < 1:
+            raise ValueError('reset must be a positive integer')
+
+        window = self._window
+        nobs, k = self._x.shape
+        store = RollingStore(params=np.full((nobs, k), np.nan),
+                             ssr=np.full(nobs, np.nan),
+                             llf=np.full(nobs, np.nan),
+                             nobs=np.zeros(nobs, dtype=np.int),
+                             s2=np.full(nobs, np.nan),
+                             xpxi=np.full((nobs, k, k), np.nan),
+                             xeex=np.full((nobs, k, k), np.nan),
+                             centered_tss=np.full(nobs, np.nan),
+                             uncentered_tss=np.full(nobs, np.nan))
+        xpx, xpy, nobs = self._reset(window)
+        w = self._window
+        if not (self._has_nan[window - 1] and self._skip_missing):
+            self._fit_single(w, xpx, xpy, nobs, store, params_only, method)
+        wx, wy = self._wx, self._wy
+        for i in range(w + 1, self._x.shape[0] + 1):
+            if self._has_nan[i - 1] and self._skip_missing:
+                continue
+            if i % reset == 0:
+                xpx, xpy, nobs = self._reset(i)
+            else:
+                if not self._is_nan[i - w - 1]:
+                    remove_x = wx[i - w - 1:i - w]
+                    xpx -= remove_x.T @ remove_x
+                    xpy -= remove_x.T @ wy[i - w - 1:i - w]
+                    nobs -= 1
+                if not self._is_nan[i - 1]:
+                    add_x = wx[i - 1:i]
+                    xpx += add_x.T @ add_x
+                    xpy += add_x.T @ wy[i - 1:i]
+                    nobs += 1
+
+            self._fit_single(i, xpx, xpy, nobs, store, params_only, method)
+
+        return RollingRegressionResults(self, store, self.k_constant, use_t,
+                                        cov_type)
+
+    @classmethod
+    @Appender(Model.from_formula.__doc__)
+    def from_formula(cls, formula, data, window, weights=None, subset=None,
+                     *args, **kwargs):
+        if subset is not None:
+            data = data.loc[subset]
+        eval_env = kwargs.pop('eval_env', None)
+        if eval_env is None:
+            eval_env = 2
+        elif eval_env == -1:
+            from patsy import EvalEnvironment
+            eval_env = EvalEnvironment({})
+        else:
+            eval_env += 1  # we're going down the stack again
+        missing = kwargs.get('missing', 'skip')
+        from patsy import dmatrices, NAAction
+        na_action = NAAction(on_NA='raise', NA_types=[])
+        result = dmatrices(formula, data, eval_env, return_type='dataframe',
+                           NA_action=na_action)
+
+        endog, exog = result
+        if (endog.ndim > 1 and endog.shape[1] > 1) or endog.ndim > 2:
+            raise ValueError('endog has evaluated to an array with multiple '
+                             'columns that has shape {0}. This occurs when '
+                             'the variable converted to endog is non-numeric'
+                             ' (e.g., bool or str).'.format(endog.shape))
+
+        kwargs.update({'missing': missing,
+                       'window': window})
+        if weights is not None:
+            kwargs['weights'] = weights
+        mod = cls(endog, exog, *args, **kwargs)
+        mod.formula = formula
+        # since we got a dataframe, attach the original
+        mod.data.frame = data
+        return mod
+
+
+extra_parameters = window_parameters + extra_base
+
+
+@Substitution(model='OLS',
+              parameters=common_params,
+              extra_parameters=extra_parameters)
+@Appender(_doc)
+class RollingOLS(RollingWLS):
+    def __init__(self, endog, exog, window=None, min_nobs=None,
+                 missing='drop'):
+        super().__init__(endog, exog, window, None, min_nobs, missing)
+
+
+class RollingRegressionResults(object):
+    """
+    Results from rolling regressions
+
+    Parameters
+    ----------
+    model : RollingWLS
+        Model instance
+    store : RollingStore
+        Container for raw moving window results
+    k_constant : bool
+        Flag indicating that the model contains a constant
+    use_t : bool
+        Flag indicating to use the Student's t distribution when computing
+        p-values.
+    cov_type : str
+        Name of covariance estimator
+    """
+
+    def __init__(self, model, store: RollingStore, k_constant, use_t,
+                 cov_type):
+        self.model = model
+        self._params = store.params
+        self._ssr = store.ssr
+        self._llf = store.llf
+        self._nobs = store.nobs
+        self._s2 = store.s2
+        self._xpxi = store.xpxi
+        self._xepxe = store.xeex
+        self._centered_tss = store.centered_tss
+        self._uncentered_tss = store.uncentered_tss
+        self._k_constant = k_constant
+        self._nvar = self._xpxi.shape[-1]
+        if use_t is None:
+            use_t = cov_type == 'nonrobust'
+        self._use_t = use_t
+        self._cov_type = cov_type
+        self._use_pandas = self.model.data.row_labels is not None
+        self._data_attr = []
+
+    def _wrap(self, val):
+        """Wrap output as pandas Series or DataFrames as needed"""
+        if not self._use_pandas:
+            return val
+        col_names = self.model.data.param_names
+        row_names = self.model.data.row_labels
+        if val.ndim == 1:
+            return Series(val, index=row_names)
+        if val.ndim == 2:
+            return DataFrame(val, columns=col_names, index=row_names)
+        else:  # ndim == 3
+            mi = MultiIndex.from_product((row_names, col_names))
+            val = np.reshape(val, (-1, val.shape[-1]))
+            return DataFrame(val, columns=col_names, index=mi)
+
+    @cache_readonly
+    @Appender(RegressionResults.aic.func.__doc__)
+    def aic(self):
+        return self._wrap(RegressionResults.aic.func(self))
+
+    @cache_readonly
+    @Appender(RegressionResults.bic.func.__doc__)
+    def bic(self):
+        with np.errstate(divide='ignore'):
+            return self._wrap(RegressionResults.bic.func(self))
+
+    @cache_readonly
+    def params(self):
+        """Estimated model parameters"""
+        return self._wrap(self._params)
+
+    @cache_readonly
+    @Appender(RegressionResults.ssr.func.__doc__)
+    def ssr(self):
+        return self._wrap(self._ssr)
+
+    @cache_readonly
+    @Appender(RegressionResults.llf.func.__doc__)
+    def llf(self):
+        return self._wrap(self._llf)
+
+    @cache_readonly
+    @Appender(RegressionModel.df_model.__doc__)
+    def df_model(self):
+        return self._nvar - self._k_constant
+
+    @cache_readonly
+    def k_constant(self):
+        """Flag indicating whether the model contains a constant"""
+        return self._k_constant
+
+    @cache_readonly
+    @Appender(RegressionResults.centered_tss.func.__doc__)
+    def centered_tss(self):
+        return self._centered_tss
+
+    @cache_readonly
+    @Appender(RegressionResults.uncentered_tss.func.__doc__)
+    def uncentered_tss(self):
+        return self._uncentered_tss
+
+    @cache_readonly
+    @Appender(RegressionResults.rsquared.func.__doc__)
+    def rsquared(self):
+        return self._wrap(RegressionResults.rsquared.func(self))
+
+    @cache_readonly
+    @Appender(RegressionResults.rsquared_adj.func.__doc__)
+    def rsquared_adj(self):
+        return self._wrap(RegressionResults.rsquared_adj.func(self))
+
+    @cache_readonly
+    @Appender(RegressionResults.nobs.func.__doc__)
+    def nobs(self):
+        return self._wrap(self._nobs)
+
+    @cache_readonly
+    @Appender(RegressionModel.df_resid.__doc__)
+    def df_resid(self):
+        return self._wrap(self._nobs - self.df_model - self._k_constant)
+
+    @cache_readonly
+    @Appender(RegressionResults.use_t.__doc__)
+    def use_t(self):
+        return self._use_t
+
+    @cache_readonly
+    @Appender(RegressionResults.ess.func.__doc__)
+    def ess(self):
+        return self._wrap(RegressionResults.ess.func(self))
+
+    @cache_readonly
+    @Appender(RegressionResults.mse_model.func.__doc__)
+    def mse_model(self):
+        return self._wrap(RegressionResults.mse_model.func(self))
+
+    @cache_readonly
+    @Appender(RegressionResults.mse_resid.func.__doc__)
+    def mse_resid(self):
+        return self._wrap(RegressionResults.mse_resid.func(self))
+
+    @cache_readonly
+    @Appender(RegressionResults.mse_total.func.__doc__)
+    def mse_total(self):
+        return self._wrap(RegressionResults.mse_total.func(self))
+
+    @cache_readonly
+    def _cov_params(self):
+        if self._cov_type == 'nonrobust':
+            return self._s2[:, None, None] * self._xpxi
+        else:
+            return self._xpxi @ self._xepxe @ self._xpxi
+
+    def cov_params(self):
+        """
+        Estimated parameter covariance
+
+        Returns
+        -------
+        array_like
+            The estimated model covariances. If the original input is a numpy
+            array, the returned covariance is a 3-d array with shape
+            (nobs, nvar, nvar). If the original inputs are pandas types, then
+            the returned covariance is a DataFrame with a MultiIndex with
+            key (observation, variable), so that the covariance for
+            observation with index i is cov.loc[i].
+         """
+        return self._wrap(self._cov_params)
+
+    @cache_readonly
+    @Appender(RegressionResults.f_pvalue.func.__doc__)
+    def f_pvalue(self):
+        with np.errstate(invalid='ignore'):
+            return self._wrap(RegressionResults.f_pvalue.func(self))
+
+    @cache_readonly
+    @Appender(RegressionResults.fvalue.func.__doc__)
+    def fvalue(self):
+        if self._cov_type == 'nonrobust':
+            return self.mse_model / self.mse_resid
+        else:
+            nobs = self._params.shape[0]
+            stat = np.full(nobs, np.nan)
+            k = self._params.shape[1]
+            r = np.eye(k)
+            locs = list(range(k))
+            if self.k_constant:
+                locs.pop(self.model.const_idx)
+            if not locs:
+                return stat
+            r = r[locs]
+            vcv = self._cov_params
+            rvcvr = r @ vcv @ r.T
+            p = self._params
+            for i in range(nobs):
+                rp = p[i:i + 1] @ r.T
+                denom = rp.shape[1]
+                inv_cov = np.linalg.inv(rvcvr[i])
+                stat[i] = (rp @ inv_cov @ rp.T) / denom
+            return stat
+
+    @cache_readonly
+    @Appender(RegressionResults.bse.func.__doc__)
+    def bse(self):
+        with np.errstate(invalid='ignore'):
+            return self._wrap(np.sqrt(np.diagonal(self._cov_params, 0, 2)))
+
+    @cache_readonly
+    @Appender(LikelihoodModelResults.tvalues.func.__doc__)
+    def tvalues(self):
+        with np.errstate(invalid='ignore'):
+            return self._wrap(LikelihoodModelResults.tvalues.func(self))
+
+    @cache_readonly
+    @Appender(LikelihoodModelResults.pvalues.func.__doc__)
+    def pvalues(self):
+        """The two-tailed p values for the t-stats of the params."""
+        if self.use_t:
+            df_resid = getattr(self, 'df_resid_inference', self.df_resid)
+            df_resid = np.asarray(df_resid)[:, None]
+            with np.errstate(invalid='ignore'):
+                return stats.t.sf(np.abs(self.tvalues), df_resid) * 2
+        else:
+            with np.errstate(invalid='ignore'):
+                return stats.norm.sf(np.abs(self.tvalues)) * 2
+
+    def _conf_int(self, alpha, cols):
+        bse = np.asarray(self.bse)
+
+        if self.use_t:
+            dist = stats.t
+            df_resid = getattr(self, 'df_resid_inference', self.df_resid)
+            df_resid = np.asarray(df_resid)[:, None]
+            q = dist.ppf(1 - alpha / 2, df_resid)
+        else:
+            dist = stats.norm
+            q = dist.ppf(1 - alpha / 2)
+
+        params = np.asarray(self.params)
+        lower = params - q * bse
+        upper = params + q * bse
+        if cols is not None:
+            cols = np.asarray(cols)
+            lower = lower[:, cols]
+            upper = upper[:, cols]
+        return np.asarray(list(zip(lower, upper)))
+
+    @Appender(LikelihoodModelResults.conf_int.__doc__)
+    def conf_int(self, alpha=.05, cols=None):
+        ci = self._conf_int(alpha, cols)
+        if not self._use_pandas:
+            return ci
+        ci_names = ('lower', 'upper')
+        row_names = self.model.data.row_labels
+        col_names = self.model.data.param_names
+        if cols is not None:
+            col_names = [col_names[i] for i in cols]
+        mi = MultiIndex.from_product((col_names, ci_names))
+        ci = np.reshape(np.swapaxes(ci, 1, 2), (ci.shape[0], -1))
+        return DataFrame(ci, columns=mi, index=row_names)
+
+    @property
+    def cov_type(self):
+        """Name of covariance estimator"""
+        return self._cov_type
+
+    @classmethod
+    @Appender(LikelihoodModelResults.load.__doc__)
+    def load(cls, fname):
+        return LikelihoodModelResults.load(fname)
+
+    remove_data = LikelihoodModelResults.remove_data
+
+    @Appender(LikelihoodModelResults.save.__doc__)
+    def save(self, fname, remove_data=False):
+        return LikelihoodModelResults.save(self, fname, remove_data)
+
+    def plot_recursive_coefficient(self, variables=None, alpha=0.05,
+                                   legend_loc='upper left', fig=None,
+                                   figsize=None):
+        r"""
+        Plot the recursively estimated coefficients on a given variable
+
+        Parameters
+        ----------
+        variables : {int, str, Iterable[int], Iterable[str], None}, optional
+            Integer index or string name of the variables whose coefficients
+            to plot. Can also be an iterable of integers or strings. Default
+            plots all coefficients.
+        alpha : float, optional
+            The confidence intervals for the coefficient are (1 - alpha)%. Set
+            to None to exclude confidence intervals.
+        legend_loc : string, optional
+            The location of the legend in the plot. Default is upper left.
+        fig : Matplotlib Figure instance, optional
+            If given, subplots are created in this figure instead of in a new
+            figure. Note that the grid will be created in the provided
+            figure using `fig.add_subplot()`.
+        figsize : tuple, optional
+            If a figure is created, this argument allows specifying a size.
+            The tuple is (width, height).
+
+        Returns
+        -------
+        Figure
+            The matplotlib Figure object.
+        """
+        from statsmodels.graphics.utils import _import_mpl, create_mpl_fig
+
+        if alpha is not None:
+            ci = self._conf_int(alpha, None)
+
+        row_labels = self.model.data.row_labels
+        if row_labels is None:
+            row_labels = np.arange(self._params.shape[0])
+        k_variables = self._params.shape[1]
+        param_names = self.model.data.param_names
+        if variables is None:
+            variable_idx = list(range(k_variables))
+        else:
+            if isinstance(variables, (int, str)):
+                variables = [variables]
+            variable_idx = []
+            for i in range(len(variables)):
+                variable = variables[i]
+                if variable in param_names:
+                    variable_idx.append(param_names.index(variable))
+                elif isinstance(variable, int):
+                    variable_idx.append(variable)
+                else:
+                    msg = ('variable {0} is not an integer and was not found '
+                           'in the list of variable '
+                           'names: {1}'.format(variables[i],
+                                               ', '.join(param_names)))
+                    raise ValueError(msg)
+
+        _import_mpl()
+        fig = create_mpl_fig(fig, figsize)
+
+        loc = 0
+        import pandas as pd
+        if isinstance(row_labels, pd.PeriodIndex):
+            row_labels = row_labels.to_timestamp()
+        row_labels = np.asarray(row_labels)
+        for i in variable_idx:
+            ax = fig.add_subplot(len(variable_idx), 1, loc + 1)
+            params = self._params[:, i]
+            valid = ~np.isnan(self._params[:, i])
+            row_lbl = row_labels[valid]
+            ax.plot(row_lbl, params[valid])
+            if alpha is not None:
+                this_ci = np.reshape(ci[:, :, i], (-1, 2))
+                if not np.all(np.isnan(this_ci)):
+                    ax.plot(row_lbl, this_ci[:, 0][valid], 'k:',
+                            label='Lower CI')
+                    ax.plot(row_lbl, this_ci[:, 1][valid], 'k:',
+                            label='Upper CI')
+                    if loc == 0:
+                        ax.legend(loc=legend_loc)
+            ax.set_xlim(row_lbl[0], row_lbl[-1])
+            ax.set_title(param_names[i])
+            loc += 1
+
+        fig.tight_layout()
+        return fig


Test patch:
diff --git a/statsmodels/regression/tests/test_rolling.py b/statsmodels/regression/tests/test_rolling.py
new file mode 100644
--- /dev/null
+++ b/statsmodels/regression/tests/test_rolling.py
@@ -0,0 +1,284 @@
+from io import BytesIO
+from itertools import product
+
+import numpy as np
+import pandas as pd
+import pytest
+from numpy.testing import assert_allclose, assert_array_equal
+
+from statsmodels import tools
+from statsmodels.regression.linear_model import WLS
+from statsmodels.regression.rolling import RollingWLS, RollingOLS
+
+
+def gen_data(nobs, nvar, const, pandas=False, missing=0.0,
+             weights=False):
+    rs = np.random.RandomState(987499302)
+    x = rs.standard_normal((nobs, nvar))
+    cols = ['x{0}'.format(i) for i in range(nvar)]
+    if const:
+        x = tools.add_constant(x)
+        cols = ['const'] + cols
+    if missing > 0.0:
+        mask = rs.random_sample(x.shape) < missing
+        x[mask] = np.nan
+    if x.shape[1] > 1:
+        y = x[:, :-1].sum(1) + rs.standard_normal(nobs)
+    else:
+        y = x.sum(1) + rs.standard_normal(nobs)
+    w = rs.chisquare(5, y.shape[0]) / 5
+    if pandas:
+        idx = pd.date_range('12-31-1999', periods=nobs)
+        x = pd.DataFrame(x, index=idx, columns=cols)
+        y = pd.Series(y, index=idx, name='y')
+        w = pd.Series(w, index=idx, name='weights')
+    if not weights:
+        w = None
+
+    return y, x, w
+
+
+nobs = (250,)
+nvar = (3, 0)
+tf = (True, False)
+missing = (0, 0.1)
+params = list(product(nobs, nvar, tf, tf, missing))
+params = [param for param in params if param[1] + param[2] > 0]
+ids = ['-'.join(map(str, param)) for param in params]
+
+basic_params = [param for param in params if params[2] and params[4]]
+weighted_params = [param + (tf,) for param in params for tf in (True, False)]
+weighted_ids = ['-'.join(map(str, param)) for param in weighted_params]
+
+
+@pytest.fixture(scope='module', params=params, ids=ids)
+def data(request):
+    return gen_data(*request.param)
+
+
+@pytest.fixture(scope='module', params=basic_params, ids=ids)
+def basic_data(request):
+    return gen_data(*request.param)
+
+
+@pytest.fixture(scope='module', params=weighted_params, ids=weighted_ids)
+def weighted_data(request):
+    return gen_data(*request.param)
+
+
+def get_single(x, idx):
+    if isinstance(x, (pd.Series, pd.DataFrame)):
+        return x.iloc[idx]
+    return x[idx]
+
+
+def get_sub(x, idx, window):
+    if isinstance(x, (pd.Series, pd.DataFrame)):
+        out = x.iloc[idx - window:idx]
+        return np.asarray(out)
+    return x[idx - window:idx]
+
+
+def test_has_nan(data):
+    y, x, w = data
+    mod = RollingWLS(y, x, window=100, weights=w)
+    has_nan = np.zeros(y.shape[0], dtype=np.bool)
+    for i in range(100, y.shape[0] + 1):
+        _y = get_sub(y, i, 100)
+        _x = get_sub(x, i, 100)
+        has_nan[i - 1] = np.squeeze((np.any(np.isnan(_y)) or
+                                     np.any(np.isnan(_x))))
+    assert_array_equal(mod._has_nan, has_nan)
+
+
+def test_weighted_against_wls(weighted_data):
+    y, x, w = weighted_data
+    mod = RollingWLS(y, x, weights=w, window=100)
+    res = mod.fit(use_t=True)
+    for i in range(100, y.shape[0]):
+        _y = get_sub(y, i, 100)
+        _x = get_sub(x, i, 100)
+        if w is not None:
+            _w = get_sub(w, i, 100)
+        else:
+            _w = np.ones_like(_y)
+        wls = WLS(_y, _x, weights=_w, missing='drop').fit()
+        rolling_params = get_single(res.params, i - 1)
+        rolling_nobs = get_single(res.nobs, i - 1)
+        assert_allclose(rolling_params, wls.params)
+        assert_allclose(rolling_nobs, wls.nobs)
+        assert_allclose(get_single(res.ssr, i - 1), wls.ssr)
+        assert_allclose(get_single(res.llf, i - 1), wls.llf)
+        assert_allclose(get_single(res.aic, i - 1), wls.aic)
+        assert_allclose(get_single(res.bic, i - 1), wls.bic)
+        assert_allclose(get_single(res.centered_tss, i - 1), wls.centered_tss)
+        assert_allclose(res.df_model, wls.df_model)
+        assert_allclose(get_single(res.df_resid, i - 1), wls.df_resid)
+        assert_allclose(get_single(res.ess, i - 1), wls.ess, atol=1e-8)
+        assert_allclose(res.k_constant, wls.k_constant)
+        assert_allclose(get_single(res.mse_model, i - 1), wls.mse_model)
+        assert_allclose(get_single(res.mse_resid, i - 1), wls.mse_resid)
+        assert_allclose(get_single(res.mse_total, i - 1), wls.mse_total)
+        assert_allclose(get_single(res.rsquared, i - 1), wls.rsquared,
+                        atol=1e-8)
+        assert_allclose(get_single(res.rsquared_adj, i - 1), wls.rsquared_adj,
+                        atol=1e-8)
+        assert_allclose(get_single(res.uncentered_tss, i - 1),
+                        wls.uncentered_tss)
+
+
+@pytest.mark.parametrize('cov_type', ['nonrobust', 'HC0'])
+@pytest.mark.parametrize('use_t', [None, True, False])
+def test_against_wls_inference(data, use_t, cov_type):
+    y, x, w = data
+    mod = RollingWLS(y, x, window=100, weights=w)
+    res = mod.fit(use_t=use_t, cov_type=cov_type)
+    ci_cols = ci = res.conf_int()
+    test_cols = x.shape[1] > 3
+
+    # This is a smoke test of cov_params to make sure it works
+    res.cov_params()
+
+    if test_cols:
+        ci_cols = res.conf_int(cols=[0, 2])
+    # Skip to improve performance
+    for i in range(100, y.shape[0]):
+        _y = get_sub(y, i, 100)
+        _x = get_sub(x, i, 100)
+        wls = WLS(_y, _x, missing='drop').fit(use_t=use_t, cov_type=cov_type)
+        assert_allclose(get_single(res.tvalues, i - 1), wls.tvalues)
+        assert_allclose(get_single(res.bse, i - 1), wls.bse)
+        assert_allclose(get_single(res.pvalues, i - 1), wls.pvalues, atol=1e-8)
+        assert_allclose(get_single(res.fvalue, i - 1), wls.fvalue)
+        with np.errstate(invalid='ignore'):
+            assert_allclose(get_single(res.f_pvalue, i - 1), wls.f_pvalue,
+                            atol=1e-8)
+        assert res.cov_type == wls.cov_type
+        assert res.use_t == wls.use_t
+        wls_ci = wls.conf_int()
+        if isinstance(ci, pd.DataFrame):
+            ci_val = ci.iloc[i - 1]
+            ci_val = np.asarray(ci_val).reshape((-1, 2))
+        else:
+            ci_val = ci[i - 1].T
+        assert_allclose(ci_val, wls_ci)
+        if test_cols:
+            wls_ci = wls.conf_int(cols=[0, 2])
+            if isinstance(ci_cols, pd.DataFrame):
+                ci_val = ci_cols.iloc[i - 1]
+                ci_val = np.asarray(ci_val).reshape((-1, 2))
+            else:
+                ci_val = ci_cols[i - 1].T
+            assert_allclose(ci_val, wls_ci)
+
+
+def test_raise(data):
+    y, x, w = data
+
+    mod = RollingWLS(y, x, window=100, missing='drop', weights=w)
+    res = mod.fit()
+    params = np.asarray(res.params)
+    assert np.all(np.isfinite(params[99:]))
+
+    if not np.any(np.isnan(y)):
+        return
+    mod = RollingWLS(y, x, window=100, missing='skip')
+    res = mod.fit()
+    params = np.asarray(res.params)
+    assert np.any(np.isnan(params[100:]))
+
+
+def test_error():
+    y, x, _ = gen_data(250, 2, True)
+    with pytest.raises(ValueError, match='reset must be a positive integer'):
+        RollingWLS(y, x, ).fit(reset=-1)
+    with pytest.raises(ValueError):
+        RollingWLS(y, x).fit(method='unknown')
+    with pytest.raises(ValueError, match='min_nobs must be larger'):
+        RollingWLS(y, x, min_nobs=1)
+    with pytest.raises(ValueError, match='min_nobs must be larger'):
+        RollingWLS(y, x, window=60, min_nobs=100)
+
+
+def test_save_load(data):
+    y, x, w = data
+    res = RollingOLS(y, x, window=60).fit()
+    fh = BytesIO()
+    # test wrapped results load save pickle
+    res.save(fh)
+    fh.seek(0, 0)
+    res_unpickled = res.__class__.load(fh)
+    assert type(res_unpickled) is type(res)  # noqa: E721
+
+    fh = BytesIO()
+    # test wrapped results load save pickle
+    res.save(fh, remove_data=True)
+    fh.seek(0, 0)
+    res_unpickled = res.__class__.load(fh)
+    assert type(res_unpickled) is type(res)  # noqa: E721
+
+
+def test_formula():
+    y, x, w = gen_data(250, 3, True, pandas=True)
+    fmla = 'y ~ 1 + x0 + x1 + x2'
+    data = pd.concat([y, x], axis=1)
+    mod = RollingWLS.from_formula(fmla, window=100, data=data, weights=w)
+    res = mod.fit()
+    alt = RollingWLS(y, x, window=100)
+    alt_res = alt.fit()
+    assert_allclose(res.params, alt_res.params)
+    ols_mod = RollingOLS.from_formula(fmla, window=100, data=data)
+    ols_mod.fit()
+
+
+@pytest.mark.matplotlib
+def test_plot():
+    import matplotlib.pyplot as plt
+
+    y, x, w = gen_data(250, 3, True, pandas=True)
+    fmla = 'y ~ 1 + x0 + x1 + x2'
+    data = pd.concat([y, x], axis=1)
+    mod = RollingWLS.from_formula(fmla, window=100, data=data, weights=w)
+    res = mod.fit()
+    fig = res.plot_recursive_coefficient()
+    assert isinstance(fig, plt.Figure)
+    res.plot_recursive_coefficient(variables=2, alpha=None,
+                                   figsize=(30, 7))
+    res.plot_recursive_coefficient(variables='x0', alpha=None,
+                                   figsize=(30, 7))
+    res.plot_recursive_coefficient(variables=[0, 2], alpha=None,
+                                   figsize=(30, 7))
+    res.plot_recursive_coefficient(variables=['x0'], alpha=None,
+                                   figsize=(30, 7))
+    res.plot_recursive_coefficient(variables=['x0', 'x1', 'x2'], alpha=None,
+                                   figsize=(30, 7))
+    with pytest.raises(ValueError, match='variable x4 is not an integer'):
+        res.plot_recursive_coefficient(variables='x4')
+
+    fig = plt.Figure()
+    out = res.plot_recursive_coefficient(fig=fig)
+    assert out is fig
+    res.plot_recursive_coefficient(alpha=None, figsize=(30, 7))
+
+
+def test_methods(basic_data):
+    y, x, _ = basic_data
+    mod = RollingOLS(y, x, 150)
+    res_inv = mod.fit(method='inv')
+    res_lstsq = mod.fit(method='lstsq')
+    res_pinv = mod.fit(method='pinv')
+    assert_allclose(res_inv.params, res_lstsq.params)
+    assert_allclose(res_inv.params, res_pinv.params)
+
+
+def test_min_nobs(basic_data):
+    y, x, w = basic_data
+    if not np.any(np.isnan(np.asarray(x))):
+        return
+    mod = RollingOLS(y, x, 150)
+    res = mod.fit()
+    # Ensures that the constraint binds
+    min_nobs = res.nobs[res.nobs != 0].min() + 1
+    mod = RollingOLS(y, x, 150, min_nobs=min_nobs)
+    res = mod.fit()
+    assert np.all(res.nobs[res.nobs != 0] >= min_nobs)