Repository: statsmodels/statsmodels

Issue:
groups specification under cov_type='hac-panel' on unbalanced panel data
A standard call to do do this on a panel data would be:

```
import statsmodels.formula.api as smf
smf.ols(fmla,data=regr_frame).fit(cov_type='nw-panel',cov_kwds={'maxlags':5,'time':regr_frame['date']})
```

I noticed that `get_robustcov_results` does 

```
tt = (np.nonzero(np.diff(time) < 0)[0] + 1).tolist()
groupidx = lzip([0] + tt, tt + [len(time)])
```

to get the groups. However, this might not work if the panel data has non overlapping time stamps for two groups. For example, group A and B are adjacent. Group A has periods 1927 to 1945, group B has periods 1947 to 1967. Then the above code snippet would say these two are one group, while actually they are two groups.

Such input panel data usually results from dropping NAs in a stacked time series. But if I keep the NAs, `get_robustcov_results` gives me alignment error. 

```
    731     for lag in range(1, nlags+1):
    732         xw0, xwlag = lagged_groups(xw, lag, groupidx)
--> 733         s = np.dot(xw0.T, xwlag)
    734         S += weights[lag] * (s + s.T)
    735     return S

ValueError: matrices are not aligned
```

Is there (Could there be) some additional cov_kwds (such as 'groups') to specify the groups manually? Or, is there nice way to avoid the alignment error for unbalance panel?



Discussion:
can you try hac-groupsum  ? (guess not, see below)

It will take me a bit of time to remember (remind myself) how the aggregation is supposed to go in this.

For the specific issue:
Right now I don't see a reason not to add a optional `group` indes. as an explicit identification of groups instead of implicitly inferring it from the time array

(hac-groupsum assumes overlapping time index IIRC)

I'm a bit puzzled.

What's the implied correlation structure in your data? Do you have a full example.

AFAIR, the standard usage for panel data sandwiches is to have the HAC part taking care of serial correlation, and the `group` part taking case of the cross-sectional correlation and heteroscedasticity. 
But maybe I don't remember this, and panel-hac is just serial correlation within a panel. In that case, adding the optional group index for identification would be an obvious explicit choice without changing the interpretation.
(I might be mixing up the assumptions of hac-panel versus hac-groupsum)

To the last question:

> Or, is there nice way to avoid the alignment error for unbalance panel?

At this stage there is no nan/missing handling nor is the time index checked for irregular or missing time periods. So, the assumption is no nans and for hac autocorrelation no gaps in the time series.
Balanced panel, i.e. equal number of time periods for each group, is not be assumed.

Thank you Josef. So 'nw-panel' assumes equally spaced time periods within each group, no nans, but does not assume the panel is balanced. And it essentially extracts the _time bounds_ from the `groupidx` and within each group, lagged terms are added to the overall S. (Cov = (x'x)^(-1) S (x'x)^(-1))
That makes sense.
However the edge case mentioned above satisfies the assumptions but the produced covariance is not correct, since the group bounds are wrong. 

I guess right now the users can find the group bounds themselves and use `cov_nw_panel(results, nlags, groupidx, ...)` function to get a correct covariance in this case. But would you consider adding the optional `group` kwd in future releases? Thanks.

(I added backport label to remind me again, since it already slipped my mind)

A workaround would be to use group labels as time, but they would have to be sorted by decreasing number.
Yes, I'm in favor of adding the option, especially given that the original design only require group indices (indices of break points).  I switched to time index mainly for API consistency, and because most users will have it already available as part of their data(frame).
I think it's just changing or adding two lines in each of the methods/functions, (plus 10 to 20 lines of unit tests). I will try to look into it in the coming week

Thank you. The workaround works perfectly.


Resolving patch:
diff --git a/statsmodels/base/covtype.py b/statsmodels/base/covtype.py
--- a/statsmodels/base/covtype.py
+++ b/statsmodels/base/covtype.py
@@ -99,11 +99,15 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
     - 'hac-panel' heteroscedasticity and autocorrelation robust standard
         errors in panel data.
         The data needs to be sorted in this case, the time series for
-        each panel unit or cluster need to be stacked.
-        keywords
+        each panel unit or cluster need to be stacked. The membership to
+        a timeseries of an individual or group can be either specified by
+        group indicators or by increasing time periods.
 
-        - `time` array_like (required) : index of time periods
+        keywords
 
+        - either `groups` or `time` : array_like (required)
+          `groups` : indicator for groups
+          `time` : index of time periods
         - `maxlag` integer (required) : number of lags to use
         - `kernel` string (optional) : kernel, default is Bartlett
         - `use_correction` False or string in ['hac', 'cluster'] (optional) :
@@ -163,7 +167,7 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
     # TODO: this should be outsourced in a function so we can reuse it in
     #       other models
     # TODO: make it DRYer   repeated code for checking kwds
-    if cov_type in ('HC0', 'HC1', 'HC2', 'HC3'):
+    if cov_type.upper() in ('HC0', 'HC1', 'HC2', 'HC3'):
         if kwds:
             raise ValueError('heteroscedasticity robust covarians ' +
                              'does not use keywords')
@@ -175,7 +179,7 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
             # results classes that don't have cov_HCx attribute
             res.cov_params_default = sw.cov_white_simple(self,
                                                          use_correction=False)
-    elif cov_type == 'HAC':
+    elif cov_type.lower() == 'hac':
         maxlags = kwds['maxlags']   # required?, default in cov_hac_simple
         res.cov_kwds['maxlags'] = maxlags
         weights_func = kwds.get('weights_func', sw.weights_bartlett)
@@ -189,7 +193,7 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
         res.cov_params_default = sw.cov_hac_simple(self, nlags=maxlags,
                                              weights_func=weights_func,
                                              use_correction=use_correction)
-    elif cov_type == 'cluster':
+    elif cov_type.lower() == 'cluster':
         #cluster robust standard errors, one- or two-way
         groups = kwds['groups']
         if not hasattr(groups, 'shape'):
@@ -229,9 +233,10 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
         res.cov_kwds['description'] = ('Standard Errors are robust to' +
                             'cluster correlation ' + '(' + cov_type + ')')
 
-    elif cov_type == 'hac-panel':
+    elif cov_type.lower() == 'hac-panel':
         #cluster robust standard errors
-        res.cov_kwds['time'] = time = kwds['time']
+        res.cov_kwds['time'] = time = kwds.get('time', None)
+        res.cov_kwds['groups'] = groups = kwds.get('groups', None)
         #TODO: nlags is currently required
         #nlags = kwds.get('nlags', True)
         #res.cov_kwds['nlags'] = nlags
@@ -242,15 +247,23 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
         weights_func = kwds.get('weights_func', sw.weights_bartlett)
         res.cov_kwds['weights_func'] = weights_func
         # TODO: clumsy time index in cov_nw_panel
-        tt = (np.nonzero(np.diff(time) < 0)[0] + 1).tolist()
-        groupidx = lzip([0] + tt, tt + [len(time)])
+        if groups is not None:
+            tt = (np.nonzero(groups[:-1] != groups[1:])[0] + 1).tolist()
+            nobs_ = len(groups)
+        elif time is not None:
+            # TODO: clumsy time index in cov_nw_panel
+            tt = (np.nonzero(time[1:] < time[:-1])[0] + 1).tolist()
+            nobs_ = len(time)
+        else:
+            raise ValueError('either time or groups needs to be given')
+        groupidx = lzip([0] + tt, tt + [nobs_])
         self.n_groups = n_groups = len(groupidx)
         res.cov_params_default = sw.cov_nw_panel(self, maxlags, groupidx,
                                             weights_func=weights_func,
                                             use_correction=use_correction)
         res.cov_kwds['description'] = ('Standard Errors are robust to' +
                             'cluster correlation ' + '(' + cov_type + ')')
-    elif cov_type == 'hac-groupsum':
+    elif cov_type.lower() == 'hac-groupsum':
         # Driscoll-Kraay standard errors
         res.cov_kwds['time'] = time = kwds['time']
         #TODO: nlags is currently required
@@ -264,7 +277,7 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
         res.cov_kwds['weights_func'] = weights_func
         if adjust_df:
             # need to find number of groups
-            tt = (np.nonzero(np.diff(time) < 0)[0] + 1)
+            tt = (np.nonzero(time[1:] < time[:-1])[0] + 1)
             self.n_groups = n_groups = len(tt) + 1
         res.cov_params_default = sw.cov_nw_groupsum(self, maxlags, time,
                                         weights_func=weights_func,
diff --git a/statsmodels/regression/linear_model.py b/statsmodels/regression/linear_model.py
--- a/statsmodels/regression/linear_model.py
+++ b/statsmodels/regression/linear_model.py
@@ -1900,11 +1900,15 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
         - 'hac-panel' heteroscedasticity and autocorrelation robust standard
             errors in panel data.
             The data needs to be sorted in this case, the time series for
-            each panel unit or cluster need to be stacked.
+            each panel unit or cluster need to be stacked. The membership to
+            a timeseries of an individual or group can be either specified by 
+            group indicators or by increasing time periods.
+            
             keywords
 
-            - `time` array_like (required) : index of time periods
-
+            - either `groups` or `time` : array_like (required)
+              `groups` : indicator for groups
+              `time` : index of time periods
             - `maxlag` integer (required) : number of lags to use
             - `kernel` string (optional) : kernel, default is Bartlett
             - `use_correction` False or string in ['hac', 'cluster'] (optional) :
@@ -1969,7 +1973,7 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
 
             res.cov_kwds['scale'] = scale = kwds.get('scale', 1.)
             res.cov_params_default = scale * res.normalized_cov_params
-        elif cov_type in ('HC0', 'HC1', 'HC2', 'HC3'):
+        elif cov_type.upper() in ('HC0', 'HC1', 'HC2', 'HC3'):
             if kwds:
                 raise ValueError('heteroscedasticity robust covarians ' +
                                  'does not use keywords')
@@ -1978,7 +1982,7 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
             # TODO cannot access cov without calling se first
             getattr(self, cov_type.upper() + '_se')
             res.cov_params_default = getattr(self, 'cov_' + cov_type.upper())
-        elif cov_type == 'HAC':
+        elif cov_type.lower() == 'hac':
             maxlags = kwds['maxlags']   # required?, default in cov_hac_simple
             res.cov_kwds['maxlags'] = maxlags
             weights_func = kwds.get('weights_func', sw.weights_bartlett)
@@ -1992,7 +1996,7 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
             res.cov_params_default = sw.cov_hac_simple(self, nlags=maxlags,
                                                  weights_func=weights_func,
                                                  use_correction=use_correction)
-        elif cov_type == 'cluster':
+        elif cov_type.lower() == 'cluster':
             #cluster robust standard errors, one- or two-way
             groups = kwds['groups']
             if not hasattr(groups, 'shape'):
@@ -2032,9 +2036,10 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
             res.cov_kwds['description'] = ('Standard Errors are robust to' +
                                 'cluster correlation ' + '(' + cov_type + ')')
 
-        elif cov_type == 'hac-panel':
+        elif cov_type.lower() == 'hac-panel':
             #cluster robust standard errors
-            res.cov_kwds['time'] = time = kwds['time']
+            res.cov_kwds['time'] = time = kwds.get('time', None)
+            res.cov_kwds['groups'] = groups = kwds.get('groups', None)
             #TODO: nlags is currently required
             #nlags = kwds.get('nlags', True)
             #res.cov_kwds['nlags'] = nlags
@@ -2044,16 +2049,23 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
             res.cov_kwds['use_correction'] = use_correction
             weights_func = kwds.get('weights_func', sw.weights_bartlett)
             res.cov_kwds['weights_func'] = weights_func
-            # TODO: clumsy time index in cov_nw_panel
-            tt = (np.nonzero(np.diff(time) < 0)[0] + 1).tolist()
-            groupidx = lzip([0] + tt, tt + [len(time)])
+            if groups is not None:
+                tt = (np.nonzero(groups[:-1] != groups[1:])[0] + 1).tolist()
+                nobs_ = len(groups)
+            elif time is not None:
+                # TODO: clumsy time index in cov_nw_panel
+                tt = (np.nonzero(time[1:] < time[:-1])[0] + 1).tolist()
+                nobs_ = len(time)
+            else:
+                raise ValueError('either time or groups needs to be given')
+            groupidx = lzip([0] + tt, tt + [nobs_])
             self.n_groups = n_groups = len(groupidx)
             res.cov_params_default = sw.cov_nw_panel(self, maxlags, groupidx,
                                                 weights_func=weights_func,
                                                 use_correction=use_correction)
             res.cov_kwds['description'] = ('Standard Errors are robust to' +
                                 'cluster correlation ' + '(' + cov_type + ')')
-        elif cov_type == 'hac-groupsum':
+        elif cov_type.lower() == 'hac-groupsum':
             # Driscoll-Kraay standard errors
             res.cov_kwds['time'] = time = kwds['time']
             #TODO: nlags is currently required
@@ -2067,7 +2079,7 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwds):
             res.cov_kwds['weights_func'] = weights_func
             if adjust_df:
                 # need to find number of groups
-                tt = (np.nonzero(np.diff(time) < 0)[0] + 1)
+                tt = (np.nonzero(time[1:] < time[:-1])[0] + 1)
                 self.n_groups = n_groups = len(tt) + 1
             res.cov_params_default = sw.cov_nw_groupsum(self, maxlags, time,
                                             weights_func=weights_func,


Test patch:
diff --git a/statsmodels/discrete/tests/test_sandwich_cov.py b/statsmodels/discrete/tests/test_sandwich_cov.py
--- a/statsmodels/discrete/tests/test_sandwich_cov.py
+++ b/statsmodels/discrete/tests/test_sandwich_cov.py
@@ -619,7 +619,6 @@ class TestGLMGaussHACPanel(CheckDiscreteGLM):
 
     @classmethod
     def setup_class(cls):
-        import statsmodels.stats.sandwich_covariance as sw
         cls.cov_type = 'hac-panel'
         # time index is just made up to have a test case
         time = np.tile(np.arange(7), 5)[:-1]
@@ -640,6 +639,25 @@ def test_kwd(self):
         assert_allclose(self.res1b.bse, self.res1.bse, rtol=1e-12)
 
 
+class TestGLMGaussHACPanelGroups(CheckDiscreteGLM):
+
+    @classmethod
+    def setup_class(cls):
+        cls.cov_type = 'hac-panel'
+        # time index is just made up to have a test case
+        groups = np.repeat(np.arange(5), 7)[:-1]
+        mod1 = GLM(endog.copy(), exog.copy(), family=families.Gaussian())
+        kwds = dict(groups=groups,
+                    maxlags=2,
+                    kernel=sw.weights_uniform,
+                    use_correction='hac',
+                    df_correction=False)
+        cls.res1 = mod1.fit(cov_type='hac-panel', cov_kwds=kwds)
+
+        mod2 = OLS(endog, exog)
+        cls.res2 = mod2.fit(cov_type='hac-panel', cov_kwds=kwds)
+
+
 class TestGLMGaussHACGroupsum(CheckDiscreteGLM):
 
     @classmethod
diff --git a/statsmodels/regression/tests/test_robustcov.py b/statsmodels/regression/tests/test_robustcov.py
--- a/statsmodels/regression/tests/test_robustcov.py
+++ b/statsmodels/regression/tests/test_robustcov.py
@@ -605,6 +605,32 @@ def test_keyword(self):
         assert_allclose(res_ols.bse, self.res1.bse, rtol=1e-12)
 
 
+class TestOLSRobustClusterNWPGroupsFit(CheckOLSRobustCluster, CheckOLSRobustNewMixin):
+    # compare with `reg cluster`
+
+    def setup(self):
+        res_ols = self.res1.model.fit(cov_type='nw-panel',
+                                      cov_kwds = dict(groups=self.groups,
+                                                      maxlags=4,
+                                                      use_correction='hac',
+                                                      use_t=True,
+                                                      df_correction=False))
+        self.res3 = self.res1
+        self.res1 = res_ols
+        self.bse_robust = res_ols.bse
+        self.cov_robust = res_ols.cov_params()
+        cov1 = sw.cov_nw_panel(self.res1, 4, self.tidx)
+        se1 =  sw.se_cov(cov1)
+        self.bse_robust2 = se1
+        self.cov_robust2 = cov1
+        self.small = True
+        self.res2 = res2.results_nw_panel4
+
+        self.skip_f = True
+        self.rtol = 1e-6
+        self.rtolh = 1e-10
+
+
 # TODO: low precision/agreement
 class TestOLSRobustCluster2G(CheckOLSRobustCluster, CheckOLSRobustNewMixin):
     # compare with `reg cluster`