Repository: statsmodels/statsmodels

Issue:
FAQ: Why are cluster robust p-values so different from those reported by STATA package?
I'm running a OLS regression in STATA and the same one in python's Statsmodels. Note that I adjust for clusters (for id and year).

All the outcomes are very similar if not the same. 

But, the Statsmodels *p*-values are off. 

For example, the one for X3 has a *t*-value of 1.951. But Statsmodels assigns a *p*-value of 0.109, while STATA returns 0.052 (as does Excel for 2-tailed tests and df of 573). 

What's going on? 

Below is the output using `import statsmodels.formula.api as sm`, `mod = sm.ols(formula=regression_model, data=data)` and `res = mod.fit(cov_type='cluster', cov_kwds={'groups': np.array(data[[period_id, firm_id]])}, use_t=True)`:


    Statsmodels
    ==============================================================================
    Dep. Variable:                      Y   R-squared:                       0.362
    Model:                            OLS   Adj. R-squared:                  0.355
    Method:                 Least Squares   F-statistic:                     12.90
    Date:                Thu, 27 Feb 2020   Prob (F-statistic):            0.00654
    Time:                        17:16:54   Log-Likelihood:                -1631.8
    No. Observations:                 580   AIC:                             3278.
    Df Residuals:                     573   BIC:                             3308.
    Df Model:                           6                                         
    Covariance Type:              cluster                                         
    ================================================================================
                       coef    std err          t      P>|t|      [0.025      0.975]
    --------------------------------------------------------------------------------
    Intercept       30.4376      4.711      6.461      0.001      18.328      42.548
    X1              -0.3373      0.574     -0.587      0.583      -1.814       1.139
    X2              -1.0456      0.173     -6.043      0.002      -1.490      -0.601
    X3               0.0293      0.015      1.951      0.109      -0.009       0.068
    X4              -0.5298      0.127     -4.186      0.009      -0.855      -0.204
    X5             -14.2937      3.132     -4.564      0.006     -22.344      -6.243
    X6              -1.0144      0.272     -3.733      0.014      -1.713      -0.316
    ==============================================================================
    Omnibus:                      397.145   Durbin-Watson:                   0.485
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):             5643.097
    Skew:                           2.865   Prob(JB):                         0.00
    Kurtosis:                      17.166   Cond. No.                         218.
    ==============================================================================
    
    Warnings:
    [1] Standard Errors are robust to cluster correlation (cluster)
    ==============================================================================
    STATA:
    
    . cluster2 Y X1 X2 X3 X4 X5 X6, fcluster(id) tcluster(year)
     
    Linear regression with 2D clustered SEs                Number of obs =     580
                                                           F(  6,   573) =   32.41
                                                           Prob > F      =  0.0000
    Number of clusters (id) =        99                    R-squared     =  0.3617
    Number of clusters (year) =       6                    Root MSE      =  4.0575
    ------------------------------------------------------------------------------
      man_buffer |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
    -------------+----------------------------------------------------------------
           _cons |   30.43759   4.711003     6.46   0.000     21.18465    39.69053
             X1  |  -.3373264   .5744737    -0.59   0.557    -1.465658    .7910047
             X2  |  -1.045579   .1730091    -6.04   0.000    -1.385388   -.7057694
             X3  |    .029327    .015031     1.95   0.052    -.0001955    .0588496
             X4  |  -.5297968   .1265761    -4.19   0.000    -.7784065    -.281187
             X5  |  -14.29367   3.131845    -4.56   0.000    -20.44497   -8.142374
             X6  |  -1.014446   .2717238    -3.73   0.000    -1.548142   -.4807496
    
    ------------------------------------------------------------------------------
     
    SE clustered by id and year

I run `Statsmodels api:  0.11.0` and `Pandas:  1.0.1`.


Discussion:
Can you provide some code that will reproduce the problem? We only need the statsmodels part. 
The width of the CI are 2.570579494799406 * 2 * se which is surprising.
SM appears to be using a t_5 distribution to compute the pvalues and CIs.

```
stats.t(5).ppf(1-0.025)
Out[10]: 2.5705818366147395
```

I suspect that if you use_t=False you will get very similar results. 

This choice is probably not crazy since when you cluster by a variable you allow for arbitrary dependence within that variable, as with T=6 it is as-if you have 6 observations.  The question is whether the DoF can be justified and documented. 

use_t should probably no be used with clustered se since these have an asymptotic justification. 

FWIW stata is using the normal.

The unit tests are written against Stata as far as we overlap.
The defaults are not always the same, but AFAIR I tried to match it for OLS. (*)

IIRC, Stata has a `small` option.

The details for the difference in correction factors, degrees of freedom and small sample options are in the unit tests. They should show where and how we match up.
But maybe use_t = False is more unit tested than use_t = True.


(*) The defaults differ from Stata for GLM and discrete.
statsmodels is using the same defaults as for OLS.
Stata does not use some of the same small sample corrections/df in those other models as in OLS.

This is a two-way cluster. AFAIR, Stata did not have it at the time I wrote this.
IIRC, I used the min of cluster sizes for the df
It looks like two cluster was unit tested against ivreg2
`cmdline="ivreg2 invest mvalue kstock, cluster(company time)",`
But there is a code comment that confint don't agree well with `small` options

stata results in  `statsmodels.regression.tests.results.results_grunfeld_ols_robust_cluster.py`
unit tests in  statsmodels.regression.tests.test_robustcov  TestOLSRobustCluster2GLarge

https://www.stata.com/meeting/boston10/boston10_baum.pdf
p 29 M = min(G1, G2)


labeled as FAQ so we can leave it open as reference
Stata 14 still does not have two cluster vce option

What's `cluster2` used in the Stata version? Is it from a user provided package?

Petersen has a cluster2.ado, found with google search
https://www.kellogg.northwestern.edu/faculty/petersen/htm/papers/se/se_programming.htm
Cluster2 is indeed from Peteren. But I get same results if I use [VCE2WAY](https://ideas.repec.org/c/boc/bocode/s458311.html) - and ... vernerable Excel.  

However, please do not be blindsided by Stata. The mapping of _t_-values to _p_-values by statsmodels is not clear to me. AFAIK a _t_-value of  1.95 should lead to a _p_-value of around 5 pct, not 10.
> a t-value of 1.95 should lead to a p-value of around 5 pct

that's for normal distribution. 
In this case you have a t distribution with only 5 degrees of freedom, which has much larger confidence interval than under normal distribution or t-distribution with large df.

AFAIR, the recommendation came from Cameron and Trivedi which is the main reference for performance of multi-way cluster robust standard errors.

In the one-way cluster case, the official Stata also uses `df = n_groups - 1`, I assume also for the p-values.


You can `use_t=False`, then you will get p-values close to t distribution with large df

according to the docstring, there is an option to turn off the df correction.
You could try `df_correction=False` in the `cov_kwds`. I don't remember the details for that. 

```
- `df_correction` bool (optional)
              If True (default), then the degrees of freedom for the
              inferential statistics and hypothesis tests, such as
              pvalues, f_pvalue, conf_int, and t_test and f_test, are
              based on the number of groups minus one instead of the
              total number of observations minus the number of explanatory
              variables. `df_resid` of the results instance is adjusted.
              If False, then `df_resid` of the results instance is not
              adjusted.
```
Wow, using 5 df gets that _p_-value indeed. Perhaps explain that in the docs more clearly. 
AFAIR, the theoretical arguments

The df would depend on where we have the variation in an explanatory variable, i.e. from where do we get the information about the parameters.
using the minimum of the number of groups is conservative (AFAIR), that would be the case if we have only between variation across those groups, but no within variation in other directions.

In the example the short dimension is the cross-section. So our default kind of assumes that we only have cross-sectional variation and constant across time periods.

To take this into account in the implementation of cluster robust standard errors is very difficult and I haven't tried yet.

Here are issues with some of my notes, there might be more notes in other issues or PRs
#1201
#2136
FWIW I think statsmodels is correct and Petersen is wrong here.  The number of clusters is the number of uncorrelated observations in the sample, so using the min for small sample adjustment seems reasonable. 
Wow, using 5 df gets that _p_-value indeed. Perhaps explain that in the docs more clearly. 

> FWIW I think statsmodels is correct and Petersen is wrong here. The number of clusters is the number of uncorrelated observations in the sample, so using the min for small sample adjustment seems reasonable.

Will keep that in mind, thanks!
https://github.com/statsmodels/statsmodels/issues/6564#issuecomment-593134953

I found a reference again that I saw last week.
There is some literature on finding data/design driven degrees of freedom for small sample cases, but I never tried to get further than reading abstracts.


Resolving patch:
diff --git a/statsmodels/regression/linear_model.py b/statsmodels/regression/linear_model.py
--- a/statsmodels/regression/linear_model.py
+++ b/statsmodels/regression/linear_model.py
@@ -2338,7 +2338,12 @@ def get_robustcov_results(self, cov_type='HC1', use_t=None, **kwargs):
                   pvalues, f_pvalue, conf_int, and t_test and f_test, are
                   based on the number of groups minus one instead of the
                   total number of observations minus the number of explanatory
-                  variables. `df_resid` of the results instance is adjusted.
+                  variables. `df_resid` of the results instance is also
+                  adjusted. When `use_t` is also True, then pvalues are
+                  computed using the Student's t distribution using the
+                  corrected values. These may differ substantially from
+                  p-values based on the normal is the number of groups is
+                  sma...
                   If False, then `df_resid` of the results instance is not
                   adjusted.
 


Test patch: