Repository: statsmodels/statsmodels

Issue:
Set degrees of freemdom in reset_ramsey
Does https://github.com/statsmodels/statsmodels/issues/5365 also affect `reset_ramsey` in `statsmodels.stats.outliers_influence`? 

I tried with a dataset and the results I got doesn't seem to match with Stata and then I realized that it may be due to the difference of degrees of freedom. 

From [Stata][2]: `ovtest`
```
Ramsey RESET test using powers of the fitted values of csat
       Ho:  model has no omitted variables
                  F(3, 42) =      5.54
                  Prob > F =      0.0027
```

From SM: `reset_ramsey(results, degree=3)`
```
<class 'statsmodels.stats.contrast.ContrastResults'>
<F test: F=array([[8.25164379]]), p=0.000926704186950011, df_denom=43, df_num=2>
```

Is there any workaround to set the df? Btw, how the `degree` is defined? why is it set to a scalar of 5? [Doc][1]

[1]: http://www.statsmodels.org/0.6.1/_modules/statsmodels/stats/outliers_influence.html
[2]: https://www.stata.com/manuals13/rregresspostestimation.pdf



Discussion:
I wrote reset_ramsey and most other diagnostic, outlier, influence for OLS before I had a copy of Stata to compare with. unit tests for reset_ramsey are in "statsmodels\regression\tests\test_glsar_gretl.py". So it looks like I used Gretl for comparison.

I don't remember much right now. The ramsey test itself just uses the f_test for a auxiliary, augmented OLS regression.

One problem that I see right now is that there is no check that the fittedvalues are properly scaled before computing the polynomial. So this is numerically not very stable if fittedvalues are large.
`y_fitted_vander = np.vander(res.fittedvalues, order)[:, :-2] #drop constant`

5365 is only for penalized estimation.
(There are other issues with df when adding categorical variables, but those don't apply when adding fittedvalues. I think in those cases a bug would make df_num too large and not too small. df should be correct and is based on using the matrix rank of the auxiliary exog, AFAIR. )

I don't see how Stata can have df=3 if degree=3.
In the vander code, copied above, I drop the constant and the fittedvalues. So, "my" reset_test only includes `y_hat**2, ..., yhat**degree`, AFAICS.

Note, I have more recent issues to look at this again, but didn't get around to it yet. e.g. #3690 
and some related issues for variable addition score tests
#1479 #2041


It seems like the `ovtest` in [Stata][1] includes 2nd, 3rd, and 4th power of the preditted y (i.e., y hat). I tried with another R package [lmtest][2], and after I set `power=2:4`, I did get the same results.

**From Stata:** 
`estat ovtest`
```
Ramsey RESET test using powers of the fitted values of csat
       Ho:  model has no omitted variables
                  F(3, 42) =      5.54
                  Prob > F =      0.0027
```
`estat ovtest, rhs`
```
Ramsey RESET test using powers of the independent variables
       Ho:  model has no omitted variables
                 F(15, 30) =      6.61
                  Prob > F =      0.0000
```

**From R:**
`resettest(lmfit, power=2:4, type='fitted')`
```
	RESET test

data:  lmfit
RESET = 5.5417, df1 = 3, df2 = 42, p-value = 0.002686
```
`resettest(lmfit, power=2:4, type='regressor')`
```
	RESET test

data:  lmfit
RESET = 6.6142, df1 = 15, df2 = 30, p-value = 6.098e-06
```
**From SM 0.9.0:**
`reset_ramsey(results, degree=4)`
```
ValueWarning: covariance of constraints does not have full rank. The number of constraints is 3, but rank is 2
  'rank is %d' % (J, J_), ValueWarning)
<class 'statsmodels.stats.contrast.ContrastResults'>
<F test: F=array([[2.24004773]]), p=0.11873335790170665, df_denom=43, df_num=2>
```



I think those args such as `power` and `type` offered by `lmfit` package are worth adding to our SM


[1]: https://www.stata.com/statalist/archive/2004-06/msg00264.html
[2]: https://www.rdocumentation.org/packages/lmtest/versions/0.9-36/topics/resettest
You could add the results for statmodels reset test with `degree=4` for comparison.

I'm adding a bug label for the numerical stability, e.g. we need a `scaling` or similarly named option.

what variables does this R function add?
`resettest(lmfit, power=2:4, type='regressor')`
adding powers of regressors is white test in stats.diagnostic AFAIR.
another option: cov_type, base on Joao Santos Silva's comment quoted in #3690 
It would essentially be trivial to add an option that the user can specify `cov_type` and `cov_kwds` for the auxiliary regression.
I did some examples like that when I read the relevant parts in textbooks like Wooldridge or Davidson MacKinnon.  The problem are unit test because I have not seen it being available in any other stats or econometrics package.


Just edited. What do you mean by "varaibles"? I didn't actaully know much about this R package except for the doc I linked.
```
type a string indicating whether powers of the fitted response, the regressor variables
(factors are left out), or the first principal component of the regressor matrix
should be included in the extended model.
```
https://cran.r-project.org/web/packages/lmtest/lmtest.pdf

Adding the powers of regressors to the existing model is not a reset test (in the strict sense, AFAIU)
Note, white test is for heteroscedasticity, is not like this (it uses squared residuals as endog in auxiliary regression)

there is a general `linear_lm` test, that adds squares of exog or a user provided `func` of exog
http://www.statsmodels.org/dev/stats.html#module-statsmodels.stats.stattools

I think I prefer to keep the name `reset test` for  the case when we add only functions of fittedvalues.

Adding higher powers and cross product of all explanatory variables quickly runs out of degrees of freedom, except in very large datasets.
see answer here https://stats.stackexchange.com/questions/96510/resettest-in-lmtest-unexpected-behavior-on-perfect-fit
R resettest scales endog y in the regression. question is for `type="regressor"`

https://github.com/statsmodels/statsmodels/issues/2091
I just did a quick test to reproduce Stata from the following attached pdf:
[Omitted Variable Tests.pdf](https://github.com/statsmodels/statsmodels/files/2890700/Omitted.Variable.Tests.pdf)

```python
import pandas as pd
import seaborn as sns
import statsmodels.formula.api as smf
#%%
df = sns.load_dataset('tips')
df.head()
# regression
formula = 'tip ~ total_bill + size'
results = smf.ols(formula, data=df).fit()
# yhat2, yhat3, yhat4
df['yhat2'] = results.predict()**2
df['yhat3'] = results.predict()**3
df['yhat4'] = results.predict()**4
# regression
formula = 'tip ~ total_bill + size + yhat2 + yhat3 + yhat4'
results = smf.ols(formula, data=df).fit()
# hypotheses test
hypotheses = 'yhat2 = yhat3 = yhat4 = 0'
results.f_test(hypotheses)
```
```
<class 'statsmodels.stats.contrast.ContrastResults'>
<F test: F=array([[2.02034279]]), p=0.1117239797938144, df_denom=238, df_num=3>
```
Stata gives me:
```
use tips.dta, clear
qui reg tip total_bill size
estat ovtest
```

```
Ramsey RESET test using powers of the fitted values of tip
       Ho:  model has no omitted variables
                 F(3, 238) =      2.02
                  Prob > F =      0.1117
```

However, when I tried with the following dataset I started to get different results: 

[states.zip](https://github.com/statsmodels/statsmodels/files/2890717/states.zip)

Sample code: 
```python
# regression
formula = 'csat ~ expense + percent + income + high + college'
results = smf.ols(formula, data=df).fit()
# 
df['yhat2'] = results.predict()**2
df['yhat3'] = results.predict()**3
df['yhat4'] = results.predict()**4
# regression
formula = 'csat ~ expense + percent + income + high + college + yhat2 + yhat3 + yhat4'
results = smf.ols(formula, data=df).fit()
# hypotheses test
hypotheses = 'yhat2 = yhat3 = yhat4 = 0'
results.f_test(hypotheses)
```
```
ValueWarning: covariance of constraints does not have full rank. The number of constraints is 3, but rank is 2
  'rank is %d' % (J, J_), ValueWarning)
<class 'statsmodels.stats.contrast.ContrastResults'>
<F test: F=array([[2.23240365]]), p=0.11955831122790375, df_denom=43, df_num=2>
```
```
use states.dta, clear
qui reg csat expense percent income high college
estat ovtest
```
```
Ramsey RESET test using powers of the fitted values of csat
       Ho:  model has no omitted variables
                  F(3, 42) =      5.54
                  Prob > F =      0.0027
```

Resolving patch:
diff --git a/statsmodels/stats/outliers_influence.py b/statsmodels/stats/outliers_influence.py
--- a/statsmodels/stats/outliers_influence.py
+++ b/statsmodels/stats/outliers_influence.py
@@ -108,6 +108,12 @@ def reset_ramsey(res, degree=5):
     This is a general specification test, for additional non-linear effects
     in a model.
 
+    Parameters
+    ----------
+    degree : int
+        Maximum power to include in the RESET test.  Powers 0 and 1 are
+        excluded, so that degree tests powers 2, ..., degree of the fitted
+        values.
 
     Notes
     -----
@@ -119,18 +125,20 @@ def reset_ramsey(res, degree=5):
     indicates that there might be additional non-linear effects in the model
     and that the linear model is mis-specified.
 
-
     References
     ----------
     https://en.wikipedia.org/wiki/Ramsey_RESET_test
-
     """
     order = degree + 1
     k_vars = res.model.exog.shape[1]
     # vander without constant and x, and drop constant
-    y_fitted_vander = np.vander(res.fittedvalues, order)[:, :-2]
+    norm_values = np.asarray(res.fittedvalues)
+    norm_values = norm_values / np.sqrt((norm_values ** 2).mean())
+    y_fitted_vander = np.vander(norm_values, order)[:, :-2]
     exog = np.column_stack((res.model.exog, y_fitted_vander))
-    res_aux = OLS(res.model.endog, exog).fit()
+    exog /= np.sqrt((exog ** 2).mean(0))
+    endog = res.model.endog / (res.model.endog ** 2).mean()
+    res_aux = OLS(endog, exog).fit()
     # r_matrix = np.eye(degree, exog.shape[1], k_vars)
     r_matrix = np.eye(degree - 1, exog.shape[1], k_vars)
     # df1 = degree - 1


Test patch:
diff --git a/statsmodels/stats/tests/test_outliers_influence.py b/statsmodels/stats/tests/test_outliers_influence.py
new file mode 100644
--- /dev/null
+++ b/statsmodels/stats/tests/test_outliers_influence.py
@@ -0,0 +1,16 @@
+from numpy.testing import assert_almost_equal
+
+from statsmodels.datasets import statecrime
+from statsmodels.regression.linear_model import OLS
+from statsmodels.stats.outliers_influence import reset_ramsey
+from statsmodels.tools import add_constant
+
+data = statecrime.load_pandas().data
+
+
+def test_reset_stata():
+    mod = OLS(data.violent, add_constant(data[['murder', 'hs_grad']]))
+    res = mod.fit()
+    stat = reset_ramsey(res, degree=4)
+    assert_almost_equal(stat.fvalue[0, 0], 1.52, decimal=2)
+    assert_almost_equal(stat.pvalue, 0.2221, decimal=4)