Repository: statsmodels/statsmodels

Issue:
when doint OLS with refit=True, the model is still penalized
I ran the following code. Since X is constant, the best linear model would have as weight the mean of y, i.e. `2.`.
The L1-penalized model should give `y.mean() - alpha`, so `1.`. When running OLS with `refit=False`, I expected the resulting model to have a weight of `1`. Which it does. When I run with `refit=True`, as I understand in the documentation, there should be no penalization, so I should get a weight of `2.`.

See code below.

```
import numpy as np
import statsmodels.api as sm

X = np.ones(5)
y = np.arange(5)

model = sm.OLS(y, X)

y.mean() # 2.

ols.fit_regularized(alpha=1, L1_wt=1, refit=True).params
# gives [1.], seems wrong

ols.fit_regularized(alpha=1, L1_wt=1, refit=False).params
# gives [1.], as expected

ols.fit_regularized(alpha=0, L1_wt=1).params
# gives [2.], as expected
```
when doint OLS with refit=True, the model is still penalized
I ran the following code. Since X is constant, the best linear model would have as weight the mean of y, i.e. `2.`.
The L1-penalized model should give `y.mean() - alpha`, so `1.`. When running OLS with `refit=False`, I expected the resulting model to have a weight of `1`. Which it does. When I run with `refit=True`, as I understand in the documentation, there should be no penalization, so I should get a weight of `2.`.

See code below.

```
import numpy as np
import statsmodels.api as sm

X = np.ones(5)
y = np.arange(5)

model = sm.OLS(y, X)

y.mean() # 2.

ols.fit_regularized(alpha=1, L1_wt=1, refit=True).params
# gives [1.], seems wrong

ols.fit_regularized(alpha=1, L1_wt=1, refit=False).params
# gives [1.], as expected

ols.fit_regularized(alpha=0, L1_wt=1).params
# gives [2.], as expected
```


Discussion:
From what I understood from the code on the `fit_elasticnet` function, when `refit=True`, the fit has 2 steps:

1. fit the model with penalization
2. fit a new model removing features whose coefficients are zero

The statement `The refitted model is not regularized` in the documentation refers to the fit in step 2. Step 1 is always a penalized fit.

Could this interpretation of the documentation explain your issue?

when reading `The refitted model is not regularized`, I expect the output of `fit_regularized(.., refit=True)` to be the output of what you describe as step 2, an un-regularized model (where some features might have been removed). In my example (where no feature have been removed), I expect `fit_regularized(..., refit=True)` to output the same result as an un-regularized model.
From what I understood from the code on the `fit_elasticnet` function, when `refit=True`, the fit has 2 steps:

1. fit the model with penalization
2. fit a new model removing features whose coefficients are zero

The statement `The refitted model is not regularized` in the documentation refers to the fit in step 2. Step 1 is always a penalized fit.

Could this interpretation of the documentation explain your issue?

when reading `The refitted model is not regularized`, I expect the output of `fit_regularized(.., refit=True)` to be the output of what you describe as step 2, an un-regularized model (where some features might have been removed). In my example (where no feature have been removed), I expect `fit_regularized(..., refit=True)` to output the same result as an un-regularized model.

Resolving patch:
diff --git a/statsmodels/base/elastic_net.py b/statsmodels/base/elastic_net.py
--- a/statsmodels/base/elastic_net.py
+++ b/statsmodels/base/elastic_net.py
@@ -9,7 +9,8 @@
 Routines for fitting regression models using elastic net
 regularization.  The elastic net minimizes the objective function
 
--llf / nobs + alpha((1 - L1_wt) * sum(params**2) / 2 + L1_wt * sum(abs(params)))
+-llf / nobs + alpha((1 - L1_wt) * sum(params**2) / 2 +
+    L1_wt * sum(abs(params)))
 
 The algorithm implemented here closely follows the implementation in
 the R glmnet package, documented here:
@@ -52,17 +53,16 @@ def npscore(params, model):
     def nphess(params, model):
         nobs = model.nobs
         pen_hess = alpha[k] * (1 - L1_wt)
-        h = -model.hessian(np.r_[params], **hess_kwds)[0,0] / nobs + pen_hess
+        h = -model.hessian(np.r_[params], **hess_kwds)[0, 0] / nobs + pen_hess
         return h
 
     return nploglike, npscore, nphess
 
 
-
 def fit_elasticnet(model, method="coord_descent", maxiter=100,
-         alpha=0., L1_wt=1., start_params=None, cnvrg_tol=1e-7,
-         zero_tol=1e-8, refit=False, check_step=True,
-         loglike_kwds=None, score_kwds=None, hess_kwds=None):
+                   alpha=0., L1_wt=1., start_params=None, cnvrg_tol=1e-7,
+                   zero_tol=1e-8, refit=False, check_step=True,
+                   loglike_kwds=None, score_kwds=None, hess_kwds=None):
     """
     Return an elastic net regularized fit to a regression model.
 
@@ -133,7 +133,6 @@ def fit_elasticnet(model, method="coord_descent", maxiter=100,
     """
 
     k_exog = model.exog.shape[1]
-    n_exog = model.exog.shape[0]
 
     loglike_kwds = {} if loglike_kwds is None else loglike_kwds
     score_kwds = {} if score_kwds is None else score_kwds
@@ -148,7 +147,6 @@ def fit_elasticnet(model, method="coord_descent", maxiter=100,
     else:
         params = start_params.copy()
 
-    converged = False
     btol = 1e-4
     params_zero = np.zeros(len(params), dtype=bool)
 
@@ -156,8 +154,9 @@ def fit_elasticnet(model, method="coord_descent", maxiter=100,
                       if k != "offset" and hasattr(model, k)])
     init_args['hasconst'] = False
 
-    fgh_list = [_gen_npfuncs(k, L1_wt, alpha, loglike_kwds, score_kwds, hess_kwds)
-                for k in range(k_exog)]
+    fgh_list = [
+        _gen_npfuncs(k, L1_wt, alpha, loglike_kwds, score_kwds, hess_kwds)
+        for k in range(k_exog)]
 
     for itr in range(maxiter):
 
@@ -181,13 +180,14 @@ def fit_elasticnet(model, method="coord_descent", maxiter=100,
                 offset += model.offset
 
             # Create a one-variable model for optimization.
-            model_1var = model.__class__(model.endog, model.exog[:, k], offset=offset,
-                                         **init_args)
+            model_1var = model.__class__(
+                model.endog, model.exog[:, k], offset=offset, **init_args)
 
             # Do the one-dimensional optimization.
             func, grad, hess = fgh_list[k]
-            params[k] = _opt_1d(func, grad, hess, model_1var, params[k], alpha[k]*L1_wt,
-                                tol=btol, check_step=check_step)
+            params[k] = _opt_1d(
+                func, grad, hess, model_1var, params[k], alpha[k]*L1_wt,
+                tol=btol, check_step=check_step)
 
             # Update the active set
             if itr > 0 and np.abs(params[k]) < zero_tol:
@@ -197,7 +197,6 @@ def fit_elasticnet(model, method="coord_descent", maxiter=100,
         # Check for convergence
         pchange = np.max(np.abs(params - params_save))
         if pchange < cnvrg_tol:
-            converged = True
             break
 
     # Set approximate zero coefficients to be exactly zero
@@ -213,9 +212,10 @@ def fit_elasticnet(model, method="coord_descent", maxiter=100,
     cov = np.zeros((k_exog, k_exog))
     init_args = dict([(k, getattr(model, k, None)) for k in model._init_keys])
     if len(ii) > 0:
-        model1 = model.__class__(model.endog, model.exog[:, ii],
-                               **init_args)
+        model1 = model.__class__(
+            model.endog, model.exog[:, ii], **init_args)
         rslt = model1.fit()
+        params[ii] = rslt.params
         cov[np.ix_(ii, ii)] = rslt.normalized_cov_params
     else:
         # Hack: no variables were selected but we need to run fit in
@@ -236,11 +236,23 @@ def fit_elasticnet(model, method="coord_descent", maxiter=100,
     else:
         scale = 1.
 
+    # The degrees of freedom should reflect the number of parameters
+    # in the refit model, not including the zeros that are displayed
+    # to indicate which variables were dropped.  See issue #1723 for
+    # discussion about setting df parameters in model and results
+    # classes.
+    p, q = model.df_model, model.df_resid
+    model.df_model = len(ii)
+    model.df_resid = model.nobs - model.df_model
+
     # Assuming a standard signature for creating results classes.
     refit = klass(model, params, cov, scale=scale)
     refit.regularized = True
     refit.method = method
-    refit.fit_history = {'iteration' : itr + 1}
+    refit.fit_history = {'iteration': itr + 1}
+
+    # Restore df in model class, see issue #1723 for discussion.
+    model.df_model, model.df_resid = p, q
 
     return refit
 


Test patch:
diff --git a/statsmodels/regression/tests/test_regression.py b/statsmodels/regression/tests/test_regression.py
--- a/statsmodels/regression/tests/test_regression.py
+++ b/statsmodels/regression/tests/test_regression.py
@@ -1201,13 +1201,15 @@ def test_regularized_refit():
     p = 5
     np.random.seed(3132)
     xmat = np.random.normal(size=(n, p))
-    yvec = xmat.sum(1) + np.random.normal(size=n)
+    # covariates 0 and 2 matter
+    yvec = xmat[:, 0] + xmat[:, 2] + np.random.normal(size=n)
     model1 = OLS(yvec, xmat)
     result1 = model1.fit_regularized(alpha=2., L1_wt=0.5, refit=True)
-    model2 = OLS(yvec, xmat)
-    result2 = model2.fit_regularized(alpha=2., L1_wt=0.5, refit=True)
-    assert_allclose(result1.params, result2.params)
-    assert_allclose(result1.bse, result2.bse)
+    model2 = OLS(yvec, xmat[:, [0, 2]])
+    result2 = model2.fit()
+    ii = [0, 2]
+    assert_allclose(result1.params[ii], result2.params)
+    assert_allclose(result1.bse[ii], result2.bse)
 
 
 def test_regularized_predict():