Repository: statsmodels/statsmodels

Issue:
Ridge regression in GLM gives incorrect results
#### Describe the bug

Something unexpected occurs when I perform a **logistic regression**. I believe it has got something to do with the **intercept** in the model. It only occurs when I use **fit_regularised()** and, in particular, when **L1_wt** is zero. In this case, the [hidden function](https://github.com/statsmodels/statsmodels/blob/1212616d27ab820d303377f0bcf421cd3f46c289/statsmodels/genmod/generalized_linear_model.py#L1260) **_fit_ridge()** is used.

**Note**:  Possibly related to https://github.com/statsmodels/statsmodels/issues/3624.

#### Code Sample, a copy-pastable example if possible

```python
import numpy
import scipy.special
import statsmodels.api
import matplotlib.pyplot
import sys
print("sys version: " + sys.version)
print("statsmodels version: " + statsmodels.__version__ )
print()
numpy.random.seed(123)
n = 50
interceptData = numpy.repeat(1, n)
timeData = numpy.arange(0, n)
sinData = numpy.sin(2 * numpy.pi * timeData / n)
errorData = numpy.random.normal(0, 1, n)
yData = scipy.special.expit(2.3 - 0.1 * timeData + 2.0 * sinData + 0.6 * errorData)
print("yData min: " + str(numpy.min(yData)))
print("yData mean: " + str(numpy.mean(yData)))
print("yData max: " + str(numpy.max(yData)))
for timeAdd in [0, 999]:
    for interceptAdd in [0, 999]:
        for L1_wtValue in [0, 1e-9]:
            print()
            print("timeAdd: " + str(timeAdd) + " interceptAdd: " + str(interceptAdd) + " L1_wtValue: " + str(L1_wtValue))
            interceptDataNew = interceptData + interceptAdd
            timeDataNew = timeData + timeAdd
            exogData = numpy.transpose([interceptDataNew, timeDataNew, sinData])
            model = statsmodels.api.GLM(endog=yData, exog=exogData, family=statsmodels.api.families.Binomial())
            fit = model.fit_regularized(alpha=0, L1_wt=L1_wtValue)
            print("parameters: " + str(fit.params))
            print("fit min: " + str(numpy.min(fit.fittedvalues)))
            print("fit mean: " + str(numpy.mean(fit.fittedvalues)))
            print("fit max: " + str(numpy.max(fit.fittedvalues)))
            print("MSE: " + str(numpy.mean(numpy.power(fit.fittedvalues - yData, 2))))
            matplotlib.pyplot.scatter(timeData, yData, color="red")
            matplotlib.pyplot.scatter(timeData, fit.fittedvalues, color="green")
            matplotlib.pyplot.show()
```

#### Expected Output

The models should all produce identical output, with a low MSE.

#### Output of ``import statsmodels.api statsmodels.__version__ `

sys version: 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
statsmodels version: 0.10.2




Discussion:
Here is a simpler example which goes wrong only for GLM ridge regression (fun: GLM | y: y2 | L1_wt: 0).

```python
import numpy
import pandas
import statsmodels.api
import matplotlib.pyplot
numpy.random.seed(123)
n = 50
data = pandas.DataFrame(numpy.arange(0, n), columns=["x"])
data["e"] = numpy.random.normal(0, 1, n)
data["y1"] = 1.0 + 1.0 * data["x"] + 1.0 * data["e"]
data["y2"] = 1000.0 + 1.0 * data["x"] + 1.0 * data["e"]
for fun in [statsmodels.api.formula.ols, statsmodels.api.formula.glm]:
    for y in ["y1", "y2"]:
        for L1_wtValue in [0, 1e-9]:
            print("fun: " + str(fun) + " | y: " + y + " | L1_wt: " + str(L1_wtValue))
            formula = y + " ~ x"
            model = fun(formula, data)
            fit = model.fit_regularized(alpha=0, L1_wt=L1_wtValue)
            print("parameters: " + str(fit.params))
            print("fit min: " + str(numpy.min(fit.fittedvalues)))
            print("fit mean: " + str(numpy.mean(fit.fittedvalues)))
            print("fit max: " + str(numpy.max(fit.fittedvalues)))
            print("MSE: " + str(numpy.mean(numpy.power(fit.fittedvalues - data[y], 2))))
            matplotlib.pyplot.scatter(data["x"], data[y], color="red")
            matplotlib.pyplot.scatter(data["x"], fit.fittedvalues, color="green")
            matplotlib.pyplot.show()
```
> Something unexpected occurs ...

What are we supposed to see in these examples? What is unexpected?

I ran both examples but I don't see anything obviously unexpected or wrong.
(using a statsmodels branch that is based on maybe 2 months old master)

> Expected Output: The models should all produce identical output, with a low MSE.
They are all close to each other when I run it, at least at 4 decimals.

collecting all mse from second example:

```
mse_all
[0.056102742263092586,
 0.056102742302644136,
 0.05610274226301426,
 0.05613534360829357,
 0.05610274226309277,
 0.05610274230264396,
 0.05610274258968962,
 0.056135343608498804]
```
with case
```
[(0.056102742263092586, 'OLS', 'y1', 0),
 (0.056102742302644136, 'OLS', 'y1', 1e-09),
 (0.05610274226301426, 'OLS', 'y2', 0),
 (0.05613534360829357, 'OLS', 'y2', 1e-09),
 (0.05610274226309277, 'GLM', 'y1', 0),
 (0.05610274230264396, 'GLM', 'y1', 1e-09),
 (0.05610274258968962, 'GLM', 'y2', 0),
 (0.056135343608498804, 'GLM', 'y2', 1e-09)]
```
The top shows much more suspect behavior:

![image](https://user-images.githubusercontent.com/5585221/72743159-05b9eb80-3ba3-11ea-94cd-03cc2a91150a.png)

Moving form 0 to 1e-09 makes a huge difference. 
Perhaps this has been updated recently. For example 1, I get:

```
[0.003934268752320802,
 0.003934454475040054,
 0.007325498193498485,
 0.003934454475040054,
 0.07199935382475343,
 0.012145489538340904,
 0.017848822470291844,
 0.012145489538340904]
```

For example 2, I get:

```
[1.4025685565773134,
 1.4025685566104167,
 1.4025685565773134,
 1.4025685566104251,
 1.4025685565773187,
 1.4025685566104173,
 143414.5987595207,
 1.4025685566104251]
```

With output for example 2:

![image](https://user-images.githubusercontent.com/22837550/72746421-5f75e200-3bb2-11ea-94c2-caaef0b0974b.png)

Note that I am not using the master branch, but the latest release:

> Output of ``import statsmodels.api statsmodels.version `: sys version: 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)], statsmodels version: 0.10.2

I'll try pulling the dev version and test again.
I didn't see the plots for the first example before. Now I also see the "bad" fit

BUG/ENH: AFAICS, there is no way to change the optimizer in `_fit_ridge` through fit regularized

using fit_ridge directly, 'bfgs' is even worse than default newton-cg
However, using "nm", I get results where all plots show approximately the same good fit.
`fit = model._fit_ridge(0, None, method='Nelder-Mead')`

So I guess this is an optimization problem in Ridge with badly scaled data.
mse_all using ridge with "nm"
`mse_all.append((mse, timeAdd, interceptAdd, L1_wtValue))`

```
mse_all
[(0.00393427035334331, 0, 0, 0),
 (0.00393427035334331, 0, 0, 1e-09),
 (0.003934227662235426, 0, 999, 0),
 (0.003934227662235426, 0, 999, 1e-09),
 (0.003934268825027017, 999, 0, 0),
 (0.003934268825027017, 999, 0, 1e-09),
 (0.003934270476165579, 999, 999, 0),
 (0.003934270476165579, 999, 999, 1e-09)]
```

Resolving patch:
diff --git a/statsmodels/genmod/generalized_linear_model.py b/statsmodels/genmod/generalized_linear_model.py
--- a/statsmodels/genmod/generalized_linear_model.py
+++ b/statsmodels/genmod/generalized_linear_model.py
@@ -1201,7 +1201,8 @@ def _fit_irls(self, start_params=None, maxiter=100, tol=1e-8,
         return GLMResultsWrapper(glm_results)
 
     def fit_regularized(self, method="elastic_net", alpha=0.,
-                        start_params=None, refit=False, **kwargs):
+                        start_params=None, refit=False,
+                        opt_method="bfgs", **kwargs):
         r"""
         Return a regularized fit to a linear regression model.
 
@@ -1220,6 +1221,8 @@ def fit_regularized(self, method="elastic_net", alpha=0.,
             If True, the model is refit using only the variables that
             have non-zero coefficients in the regularized fit.  The
             refitted model is not regularized.
+        opt_method : string
+            The method used for numerical optimization.
         **kwargs
             Additional keyword arguments used when fitting the model.
 
@@ -1258,7 +1261,7 @@ def fit_regularized(self, method="elastic_net", alpha=0.,
         """
 
         if kwargs.get("L1_wt", 1) == 0:
-            return self._fit_ridge(alpha, start_params)
+            return self._fit_ridge(alpha, start_params, opt_method)
 
         from statsmodels.base.elastic_net import fit_elasticnet
 
@@ -1280,7 +1283,7 @@ def fit_regularized(self, method="elastic_net", alpha=0.,
 
         return result
 
-    def _fit_ridge(self, alpha, start_params, method="newton-cg"):
+    def _fit_ridge(self, alpha, start_params, method):
 
         if start_params is None:
             start_params = np.zeros(self.exog.shape[1])


Test patch:
diff --git a/statsmodels/genmod/tests/test_glm.py b/statsmodels/genmod/tests/test_glm.py
--- a/statsmodels/genmod/tests/test_glm.py
+++ b/statsmodels/genmod/tests/test_glm.py
@@ -1983,9 +1983,9 @@ def test_tweedie_EQL():
         rtol=rtol, atol=atol)
 
     # Series of ridge fits using gradients
-    ev = (np.array([1.00186882, -0.99213087, 0.00717758, 0.50610942]),
-          np.array([0.98560143, -0.96976442, 0.00727526, 0.49749763]),
-          np.array([0.20643362, -0.16456528, 0.00023651, 0.10249308]))
+    ev = (np.array([1.001778, -0.99388, 0.00797, 0.506183]),
+          np.array([0.985841, -0.969124, 0.007319, 0.497649]),
+          np.array([0.206429, -0.164547, 0.000235, 0.102489]))
     for j, alpha in enumerate([0.05, 0.5, 0.7]):
         model3 = sm.GLM(y, x, family=fam)
         result3 = model3.fit_regularized(L1_wt=0, alpha=alpha)
@@ -2092,6 +2092,34 @@ def testTweediePowerEstimate():
     p = model1.estimate_tweedie_power(res1.mu)
     assert_allclose(p, res2.params[1], rtol=0.25)
 
+def test_glm_lasso_6431():
+
+    # Based on issue #6431
+    # Fails with newton-cg as optimizer
+    np.random.seed(123)
+
+    from statsmodels.regression.linear_model import OLS
+
+    n = 50
+    x = np.ones((n, 2))
+    x[:, 1] = np.arange(0, n)
+    y = 1000 + x[:, 1] + np.random.normal(0, 1, n)
+
+    params = np.r_[999.82244338, 1.0077889]
+
+    for method in "bfgs", None:
+        for fun in [OLS, GLM]:
+
+            # Changing L1_wtValue from 0 to 1e-9 changes
+            # the algorithm from scipy gradient optimization
+            # to statsmodels coordinate descent
+            for L1_wtValue in [0, 1e-9]:
+                model = fun(y, x)
+                if fun == OLS:
+                    fit = model.fit_regularized(alpha=0, L1_wt=L1_wtValue)
+                else:
+                    fit = model._fit_ridge(alpha=0, start_params=None, method=method)
+                assert_allclose(params, fit.params, atol=1e-6, rtol=1e-6)
 
 class TestRegularized(object):