Repository: statsmodels/statsmodels

Issue:
Ridge regression gives wrong and inconsistent results.
statsmodel.api.OLS.fit_regularized goes through a shortcut when L1_wt is 0. However, that shortcut gives a wrong result when alpha is an array instead of a constant. The following code illustrates this issue with statsmodels version 0.8.0.
```python
import statsmodels.api as sm
import numpy as np
X = np.random.randn(1000, 100)
b = np.random.randn(100)
y = np.dot(X, b) + np.random.randn(1000) + 1 # y.mean() == 1.2861581555386801
res = sm.OLS(y, sm.add_constant(X)).fit_regularized(alpha=np.array([0] + [1e10] * 100), L1_wt=1e-8)
print('correct:', res.params[0]) # res.params[0] == 1.28615815554, which is correct.
sm.add_constant(X)).fit_regularized(alpha=np.array([0] + [1e10] * 100), L1_wt=0)
print('wrong:', res.params[0]) # res.params[0] == -0.00191408618425, which is wrong.
```

There is another peculiarity when L1_wt is zero. statsmodel.api.OLS.fit_regularized().params is a numpy array when L1_wt is zero. However, it becomes a pandas Series when L1_wt is not zero. This is inconsistent and confusing.
Ridge regression gives wrong and inconsistent results.
statsmodel.api.OLS.fit_regularized goes through a shortcut when L1_wt is 0. However, that shortcut gives a wrong result when alpha is an array instead of a constant. The following code illustrates this issue with statsmodels version 0.8.0.
```python
import statsmodels.api as sm
import numpy as np
X = np.random.randn(1000, 100)
b = np.random.randn(100)
y = np.dot(X, b) + np.random.randn(1000) + 1 # y.mean() == 1.2861581555386801
res = sm.OLS(y, sm.add_constant(X)).fit_regularized(alpha=np.array([0] + [1e10] * 100), L1_wt=1e-8)
print('correct:', res.params[0]) # res.params[0] == 1.28615815554, which is correct.
sm.add_constant(X)).fit_regularized(alpha=np.array([0] + [1e10] * 100), L1_wt=0)
print('wrong:', res.params[0]) # res.params[0] == -0.00191408618425, which is wrong.
```

There is another peculiarity when L1_wt is zero. statsmodel.api.OLS.fit_regularized().params is a numpy array when L1_wt is zero. However, it becomes a pandas Series when L1_wt is not zero. This is inconsistent and confusing.


Discussion:

Resolving patch:
diff --git a/statsmodels/regression/linear_model.py b/statsmodels/regression/linear_model.py
--- a/statsmodels/regression/linear_model.py
+++ b/statsmodels/regression/linear_model.py
@@ -1018,9 +1018,18 @@ def _fit_ridge(self, alpha):
 
         u, s, vt = np.linalg.svd(self.exog, 0)
         v = vt.T
-        s2 = s*s + alpha * self.nobs
-        params = np.dot(u.T, self.endog) * s / s2
-        params = np.dot(v, params)
+        q = np.dot(u.T, self.endog) * s
+        s2 = s * s
+        if np.isscalar(alpha):
+            sd = s2 + alpha * self.nobs
+            params = q / sd
+            params = np.dot(v, params)
+        else:
+            vtav = self.nobs * np.dot(vt, alpha[:, None] * v)
+            d = np.diag(vtav) + s2
+            np.fill_diagonal(vtav, d)
+            r = np.linalg.solve(vtav, q)
+            params = np.dot(v, r)
 
         from statsmodels.base.elastic_net import RegularizedResults
         return RegularizedResults(self, params)


Test patch:
diff --git a/statsmodels/regression/tests/test_regression.py b/statsmodels/regression/tests/test_regression.py
--- a/statsmodels/regression/tests/test_regression.py
+++ b/statsmodels/regression/tests/test_regression.py
@@ -1149,12 +1149,19 @@ def test_ridge():
     xmat = np.random.normal(size=(n, p))
     yvec = xmat.sum(1) + np.random.normal(size=n)
 
-    for alpha in [1., np.ones(p), 10, 10*np.ones(p)]:
-        model1 = OLS(yvec, xmat)
-        result1 = model1._fit_ridge(alpha=1.)
-        model2 = OLS(yvec, xmat)
-        result2 = model2.fit_regularized(alpha=1., L1_wt=0)
-        assert_allclose(result1.params, result2.params)
+    v = np.ones(p)
+    v[0] = 0
+
+    for a in (0, 1, 10):
+        for alpha in (a, a*np.ones(p), a*v):
+            model1 = OLS(yvec, xmat)
+            result1 = model1._fit_ridge(alpha=alpha)
+            model2 = OLS(yvec, xmat)
+            result2 = model2.fit_regularized(alpha=alpha, L1_wt=0)
+            assert_allclose(result1.params, result2.params)
+            model3 = OLS(yvec, xmat)
+            result3 = model3.fit_regularized(alpha=alpha, L1_wt=1e-10)
+            assert_allclose(result1.params, result3.params)
 
     fv1 = result1.fittedvalues
     fv2 = np.dot(xmat, result1.params)