Repository: statsmodels/statsmodels

Issue:
ENH/BUG: GLM fit_irls does not have a method option
**edit** 
Currently lstsq and pinv have the same behavior, therefore no backwards incompatible change. Trying an example with near collinearity around the threshold, both lstsq and pinv show exactly the same cross-over between regularized and unregularized solution.

Adding the method argument is therefore an ENH not BUG
`<end edit>`


also breaks backwards compatibility in corner cases

related #3994 is changing rcond for lstsq which is now the default for irls

`wls_results = reg_tools._MinimalWLS(wlsendog, wlsexog, self.weights).fit(method='lstsq')`

similarly, RLM doesn't have an option but uses the default which is 'pinv' the same as WLS
```
wls_results = reg_tools._MinimalWLS(self.endog, self.exog,
                                                weights=self.weights).fit()
```


I'm using example from #3257 with offset which raises exception `SVD not converged` in lstsq, and I cannot switch back to "pinv" to compare
`LinAlgError: SVD did not converge in Linear Least Squares`






Discussion:

```
import statsmodels.api as sm
import statsmodels.formula.api as smf
data = sm.datasets.fair.load_pandas()
data.data['Exposure'] = 1
data.data['Exposure'][1000:2000] = .5
data.data['Exposure'][2000:4000] = .2
data.data['Exposure'][4000:] = 0.1
#model = smf.glm('affairs ~ age + rate_marriage', data=data.data, exposure=data.data['Exposure'],
model = smf.glm('affairs ~ age + rate_marriage', data=data.data, offset=np.log(data.data['Exposure']),
                family=sm.families.Tweedie(link=sm.families.links.Power(1), var_power=1.5))
x = model.fit()
x2 = model.fit(method='lbfgs')
x3 = model.fit(method='newton')
```

```
---------------------------------------------------------------------------
LinAlgError                               Traceback (most recent call last)
<ipython-input-59-7b194b05aaa5> in <module>()
      9 model = smf.glm('affairs ~ age + rate_marriage', data=data.data, offset=np.log(data.data['Exposure']),
     10                 family=sm.families.Tweedie(link=sm.families.links.Power(1), var_power=1.5))
---> 11 x = model.fit()
     12 x2 = model.fit(method='lbfgs')
     13 x3 = model.fit(method='newton')

m:\...\statsmodels\genmod\generalized_linear_model.py in fit(self, start_params, maxiter, method, tol, scale, cov_type, cov_kwds, use_t, full_output, disp, max_start_irls, **kwargs)
    995             return self._fit_irls(start_params=start_params, maxiter=maxiter,
    996                                   tol=tol, scale=scale, cov_type=cov_type,
--> 997                                   cov_kwds=cov_kwds, use_t=use_t, **kwargs)
    998         else:
    999             self._optim_hessian = kwargs.get('optim_hessian')

m:\...\statsmodels\genmod\generalized_linear_model.py in _fit_irls(self, start_params, maxiter, tol, scale, cov_type, cov_kwds, use_t, **kwargs)
   1100             wlsendog = (lin_pred + self.family.link.deriv(mu) * (self.endog-mu)
   1101                         - self._offset_exposure)
-> 1102             wls_results = reg_tools._MinimalWLS(wlsendog, wlsexog, self.weights).fit(method='lstsq')
   1103             lin_pred = np.dot(self.exog, wls_results.params) + self._offset_exposure
   1104             mu = self.family.fitted(lin_pred)

m:\...\statsmodels\regression\_tools.py in fit(self, method)
     90             params = np.linalg.solve(R, np.dot(Q.T, self.wendog))
     91         else:
---> 92             params, _, _, _ = np.linalg.lstsq(self.wexog, self.wendog)
     93 
     94         fitted_values = self.exog.dot(params)

C:\...\python-3.4.4.amd64\lib\site-packages\numpy\linalg\linalg.py in lstsq(a, b, rcond)
   1919                                  0, work, lwork, iwork, 0)
   1920     if results['info'] > 0:
-> 1921         raise LinAlgError('SVD did not converge in Linear Least Squares')
   1922     resids = array([], result_real_t)
   1923     if is_1d:

LinAlgError: SVD did not converge in Linear Least Squares
```
suggested name for irls keyword kwarg option `wls_method` 
maybe misleading,
This is similar to EIM option in the fit_gradient method in that it is only for the optimization. The current irls in GLM calls WLS at the end independently of the MinimalWLS method used in the irls loop.

(I just checked, all current MinimalWLS methods use the decomposition of exog, and none uses the product matrix directly x.T dot x)

possible change: make wls_method also apply to the final WLS call (which does not yet have lstsq as option).
Another option would be to add `rcond`, which we don't expose anywhere, AFAIK. (But we can leave that open until we know how to resolve #3994 and #2628)

If difference pinv and lstsq is minor (they essentially use the same general SVD based approach, AFAIR), then this is a minor issue. (but we still need method option for testing)

minor aside: scale is not recomputed after the final WLS call

Resolving patch:
diff --git a/statsmodels/genmod/generalized_linear_model.py b/statsmodels/genmod/generalized_linear_model.py
--- a/statsmodels/genmod/generalized_linear_model.py
+++ b/statsmodels/genmod/generalized_linear_model.py
@@ -981,6 +981,15 @@ def fit(self, start_params=None, maxiter=100, method='IRLS', tol=1e-8,
             :math:`rtol * prior + atol > abs(current - prior)`
         tol_criterion : str, optional
             Defaults to ``'deviance'``. Can optionally be ``'params'``.
+        wls_method : str, optional
+            options are 'lstsq', 'pinv' and 'qr'
+            specifies which linear algebra function to use for the irls
+            optimization. Default is `lstsq` which uses the same underlying
+            svd based approach as 'pinv', but is faster during iterations.
+            'lstsq' and 'pinv' regularize the estimate in singular and
+            near-singular cases by truncating small singular values based
+            on `rcond` of the respective numpy.linalg function. 'qr' is
+            only valied for cases that are not singular nor near-singular.
 
         If a scipy optimizer is used, the following additional parameter is
         available:
@@ -989,6 +998,15 @@ def fit(self, start_params=None, maxiter=100, method='IRLS', tol=1e-8,
             When 'oim', the default, the observed Hessian is used in fitting.
             'eim' is the expected Hessian. This may provide more stable fits,
             but adds assumption that the Hessian is correctly specified.
+
+        Notes
+        -----
+        If method is 'IRLS', then an additional keyword 'attach_wls' is
+        availble. This is currently for internal use only and might change
+        in future versions. If attach_wls' is true, then the final WLS
+        instance of the IRLS iteration is attached to the results instance
+        as `results_wls` attribute.
+
         """
         self.scaletype = scale
 
@@ -1063,9 +1081,11 @@ def _fit_irls(self, start_params=None, maxiter=100, tol=1e-8,
         Fits a generalized linear model for a given family using
         iteratively reweighted least squares (IRLS).
         """
+        attach_wls = kwargs.pop('attach_wls', False)
         atol = kwargs.get('atol')
         rtol = kwargs.get('rtol', 0.)
         tol_criterion = kwargs.get('tol_criterion', 'deviance')
+        wls_method = kwargs.get('wls_method', 'lstsq')
         atol = tol if atol is None else atol
 
         endog = self.endog
@@ -1100,7 +1120,8 @@ def _fit_irls(self, start_params=None, maxiter=100, tol=1e-8,
                             self.family.weights(mu))
             wlsendog = (lin_pred + self.family.link.deriv(mu) * (self.endog-mu)
                         - self._offset_exposure)
-            wls_results = reg_tools._MinimalWLS(wlsendog, wlsexog, self.weights).fit(method='lstsq')
+            wls_model = reg_tools._MinimalWLS(wlsendog, wlsexog, self.weights)
+            wls_results = wls_model.fit(method=wls_method)
             lin_pred = np.dot(self.exog, wls_results.params) + self._offset_exposure
             mu = self.family.fitted(lin_pred)
             history = self._update_history(wls_results, mu, history)
@@ -1115,7 +1136,9 @@ def _fit_irls(self, start_params=None, maxiter=100, tol=1e-8,
         self.mu = mu
 
         if maxiter > 0:  # Only if iterative used
-            wls_results = lm.WLS(wlsendog, wlsexog, self.weights).fit()
+            wls_method2 = 'pinv' if wls_method == 'lstsq' else wls_method
+            wls_model = lm.WLS(wlsendog, wlsexog, self.weights)
+            wls_results = wls_model.fit(method=wls_method2)
 
         glm_results = GLMResults(self, wls_results.params,
                                  wls_results.normalized_cov_params,
@@ -1124,6 +1147,11 @@ def _fit_irls(self, start_params=None, maxiter=100, tol=1e-8,
                                  use_t=use_t)
 
         glm_results.method = "IRLS"
+        glm_results.mle_settings = {}
+        glm_results.mle_settings['wls_method'] = wls_method
+        glm_results.mle_settings['optimizer'] = glm_results.method
+        if (maxiter > 0) and (attach_wls is True):
+            glm_results.results_wls = wls_results
         history['iteration'] = iteration + 1
         glm_results.fit_history = history
         glm_results.converged = converged
@@ -1160,8 +1188,8 @@ def fit_regularized(self, method="elastic_net", alpha=0.,
         The penalty is the ``elastic net`` penalty, which is a
         combination of L1 and L2 penalties.
 
-        The function that is minimized is: 
-        
+        The function that is minimized is:
+
         .. math::
 
             -loglike/n + alpha*((1-L1\_wt)*|params|_2^2/2 + L1\_wt*|params|_1)


Test patch:
diff --git a/statsmodels/genmod/tests/test_glm.py b/statsmodels/genmod/tests/test_glm.py
--- a/statsmodels/genmod/tests/test_glm.py
+++ b/statsmodels/genmod/tests/test_glm.py
@@ -1170,6 +1170,35 @@ def test_gradient_irls_eim():
                                     atol=5e-5)
 
 
+def test_glm_irls_method():
+    nobs, k_vars = 50, 4
+    np.random.seed(987126)
+    x = np.random.randn(nobs, k_vars - 1)
+    exog = add_constant(x, has_constant='add')
+    y = exog.sum(1) + np.random.randn(nobs)
+
+    mod = GLM(y, exog)
+    res1 = mod.fit()
+    res2 = mod.fit(wls_method='pinv', attach_wls=True)
+    res3 = mod.fit(wls_method='qr', attach_wls=True)
+    # fit_gradient does not attach mle_settings
+    res_g1 = mod.fit(start_params=res1.params, method='bfgs')
+
+    for r in [res1, res2, res3]:
+        assert_equal(r.mle_settings['optimizer'], 'IRLS')
+        assert_equal(r.method, 'IRLS')
+
+    assert_equal(res1.mle_settings['wls_method'], 'lstsq')
+    assert_equal(res2.mle_settings['wls_method'], 'pinv')
+    assert_equal(res3.mle_settings['wls_method'], 'qr')
+
+    assert_(hasattr(res2.results_wls.model, 'pinv_wexog'))
+    assert_(hasattr(res3.results_wls.model, 'exog_Q'))
+
+    # fit_gradient currently does not attach mle_settings
+    assert_equal(res_g1.method, 'bfgs')
+
+
 class CheckWtdDuplicationMixin(object):
     decimal_params = DECIMAL_4
 
@@ -1261,7 +1290,7 @@ def test_tpvalues(self):
 
 
 class TestWtdGlmPoisson(CheckWtdDuplicationMixin):
-    
+
     @classmethod
     def setup_class(cls):
         '''