Repository: statsmodels/statsmodels

Issue:
ENH: faster whitening matrix calculation for sm.GLS()
`sm.GLS()` starts by calculating a whitening matrix, which the code calls `cholsigmainv`:

`cholsigmainv = np.linalg.cholesky(np.linalg.inv(sigma)).T`

But if you switch the order of operations so the Cholesky decomposition goes first (which is mathematically valid, if you get rid of the transpose), you only have to invert a lower triangular matrix instead of a dense matrix, which can be done efficiently with Lapack's [`dtrtri()`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lapack.dtrtri.html). 

`scipy.linalg.cholesky(..., lower=True)` also gives a slight speed-up over `np.linalg.cholesky(...)`, as does adding `overwrite_c=True` to `dtrtri()`. 

So I'd propose switching [this line](https://github.com/statsmodels/statsmodels/blob/edd843f0afb37b30cb503709e2d4ff4bc57f9c04/statsmodels/regression/linear_model.py#L181) to:

```
from scipy.linalg import cholesky
from scipy.linalg.lapack import dtrtri
cholsigmainv = dtrtri(cholesky(sigma, lower=True), lower=True, overwrite_c=True)[0]
```

This formulation is about four and a half times faster for a random 10,000 x 10,000 covariance matrix:

```
import numpy as np
from scipy.linalg import cholesky
from scipy.linalg.lapack import dtrtri
from timeit import timeit

arr = np.random.RandomState(0).random(size=(10000, 1000))
sigma = np.cov(arr) + 1e-8 * np.eye(len(arr))
timeit(lambda: np.linalg.cholesky(np.linalg.inv(sigma)).T, number=1)  # 304.3108371179551 seconds
timeit(lambda: dtrtri(cholesky(sigma, lower=True), lower=True, overwrite_c=True), number=1)  # 68.95121738314629 seconds
```

Calculating the whitening matrix becomes a major bottleneck for GLS when there are a lot of data points, and this enhancement should help overcome that.


Discussion:
PR welcome - hopefully it will pass all existing tests.
Does calling `dtrtri` require more exception handling by the user or in statsmodels, incase of a singular matrix? Or I suppose `cholesky` will ensure that it is non-singular?
Good point - `dtrtri` doesn't do the usual error handling that higher-level scipy functions do. Along with the inverse, it returns an integer `info`:
```
          = 0: successful exit
          < 0: if INFO = -i, the i-th argument had an illegal value
          > 0: if INFO = i, A(i,i) is exactly zero.  The triangular
               matrix is singular and its inverse can not be computed.
```
For the pull request, how should I handle info != 0? `raise LinAlgError('dtrtri returned non-zero exit code {}'.format(info))`?
You should propogate this I think.  Ideally, the function would work the same after the PR.  You could do this.

1. Add a test with `pytest.raises` against the current mater using a singular `sigma`.  Make sure it raises and note the exception (LinAlgError?).  
2. Make the PR.
3. Ensure that the code still passes the test written in 1 with identical exception and message.



Resolving patch:
diff --git a/statsmodels/regression/linear_model.py b/statsmodels/regression/linear_model.py
--- a/statsmodels/regression/linear_model.py
+++ b/statsmodels/regression/linear_model.py
@@ -40,7 +40,8 @@
 
 import numpy as np
 from scipy import optimize, stats
-from scipy.linalg import toeplitz
+from scipy.linalg import cholesky, toeplitz
+from scipy.linalg.lapack import dtrtri
 
 import statsmodels.base.model as base
 import statsmodels.base.wrapper as wrap
@@ -178,7 +179,13 @@ def _get_sigma(sigma, nobs):
         if sigma.shape != (nobs, nobs):
             raise ValueError("Sigma must be a scalar, 1d of length %s or a 2d "
                              "array of shape %s x %s" % (nobs, nobs, nobs))
-        cholsigmainv = np.linalg.cholesky(np.linalg.inv(sigma)).T
+        cholsigmainv, info = dtrtri(cholesky(sigma, lower=True),
+                                    lower=True, overwrite_c=True)
+        if info > 0:
+            raise np.linalg.LinAlgError('Cholesky decomposition of sigma '
+                                        'yields a singular matrix')
+        elif info < 0:
+            raise ValueError('Invalid input to dtrtri (info = %d)' % info)
     return sigma, cholsigmainv
 
 


Test patch: