Repository: statsmodels/statsmodels

Issue:
BUG: Hessian inversion check is inadequate
The inverse check at

https://github.com/statsmodels/statsmodels/blob/d2de2f0d91e89ff84856d86c2b2d2aa02e6c0fe5/statsmodels/base/model.py#L445

``` python
            try:
                Hinv = np.linalg.inv(-1 * self.hessian(xopt))
            except:
                #might want custom warning ResultsWarning? NumericalWarning?
                from warnings import warn
                warndoc = ('Inverting hessian failed, no bse or '
                           'cov_params available')
                warn(warndoc, RuntimeWarning)
                Hinv = None
```

is inadequate and only wil have issues when eigenvalues are very close to zero.  It doesn't verify that all eigenvalues have the same sign (positive, for `-1 * self.hessian(xopt)`).  A better check is to ensure that the minimum eigenvale of `-1 * self.hessian(xopt)` is sufficiently positive and to remove the `try-catch` block since it should not be hit when using a PD hessian.



Discussion:
When PD is enforced, there are at least 131 failing tests:

https://travis-ci.org/bashtage/statsmodels/jobs/142324304

Most failures are in `duration`, `discrete` and `miscmodels`

I wasn't immediately able to observe any problems in duration with Hessian inversion.  May be numpy version or platform dependent?

The estimated Hessian in these tests is indefinite:

```
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_1000_10.csv', 'breslow', False, False) ... 0.548443466752 -0.617201578098
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_1000_10.csv', 'breslow', False, True) ... 0.548443466752 -0.617201578098
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_1000_10.csv', 'breslow', True, False) ... 0.548443466752 -0.617201578098
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_1000_10.csv', 'breslow', True, True) ... 0.548443466752 -0.617201578098
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_1000_10.csv', 'efron', False, False) ... 0.547381212152 -0.637886570934
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_1000_10.csv', 'efron', False, True) ... 0.547381212152 -0.637886570934
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_1000_10.csv', 'efron', True, False) ... 0.547381212152 -0.637886570934
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_1000_10.csv', 'efron', True, True) ... 0.547381212152 -0.637886570934
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_100_5.csv', 'breslow', False, False) ... 0.816992449502 -0.890857857722
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_100_5.csv', 'breslow', False, True) ... 0.816992449502 -0.890857857722
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_100_5.csv', 'breslow', True, False) ... 0.816992449502 -0.890857857722
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_100_5.csv', 'breslow', True, True) ... 0.816992449502 -0.890857857722
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_100_5.csv', 'efron', False, False) ... 0.700944039145 -0.889573986453
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_100_5.csv', 'efron', False, True) ... 0.700944039145 -0.889573986453
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_100_5.csv', 'efron', True, False) ... 0.700944039145 -0.889573986453
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_100_5.csv', 'efron', True, True) ... 0.700944039145 -0.889573986453
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_50_2.csv', 'breslow', False, False) ... 0.898804677617 -0.438349348687
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_50_2.csv', 'breslow', False, True) ... 0.898804677617 -0.438349348687
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_50_2.csv', 'breslow', True, False) ... 0.898804677617 -0.438349348687
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_50_2.csv', 'breslow', True, True) ... 0.898804677617 -0.438349348687
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_50_2.csv', 'efron', False, False) ... 0.90205759612 -0.431615677753
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_50_2.csv', 'efron', False, True) ... 0.90205759612 -0.431615677753
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_50_2.csv', 'efron', True, False) ... 0.90205759612 -0.431615677753
ERROR
statsmodels.duration.tests.test_phreg.TestPHReg.test_r('survival_data_50_2.csv', 'efron', True, True) ... 0.90205759612 -0.431615677753
ERROR
```

The two numbers on each line are the maximum and minimum eigenvalue.  The minimum should be > 0 if
- The function is at an optimim
- The hessian estimate is accurate

as reference: above test failures are produced by

```
      _, v = np.linalg.eig(-1 * self.hessian(xopt))
                if np.min(v) < 0.0:
                    print(np.max(v), np.min(v))
                    raise RuntimeError
```

https://github.com/statsmodels/statsmodels/pull/3084#issuecomment-230406453

Given the large positive and negative eigenvalues, this is not just a numerical precision problem, but seems to be more serious convergence problems.

My suggestion is to add a Warning in 0.9 and then investigate some cases.
As solution for users, I would eventually set invalid hessians to nans or None similar to the case when linalg.inv fails. The users or developers can always call the hessian method again to compute it for further investigation. But we don't want to raise an exception with convergence failures so users and we can investigate where and why it gets stuck.

Two special cases:
- IIRC, I have seen a messed up hessian for extra parameters while the linear part parameters had a proper positive definite submatrix, e.g. NegativeBinomial (IIRC).
- In some cases, we have singular hessian by design. With a numerical precision problem this might result in slightly negative eigenvalues of the hessian, e.g. -1e-15. I don't remember seeing a case with small negative, but I think that's likely to show up. 
  #1717   #146 (has example with -1.9e-15)

Chad had problems in statespace models with the hessian and added OPG (outer product of gradient) for the cov_params which is by definition positive (semi) definite.

Related aside (I don't find an issue that mentions it): Stata uses in some cases the rank of cov_params instead of the rank of exog.

Don't you mean like below?  In that case, I get no errors.

```
w, _ = np.linalg.eig(-1 * hmat)
if np.min(w) < 0.0:
    print(np.max(w), np.min(w))
    raise RuntimeError
```

Yes, indeed.  I was checking wrong output.  Face palm is the correct expression I think. 

We might still want to include the check, but I think a failed Cholesky factorization is a more efficient way to go (compared to calling eig).

I think we could replace inv with an svd or eig to be able to calculate and inv and get the information about eigenvalues/rank at the same time.

This would allow us to get the rank and work towards more consistency in using pinv versus inv for cov_params.  (possibly with keyword like singular_ok = True, which is used in some cases in Stata, IIRC with different use cases.)

Related: several models calculate the cov_params in fit directly instead of doing it in base fit.


Resolving patch:
diff --git a/statsmodels/base/model.py b/statsmodels/base/model.py
--- a/statsmodels/base/model.py
+++ b/statsmodels/base/model.py
@@ -9,7 +9,8 @@
 from statsmodels.tools.decorators import resettable_cache, cache_readonly
 import statsmodels.base.wrapper as wrap
 from statsmodels.tools.numdiff import approx_fprime
-from statsmodels.tools.sm_exceptions import ValueWarning
+from statsmodels.tools.sm_exceptions import ValueWarning, \
+    HessianInversionWarning
 from statsmodels.formula import handle_formula_data
 from statsmodels.compat.numpy import np_matrix_rank
 from statsmodels.base.optimizer import Optimizer
@@ -456,14 +457,20 @@ def fit(self, start_params=None, method='newton', maxiter=100,
         elif method == 'newton' and full_output:
             Hinv = np.linalg.inv(-retvals['Hessian']) / nobs
         elif not skip_hessian:
-            try:
-                Hinv = np.linalg.inv(-1 * self.hessian(xopt))
-            except:
-                #might want custom warning ResultsWarning? NumericalWarning?
+            H = -1 * self.hessian(xopt)
+            invertible = False
+            if np.all(np.isfinite(H)):
+                eigvals, eigvecs = np.linalg.eigh(H)
+                if np.min(eigvals) > 0:
+                    invertible = True
+
+            if invertible:
+                Hinv = eigvecs.dot(np.diag(1.0 / eigvals)).dot(eigvecs.T)
+                Hinv = np.asfortranarray((Hinv + Hinv.T) / 2.0)
+            else:
                 from warnings import warn
-                warndoc = ('Inverting hessian failed, no bse or '
-                           'cov_params available')
-                warn(warndoc, RuntimeWarning)
+                warn('Inverting hessian failed, no bse or cov_params '
+                     'available', HessianInversionWarning)
                 Hinv = None
 
         if 'cov_type' in kwargs:
@@ -591,7 +598,6 @@ def initialize(self):
         #Initialize is called by
         #statsmodels.model.LikelihoodModel.__init__
         #and should contain any preprocessing that needs to be done for a model
-        from statsmodels.tools import tools
         if self.exog is not None:
             # assume constant
             er = np_matrix_rank(self.exog)
diff --git a/statsmodels/tools/sm_exceptions.py b/statsmodels/tools/sm_exceptions.py
--- a/statsmodels/tools/sm_exceptions.py
+++ b/statsmodels/tools/sm_exceptions.py
@@ -111,3 +111,7 @@ class PrecisionWarning(UserWarning):
 
 class SpecificationWarning(UserWarning):
     pass
+
+
+class HessianInversionWarning(UserWarning):
+    pass
\ No newline at end of file


Test patch: