Repository: statsmodels/statsmodels

Issue:
Maybe tolerance error for test_zero_constrained, macOS
On wheel testing:

```
=================================== FAILURES ===================================
______________ TestGenericNegativeBinomial.test_zero_constrained _______________
../venv/lib/python2.7/site-packages/statsmodels/base/tests/test_generic_methods.py:220: in test_zero_constrained
    assert_allclose(res1.resid, res2.resid, rtol=1e-10)
E   AssertionError: 
E   Not equal to tolerance rtol=1e-10, atol=0
E   
E   (mismatch 0.302129767211%)
E    x: array([-3.174188, -1.174188, -3.174188, ...,  5.713472,  5.686109,
E           3.730759])
E    y: array([-3.174188, -1.174188, -3.174188, ...,  5.713472,  5.686109,
E           3.730759])
```

Example:

https://travis-ci.org/MacPython/statsmodels-wheels/jobs/426380464

Build matrix:

https://travis-ci.org/MacPython/statsmodels-wheels/builds/426380455




Discussion:
Is there a common feature in the environment for the 4 failing python environments?

rtol is strict and imposes convergence to very similar parameters in 2 equivalent, numerically almost identical models. 
The failing assert is followed by two more asserts in the test method. It is likely that some of the other ones will also fail if we increase to eg. rtol=1e-9

I'm looking at the example on my computer with --pdb, In my case the results are identical, i.e. rtol=0 still passes. Aside: optimizer in this Negbin case is bfgs.
The resid closest to zero:
(Pdb) np.min(np.abs(res2.resid))
0.00038779800106381046

so we could try to add `atol=1e-12` and try to keep rtol=1e-10
These are all on macOS instead of Linux (or Windows).  So, maybe different libm, or different OpenBLAS routines?  
xref #2069
I did a PR with you suggested tolerance change.  It's not super-easy to test whether this helps until it's merged, making #2069 even more pertinent.
> I did a PR with you suggested tolerance change. It's not super-easy to test whether this helps until it's merged, making #2069 even more pertinent.

Maybe there should be a single OSX build on Travis. Latest packages?  IME OSX builds aren't so slow anymore.
One OSX for PRs would be very useful.
Trying to fix precision problems that might need several iterations by going through master is not very efficient (and makes for a messy git/github history)
I merged #5164 hoping the atol works

this issue can be closed when it works.
Sadly not fixed - see:

* https://travis-ci.org/MacPython/statsmodels-wheels/jobs/426763186
* https://travis-ci.org/MacPython/statsmodels-wheels/jobs/426763187

I'm not sure if the tolerance change helped, Python 2.7 and 3.7 don't have this failure on this run, but before the fix, Python 3.6 didn't have the failure, so it seems to have some random component.
not good, both cases have the same result in the convergence print, also it passes the comparison for params at rtol=1e-10 earlier in the unit test.
I have no guess which operation might have sufficient random noise to cause this.

To hopefully get rid of this we can increase rtol to `rtol=1e-8` which is still a tight result in terms of the statistics, but it lowers the bar that two essentially identical ways to compute agree at high precision.
Given that I'm not a OSX user, I have no clue what might be the problem. prediction and resid should be essentially `exp` and `dot`.
> prediction and resid should be essentially `exp` and `dot`.

Do we know more generally which operations are subject to this particular kind of noise?  e.g. if there were some narrow subset of `scipy` functions that were vulnerable, we could track them down and look for workarounds.

> To hopefully get rid of this we can increase rtol to rtol=1e-8

Ideally (well, 2nd-best or 3rd-best really) we'd use the tightest valid tolerance on each platform/config.  Might be a use case for the e.g. `DECIMAL_8 = 8` pattern in many test modules.

DECIMAL_x is outdated, it was before `assert_allclose` now it's rtol and atol (but we don't have that platform dependent)

Testing replicability is not really the point of the test, convergence tolerance would be only between 1e-5 and 1e-8. The rest is "bonus", i.e. irrelevant for checking that I/we don't have a bug. (Although it's nice to see that I do get identical results on Windows with MKL.)

I leave it to OSX users to figure out where their random noise comes from. Numerical noise is a moving target that might change with changes in implementation details, and possible bugs and bug fixes in underlying libraries, 
The problem is that for example some extra tiny numerical noise could put bfgs on a slightly different convergence path so at the end the accumulated error is 1e-11 for params which could imply 1e-8 or 1e-9 for prediction. It's still a correct answer because it is still easily within the required convergence tolerance of bfgs.
So neither we nor the underlying libraries might have a bug, but we don't get the "bonus" numerical replicability.

(aside: the failure is not important enough that I go through all my zero_constrained code again to check where some extra numerical noise might enter.)

Now that I've looked back through my notes, I'm inclined to think there may be other issues in the pertinent code that should be addressed before looking at tolerances: https://github.com/statsmodels/statsmodels/pull/4414#issuecomment-377856723
4414 is for the mixedlm test failure on OSX, not for zero_constrained.
(numerical derivative hessian are often sensitive to step length choice. The only more reliable way in general would be to use an adaptive method as in numdifftools, but that is much slower)
Not reproducible locally on OSX using pip installed numpy, scipy, pandas. 
@bashtage - from the above, my prediction would be that you will see the failure about half the time.  I guess it's conceivable that it's specific to (all) Python.org Pythons, but that does seem unlikely.
Can't reproduce after multiple runs on Homebrew 3.7 + pip everything else. Might be CPU-specific.
Might be - OpenBLAS can select different kernels at run-time.  What do you get for:

```
export OPENBLAS_VERBOSE=2
python -c 'import statsmodels.api as sapi'
```

?
Core: Sandybridge
The errors are seemingly random across the python versions.  `max(abs(resid1-resid2))` produced:
```
8.37019342725e-12
4.22950563461e-11
4.548006415916461e-11
7.99058597067e-11
```

Blas problem?
Quite a few test runs indicate that atol=1e-10 is safe to use. 
I think it's mostly propagation of floating point noise. There is no guarantee that floating point noise is deterministic, i.e. the same in repeated calls or across machines.

I will change the unit test to an assert for linear prediction which removes the exp transformation and might be numerically more stable.
Another check to see whether the param estimate itself is off, is to tighten the tolerance there.  (but again, bfgs is not required to provide that precision for params)

```
-> assert_allclose(res1.resid+1, res2.resid, rtol=1e-10, atol=1e-12)
(Pdb) res1.predict().max()
33.157131693223015
(Pdb) res1.fittedvalues.max()
3.5012578275277404
(Pdb) np.exp(res1.fittedvalues.max())
33.157131693223015
(Pdb) np.exp(res2.fittedvalues.max())
33.157131693223015
(Pdb) np.exp(res2.fittedvalues.max()+1e-11)
33.157131693554582
(Pdb) np.exp(res2.fittedvalues.max()) - np.exp(res2.fittedvalues.max()+1e-11)
-3.3156766221509315e-10
```
("You can't comment at this time."  github is still not back to normal)
The param estimates are almost always different on OSX by about 1e-13.  On Windows they are always exactly the same.  I saw one run where the parameters have 0 difference on OSX. 
> The param estimates are almost always different on OSX by about 1e-13.

Thanks, that's most likely enough to explain the difference in resid.
I will open a PR tonight base on this.  
I ran with atol=1e-10 and all passed.
**edit** IGNORE, the following is for `test_zero_collinear` and not for `test_zero_constrained` which is similar but not the reported failing test

I'm running into this now in #5255 but where convergence tolerance limits the precision which is much lower than here. (resid wasn't tested for GLM and discrete because it is currently not available.)

The most likely source for the difference in converged params is that GLM unit test doesn't use the same start_params in the two models. In that case we get only rtol=5e-5 or atol=1e-5 for resid, and a bit better for fittedvalues.
The unit test for GLM don't get the information about start_params which is missing in the results instance.
RLM and QuantileRegression do not allow start_params as fit arguments yet.

So, I will weaken the tolerance requirement in the unit test all the way to the one limited by convergence tolerance criteria.

**update**
All tests pass at current precision after changing GLM unit test to use the same start_params.
~~GEE test pass at current precision, AFAICS, because both models use default start_params, zeros,~~ GEE is skipped in the 
RLM is included and uses default start_params in both models

Resolving patch:
diff --git a/statsmodels/compat/platform.py b/statsmodels/compat/platform.py
new file mode 100644
--- /dev/null
+++ b/statsmodels/compat/platform.py
@@ -0,0 +1,5 @@
+import sys
+
+__all__ = ['PLATFORM_OSX']
+
+PLATFORM_OSX = sys.platform == 'darwin'


Test patch:
diff --git a/statsmodels/base/tests/test_generic_methods.py b/statsmodels/base/tests/test_generic_methods.py
--- a/statsmodels/base/tests/test_generic_methods.py
+++ b/statsmodels/base/tests/test_generic_methods.py
@@ -12,6 +12,7 @@
 """
 from statsmodels.compat.pandas import assert_series_equal, assert_index_equal
 from statsmodels.compat.python import range
+from statsmodels.compat.platform import PLATFORM_OSX
 
 import numpy as np
 import pandas as pd
@@ -112,7 +113,13 @@ def test_zero_constrained(self):
         if hasattr(res1, 'resid'):
             # discrete models, Logit don't have `resid` yet
             # atol discussion at gh-5158
-            assert_allclose(res1.resid, res2.resid, rtol=1e-10, atol=1e-12)
+            rtol = 1e-10
+            atol = 1e-12
+            if PLATFORM_OSX:
+                # GH 5628
+                rtol = 1e-8
+                atol = 1e-10
+            assert_allclose(res1.resid, res2.resid, rtol=rtol, atol=atol)
 
         ex = self.results.model.exog.mean(0)
         predicted1 = res1.predict(ex, **self.predict_kwds)
diff --git a/statsmodels/regression/tests/test_lme.py b/statsmodels/regression/tests/test_lme.py
--- a/statsmodels/regression/tests/test_lme.py
+++ b/statsmodels/regression/tests/test_lme.py
@@ -1,3 +1,5 @@
+from statsmodels.compat.platform import PLATFORM_OSX
+
 import os
 import csv
 import warnings
@@ -86,6 +88,8 @@ def f(x):
 class TestMixedLM(object):
 
     # Test analytic scores and Hessian using numeric differentiation
+    @pytest.mark.skipif(PLATFORM_OSX, reason='known to fail on OSX due to '
+                                             'numerical differences')
     @pytest.mark.slow
     @pytest.mark.parametrize("use_sqrt", [False, True])
     @pytest.mark.parametrize("reml", [False, True])