Issue 1:
Found when trying to build the wheels for 0.19.2:

________________________ test_non_meta_estimators[969] _________________________
self = <sklearn.utils.testing._named_check object at 0x1091aaa58>
args = ('GaussianProcess', GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
nugget=2.220446049...state=None, regr='constant',
storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
verbose=False))
kwargs = {}
def __call__(self, *args, **kwargs):
> return self.check(*args, **kwargs)
args = ('GaussianProcess', GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
nugget=2.220446049...state=None, regr='constant',
storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
verbose=False))
kwargs = {}
self = <sklearn.utils.testing._named_check object at 0x1091aaa58>
../venv/lib/python3.4/site-packages/sklearn/utils/testing.py:775:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../venv/lib/python3.4/site-packages/sklearn/utils/testing.py:291: in wrapper
return fn(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
name = 'GaussianProcess'
estimator_orig = GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
nugget=2.2204460492503131e-15, optimiz..._state=None, regr='constant',
storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
verbose=False)
@ignore_warnings(category=(DeprecationWarning, FutureWarning))
def check_supervised_y_2d(name, estimator_orig):
if "MultiTask" in name:
# These only work on 2d, so this test makes no sense
return
rnd = np.random.RandomState(0)
X = rnd.uniform(size=(10, 3))
y = np.arange(10) % 3
estimator = clone(estimator_orig)
set_random_state(estimator)
# fit
estimator.fit(X, y)
y_pred = estimator.predict(X)

set_random_state(estimator)
# Check that when a 2D y is given, a DataConversionWarning is
# raised
with warnings.catch_warnings(record=True) as w:
warnings.simplefilter("always", DataConversionWarning)
warnings.simplefilter("ignore", RuntimeWarning)
estimator.fit(X, y[:, np.newaxis])
y_pred_2d = estimator.predict(X)
msg = "expected 1 DataConversionWarning, got: %s" % (
", ".join([str(w_x) for w_x in w]))
if name not in MULTI_OUTPUT:
# check that we warned if we don't support multi-output
assert_greater(len(w), 0, msg)
assert_true("DataConversionWarning('A column-vector y"
" was passed when a 1d array was expected" in msg)
> assert_allclose(y_pred.ravel(), y_pred_2d.ravel())
E AssertionError:
E Not equal to tolerance rtol=1e-07, atol=0
E
E (mismatch 40.0%)
E x: array([ -1.086908e-13, 1.000000e+00, 2.000000e+00, 1.155742e-13,
E 1.000000e+00, 2.000000e+00, 5.062617e-14, 1.000000e+00,
E 2.000000e+00, 9.325873e-15])
E y: array([ -1.089129e-13, 1.000000e+00, 2.000000e+00, 1.272316e-13,
E 1.000000e+00, 2.000000e+00, 5.051515e-14, 1.000000e+00,
E 2.000000e+00, 9.214851e-15])
X = array([[ 0.5488135 , 0.71518937, 0.60276338],
[ 0.54488318, 0.4236548 , 0.64589411],
[ 0.43758721, ...6147936, 0.78052918],
[ 0.11827443, 0.63992102, 0.14335329],
[ 0.94466892, 0.52184832, 0.41466194]])
estimator = GaussianProcess(beta0=None,
corr=<function squared_exponential at 0x1063c0730>, normalize=True,
nugget...constant at 0x1063c02f0>, storage_mode='full',
theta0=array([[ 0.1]]), thetaL=None, thetaU=None, verbose=False)
estimator_orig = GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
nugget=2.2204460492503131e-15, optimiz..._state=None, regr='constant',
storage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
verbose=False)
msg = 'expected 1 DataConversionWarning, got: '
name = 'GaussianProcess'
rnd = <mtrand.RandomState object at 0x10ab87e58>
w = []
y = array([0, 1, 2, 0, 1, 2, 0, 1, 2, 0])
y_pred = array([ -1.08690834e-13, 1.00000000e+00, 2.00000000e+00,
1.15574217e-13, 1.00000000e+00, 2.00000000e+00,
5.06261699e-14, 1.00000000e+00, 2.00000000e+00,
9.32587341e-15])
y_pred_2d = array([[ -1.08912879e-13],
[ 1.00000000e+00],
[ 2.00000000e+00],
[ 1.27231559e-13],
[ ...0000e+00],
[ 5.05151476e-14],
[ 1.00000000e+00],
[ 2.00000000e+00],
[ 9.21485110e-15]])
../venv/lib/python3.4/site-packages/sklearn/utils/estimator_checks.py:1226: AssertionError

Details:

https://travis-ci.org/MacPython/scikit-learn-wheels/builds/400338565

we have not changed anything between 0.19.1 and 0.19.2 for this model so it's probably a change on the travis environment that caused this issue to appear now.

It would be great if someone with a mac could try to reproduce those failures both on master and 0.19.X branches.

Issue 2:
In a number of places in tests, numpy.testing.assert_allclose is used with default absolute tolerance parameter which is atol=0.

This means in particular that np.testing.assert_allclose(0, 1e-16) will fail. More context for the reasons behind this choice can be found in numpy/numpy#3183 (comment) and PEP485, which can be summed up with,

If, for a given use case, a user needs to compare to zero, the test will be guaranteed to fail the first time, and the user can select an appropriate value.

The issue is that occasionally, the tests will pass, and but then may fail on some other platform.

For instance, this test in estimator_checks.py passes CI on master, but then randomly fail for osx: [1], [2] correspond to the same commit, one for the PR, one in the forked repo (this test fails with the bellow message and not necessary for the same Python versions) ,

```
________________________ test_non_meta_estimators[1045] ________________________
args = ('GaussianProcess', GaussianProcess(beta0=None, corr='squared_exponential', normalize=True,
...orage_mode='full', theta0=0.1, thetaL=None, thetaU=None,
verbose=False))
[...]
> assert_allclose(y_pred.ravel(), y_pred_2d.ravel())
E AssertionError:
E Not equal to tolerance rtol=1e-07, atol=0
E
E (mismatch 40.0%)
E x: array([ -1.089129e-13, 1.000000e+00, 2.000000e+00, 1.272316e-13,
E 1.000000e+00, 2.000000e+00, 5.051515e-14, 1.000000e+00,
E 2.000000e+00, 9.214851e-15])
E y: array([ -1.086908e-13, 1.000000e+00, 2.000000e+00, 1.155742e-13,
E 1.000000e+00, 2.000000e+00, 5.062617e-14, 1.000000e+00,
E 2.000000e+00, 9.325873e-15])
```

this test in test_gradient_boosting_loss_functions.py compares floats to 0 with atol=0 but doesn't fail.

#10561 is another manifestation of this that fails on ppc64le

When atol is used, it not very consistent.

As to sklearn.utils.testing.assert_allclose_dense_sparse it has by default atol=1e-9 and not 0.

While the necessary absolute tolerance is test dependent, it might still be useful to

have a default value (e.g. 1e-9) when it's needed (e.g. DEFAULT_ATOL in sklearn.utils.testing) , except for the cases when it has to be increased for specific reasons.

use atol when it's definitely reasonable to do so (e.g. in probability equalities)

make sklearn.utils.testing.assert_allclose_dense_sparse and sklearn.utils.testing.assert_allclose have the same default atol.

Check that we don't have floating point equalities with 0.0 even if CI tests passes ([1], [2], [3] ..)

This might help improving the numerical stability of tests and prevent some of the tests failures on less common platforms cc @yarikoptic

What do you think?