Repository: scikit-learn/scikit-learn
Pull request: TST fix unstable test_newrand_set_seed
Fixes #25937.

This test would not break on the CI because the ordering is fixed. On my laptop where pytest run tests in non-deterministic order for some reason, this would fail randomly.

Problem statement:
Unstable sklearn/svm/tests/test_bounds.py::test_newrand_set_seed
On my local laptop I often observe a random failure for `test_newrand_set_seed`.

I am running macos m1 with Python 3.11 from conda-forge. scikit-learn has been built with the clang compilers from conda-forge.

Here is a typical run with 10 repetitions using `pytest-repeat`:

<details>

```python-traceback
pytest -v -k "test_newrand_set_seed[None" sklearn/svm/tests/test_bounds.py --count 10
================================================================================ test session starts =================================================================================
platform darwin -- Python 3.11.0, pytest-7.2.0, pluggy-1.0.0 -- /Users/ogrisel/mambaforge/envs/dev/bin/python3.11
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/ogrisel/code/scikit-learn/.hypothesis/examples')
Using --randomly-seed=1301508857
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/ogrisel/code/scikit-learn, configfile: setup.cfg
plugins: snapshot-0.9.0, xdist-3.1.0, mock-3.10.0, clarity-1.0.1, hypothesis-6.62.0, randomly-3.12.0, profiling-1.7.0, cov-4.0.0, repeat-0.9.1, anyio-3.6.2, benchmark-4.0.0
collected 260 items / 250 deselected / 10 selected                                                                                                                                   

sklearn/svm/tests/test_bounds.py::test_newrand_set_seed[None-81-5-10] PASSED                                                                                                   [ 10%]
sklearn/svm/tests/test_bounds.py::test_newrand_set_seed[None-81-4-10] FAILED                                                                                                   [ 20%]
sklearn/svm/tests/test_bounds.py::test_newrand_set_seed[None-81-7-10] FAILED                                                                                                   [ 30%]
sklearn/svm/tests/test_bounds.py::test_newrand_set_seed[None-81-1-10] FAILED                                                                                                   [ 40%]
sklearn/svm/tests/test_bounds.py::test_newrand_set_seed[None-81-9-10] FAILED                                                                                                   [ 50%]
sklearn/svm/tests/test_bounds.py::test_newrand_set_seed[None-81-8-10] FAILED                                                                                                   [ 60%]
sklearn/svm/tests/test_bounds.py::test_newrand_set_seed[None-81-10-10] FAILED                                                                                                  [ 70%]
sklearn/svm/tests/test_bounds.py::test_newrand_set_seed[None-81-3-10] FAILED                                                                                                   [ 80%]
sklearn/svm/tests/test_bounds.py::test_newrand_set_seed[None-81-6-10] FAILED                                                                                                   [ 90%]
sklearn/svm/tests/test_bounds.py::test_newrand_set_seed[None-81-2-10] FAILED                                                                                                   [100%]

====================================================================================== FAILURES ======================================================================================
________________________________________________________________________ test_newrand_set_seed[None-81-4-10] _________________________________________________________________________

seed = None, val = 81

    @pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
    def test_newrand_set_seed(seed, val):
        """Test that `set_seed` produces deterministic results"""
        if seed is not None:
            set_seed_wrap(seed)
        x = bounded_rand_int_wrap(100)
>       assert x == val, f"Expected {val} but got {x} instead"
E       AssertionError: Expected 81 but got 13 instead
E       assert 13 == 81

sklearn/svm/tests/test_bounds.py:81: AssertionError
________________________________________________________________________ test_newrand_set_seed[None-81-7-10] _________________________________________________________________________

seed = None, val = 81

    @pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
    def test_newrand_set_seed(seed, val):
        """Test that `set_seed` produces deterministic results"""
        if seed is not None:
            set_seed_wrap(seed)
        x = bounded_rand_int_wrap(100)
>       assert x == val, f"Expected {val} but got {x} instead"
E       AssertionError: Expected 81 but got 90 instead
E       assert 90 == 81

sklearn/svm/tests/test_bounds.py:81: AssertionError
________________________________________________________________________ test_newrand_set_seed[None-81-1-10] _________________________________________________________________________

seed = None, val = 81

    @pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
    def test_newrand_set_seed(seed, val):
        """Test that `set_seed` produces deterministic results"""
        if seed is not None:
            set_seed_wrap(seed)
        x = bounded_rand_int_wrap(100)
>       assert x == val, f"Expected {val} but got {x} instead"
E       AssertionError: Expected 81 but got 83 instead
E       assert 83 == 81

sklearn/svm/tests/test_bounds.py:81: AssertionError
________________________________________________________________________ test_newrand_set_seed[None-81-9-10] _________________________________________________________________________

seed = None, val = 81

    @pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
    def test_newrand_set_seed(seed, val):
        """Test that `set_seed` produces deterministic results"""
        if seed is not None:
            set_seed_wrap(seed)
        x = bounded_rand_int_wrap(100)
>       assert x == val, f"Expected {val} but got {x} instead"
E       AssertionError: Expected 81 but got 12 instead
E       assert 12 == 81

sklearn/svm/tests/test_bounds.py:81: AssertionError
________________________________________________________________________ test_newrand_set_seed[None-81-8-10] _________________________________________________________________________

seed = None, val = 81

    @pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
    def test_newrand_set_seed(seed, val):
        """Test that `set_seed` produces deterministic results"""
        if seed is not None:
            set_seed_wrap(seed)
        x = bounded_rand_int_wrap(100)
>       assert x == val, f"Expected {val} but got {x} instead"
E       AssertionError: Expected 81 but got 96 instead
E       assert 96 == 81

sklearn/svm/tests/test_bounds.py:81: AssertionError
________________________________________________________________________ test_newrand_set_seed[None-81-10-10] ________________________________________________________________________

seed = None, val = 81

    @pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
    def test_newrand_set_seed(seed, val):
        """Test that `set_seed` produces deterministic results"""
        if seed is not None:
            set_seed_wrap(seed)
        x = bounded_rand_int_wrap(100)
>       assert x == val, f"Expected {val} but got {x} instead"
E       AssertionError: Expected 81 but got 91 instead
E       assert 91 == 81

sklearn/svm/tests/test_bounds.py:81: AssertionError
________________________________________________________________________ test_newrand_set_seed[None-81-3-10] _________________________________________________________________________

seed = None, val = 81

    @pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
    def test_newrand_set_seed(seed, val):
        """Test that `set_seed` produces deterministic results"""
        if seed is not None:
            set_seed_wrap(seed)
        x = bounded_rand_int_wrap(100)
>       assert x == val, f"Expected {val} but got {x} instead"
E       AssertionError: Expected 81 but got 22 instead
E       assert 22 == 81

sklearn/svm/tests/test_bounds.py:81: AssertionError
________________________________________________________________________ test_newrand_set_seed[None-81-6-10] _________________________________________________________________________

seed = None, val = 81

    @pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
    def test_newrand_set_seed(seed, val):
        """Test that `set_seed` produces deterministic results"""
        if seed is not None:
            set_seed_wrap(seed)
        x = bounded_rand_int_wrap(100)
>       assert x == val, f"Expected {val} but got {x} instead"
E       AssertionError: Expected 81 but got 63 instead
E       assert 63 == 81

sklearn/svm/tests/test_bounds.py:81: AssertionError
________________________________________________________________________ test_newrand_set_seed[None-81-2-10] _________________________________________________________________________

seed = None, val = 81

    @pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
    def test_newrand_set_seed(seed, val):
        """Test that `set_seed` produces deterministic results"""
        if seed is not None:
            set_seed_wrap(seed)
        x = bounded_rand_int_wrap(100)
>       assert x == val, f"Expected {val} but got {x} instead"
E       AssertionError: Expected 81 but got 30 instead
E       assert 30 == 81

sklearn/svm/tests/test_bounds.py:81: AssertionError
==================================================================== 9 failed, 1 passed, 250 deselected in 0.28s =====================================================================
```

</details>

I do not understand the purpose of the test though: this test check for a deterministic result without seeding the "newrand.h" RNG. Why doesn't it fail on our CI? Can someone else confirm the failure or lack of failure of this test on their machine?

Hints from issue discussion:
I think I understand the problem: the first time the rng is used the result is 81, but then its state is mutated so the outcome is different.

On my laptop, the ordering of the pytest test in my Python 3.11 environment is not deterministic, so I trigger this often but not on the CI.

Let me do a PR to remove this test case: tests should not have depend on global mutable state and execution ordering.

Commit message: TST fix unstable test_newrand_set_seed

Patch to review:
diff --git a/sklearn/svm/tests/test_bounds.py b/sklearn/svm/tests/test_bounds.py
index 23d6be2f44e98..090959a5d1a83 100644
--- a/sklearn/svm/tests/test_bounds.py
+++ b/sklearn/svm/tests/test_bounds.py
@@ -72,13 +72,30 @@ def test_ill_posed_min_c():
 _MAX_UNSIGNED_INT = 4294967295
 
 
-@pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
-def test_newrand_set_seed(seed, val):
+def test_newrand_default():
+    """Test that bounded_rand_int_wrap without seeding respects the range
+
+    Note this test should pass either if executed alone, or in conjunctions
+    with other tests that call set_seed explicit in any order: it checks
+    invariants on the RNG instead of specific values.
+    """
+    previous = None
+    constant = True
+    for _ in range(10):
+        generated = bounded_rand_int_wrap(100)
+        assert 0 <= generated < 100
+        if previous is not None:
+            constant = constant and (generated == previous)
+        previous = generated
+    assert not constant
+
+
+@pytest.mark.parametrize("seed, expected", [(0, 54), (_MAX_UNSIGNED_INT, 9)])
+def test_newrand_set_seed(seed, expected):
     """Test that `set_seed` produces deterministic results"""
-    if seed is not None:
-        set_seed_wrap(seed)
-    x = bounded_rand_int_wrap(100)
-    assert x == val, f"Expected {val} but got {x} instead"
+    set_seed_wrap(seed)
+    generated = bounded_rand_int_wrap(100)
+    assert generated == expected
 
 
 @pytest.mark.parametrize("seed", [-1, _MAX_UNSIGNED_INT + 1])
@@ -91,6 +108,9 @@ def test_newrand_set_seed_overflow(seed):
 @pytest.mark.parametrize("range_, n_pts", [(_MAX_UNSIGNED_INT, 10000), (100, 25)])
 def test_newrand_bounded_rand_int(range_, n_pts):
     """Test that `bounded_rand_int` follows a uniform distribution"""
+    # XXX: this test is very seed sensitive: either it is wrong (too strict?)
+    # or the wrapped RNG is not uniform enough, at least on some platforms.
+    set_seed_wrap(42)
     n_iter = 100
     ks_pvals = []
     uniform_dist = stats.uniform(loc=0, scale=range_)


Code review comments:
diff_hunk: @@ -72,13 +72,30 @@ def test_ill_posed_min_c():
 _MAX_UNSIGNED_INT = 4294967295
 
 
-@pytest.mark.parametrize("seed, val", [(None, 81), (0, 54), (_MAX_UNSIGNED_INT, 9)])
-def test_newrand_set_seed(seed, val):
+def test_newrand_default():
+    """Test that bounded_rand_int_wrap without seeding respects the range
+
+    Note this test should pass either if executed alone, or in conjunctions
+    with other tests that call set_seed explicit in any order: it checks
+    invariants on the RNG instead of specific values.
+    """
+    previous = None
+    constant = True
+    for _ in range(10):
+        generated = bounded_rand_int_wrap(100)
+        assert 0 <= generated < 100
+        if previous is not None:
+            constant = constant and (generated == previous)
+        previous = generated
+    assert not constant
path: sklearn/svm/tests/test_bounds.py
text: @user1:
I think the following is more pythonic
```suggestion
    generated = [bounded_rand_int_wrap(100) for _ in range(10)]
    assert all(0 <= x < 100 for x in generated)
    assert not all(x == generated[0] for x in generated)
```