Repository: scipy/scipy

Issue:
BUG: `QMCEngine.reset()` semantics and passed `Generator` instances
### Describe your issue.

Ported from [another thread](https://github.com/scipy/scipy/pull/17955#discussion_r1122185322). I apologize for the length, but I was asked to explain in more detail.

The `QMCEngine`s use pseudorandomness through `Generator` in various ways for the purpose of scrambling. `QMCEngine`s also implement a `reset()` method that purports to reset the QMC sequence back to its initial state so that subsequent samples can be repeated. When a user chooses not to use scrambling or when a user specifies a static seed integer, this resetting works as I expect; you can get exactly the same samples repeatably.

```python
>>> q = qmc.Halton(d=2, scramble=False)

>>> q.random(4)
array([[0.        , 0.        ],
       [0.5       , 0.33333333],
       [0.25      , 0.66666667],
       [0.75      , 0.11111111]])

>>> q.reset()
<scipy.stats._qmc.Halton at 0x7f38ea51aeb0>

>>> q.random(4)
array([[0.        , 0.        ],
       [0.5       , 0.33333333],
       [0.25      , 0.66666667],
       [0.75      , 0.11111111]])

>>> q = qmc.Halton(d=2, seed=0xa29cabb11cfdf44ff6cac8bec254c2a0)

>>> q.random(4)
array([[0.92701897, 0.40341341],
       [0.42701897, 0.07008008],
       [0.67701897, 0.73674674],
       [0.17701897, 0.51452452]])

>>> q.reset()
<scipy.stats._qmc.Halton at 0x7f38e67a8250>

>>> q.random(4)
array([[0.92701897, 0.40341341],
       [0.42701897, 0.07008008],
       [0.67701897, 0.73674674],
       [0.17701897, 0.51452452]])
```

However, when the user passes an existing `Generator` instance as the seed or passes `None`, the resetting semantics work more variably. `QMCEngine` stores a deepcopy of the passed `seed` object and stores it in the `.rng_seed` attribute for later resetting. It will pass the original `seed` object through `np.random.default_rng()` (more or less) to coerce it to a `Generator` instance which it stores in the `.rng` attribute; `seed` objects that are already `Generator` instances are passed through without copying. When `.reset()` is called, the `.rng_seed` object is deepcopied again and passed through `default_rng()` to coerce it to a `Generator`.

When `seed=None`, `default_rng(None)` creates a fresh `Generator` with OS-provided entropy. Resetting the state no longer actually resets.

```python
>>> q = qmc.Halton(d=2, seed=None)

>>> q.random(4)
array([[0.86786616, 0.14418323],
       [0.36786616, 0.47751656],
       [0.61786616, 0.81084989],
       [0.11786616, 0.25529434]])

>>> q.reset()
<scipy.stats._qmc.Halton at 0x7f38e66cc7c0>

>>> q.random(4)
array([[0.42982941, 0.78171812],
       [0.92982941, 0.11505145],
       [0.17982941, 0.44838478],
       [0.67982941, 0.670607  ]])
```

Now, that might be  considered useful, as if the user is requesting that you start the QMC part of the sequence over, but do a new rescrambling because `seed=None`, but not all `QMCEngine`s work like that. `Sobol`, for instance, only uses the `Generator` during instantiation and not sampling. Resetting the `.rng` to be `default_rng(None)` does nothing, and the scrambling is not reapplied during the reset.

```python
>>> q = qmc.Sobol(d=2, seed=None)

>>> q.random(4)
array([[0.47767156, 0.93139656],
       [0.79002613, 0.37025878],
       [0.67699522, 0.67998868],
       [0.11413617, 0.11979645]])

>>> q.reset()
<scipy.stats._qmc.Sobol at 0x7f38e886ec70>

>>> q.random(4)
array([[0.47767156, 0.93139656],
       [0.79002613, 0.37025878],
       [0.67699522, 0.67998868],
       [0.11413617, 0.11979645]])
```

When a `Generator` instance is passed, we are in a bit of a correctness bind. While there are use cases for sampling the same segment of the PRNG cycle multiple times (this kind of resetting can be one of them), one has to be careful and do it under very controlled circumstances such that one is only doing it in exactly the place that is desired. The code that is doing that resetting needs to own the `Generator` entirely. If the semantics of `QMCEngine` wants to store the given `Generator` state and restore it upon `reset()`, then it cannot accept a passed `Generator` instance as-is. One could spawn a new `Generator` from it and only sample from the spawned `Generator` (and warn people against accessing it directly because only the `QMCEngine` owns it now).

```python
>>> rng = np.random.default_rng(0xa29cabb11cfdf44ff6cac8bec254c2a0)

>>> q = qmc.LatinHypercube(d=2, seed=rng)

# One continues to use the Generator somehow. This could even be something
# innocuous-looking like `qmc.Sobol(d=2, seed=rng)` which consumes some PRNG
# state during initialization.

>>> rng.random(4)
array([0.24573226, 0.85685711, 0.02918954, 0.96401299])

>>> q.random(4)
array([[0.88200533, 0.35298427],
       [0.55332016, 0.78001033],
       [0.05455107, 0.01202496],
       [0.3535495 , 0.7165475 ]])

>>> q.reset()
<scipy.stats._qmc.LatinHypercube at 0x7f38e66a0f40>

>>> q.random(4)
array([[0.68856694, 0.78578572],
       [0.24270262, 0.25899675],
       [0.38200533, 0.10298427],
       [0.80332016, 0.53001033]])
```

So `reset()` can't make good on its promised semantics in such cases. The reset's deepcopy of the passed-in `Generator` instance isn't a replacement for the thing that the `QMCEngine` used during the first `.random()` call. Parts of the PRNG cycle are reused, but not for the parts where we _wanted_ them to be used (i.e. the two `QMCEngine.random()` calls). The subsequent calls to `QMCEngine.random()` will correlate with the other outputs from the `Generator`, not the two supposed copies of the QMC sequence.

There are a couple of outs here, but it just depends on what the intended semantics of `QMCEngine.reset()` actually are. Given the variability between implementations and different scenarios for `seed=` values, I don't quite know what they are intended to be. Generally, if we want `reset()` to be a way to reproduce scrambled QMC sequences, then I think it needs to own its own `Generator` instance. When passed a `Generator` instance, it should spawn a child and store a deepcopy of it for later use in `reset()`. When passed `seed=None`, it should store a deepcopy of the instantiated `Generator` for the same purpose, not `None`. Copying the state of a `Generator` can really only be a safe and valid thing to do when you own the whole lifecycle of the `Generator`, from birth to death. If you are using a `Generator` that was passed in, you don't own it.

If the goal is just to reset the QMC parts of the state and reinitialize the scrambling "mostly as if" the user passed in the same `seed=` object again, then I would probably just avoid the deepcopy. This would get fresh `Generator`s each time when `seed=None` and would just continue to use the same `Generator` object when it was passed a `Generator` object. With a couple of fixups to recompute scrambling data for `Sobol` in the `reset()`, this would be actually be useful in cases like `qmc_quad()` where we want to repeat the same QMC setup, but with different scrambling in order to estimate the error. Right now, one just has to instantiate multiple `QMCEngine`s carefully. While I think this is a coherent set of semantics, it _is_ relatively complicated.

### Reproducing Code Example

```python
As above.
```


### Error message

```shell
As above.
```


### SciPy/NumPy/Python version and system information

```shell
1.8.0 1.21.2 sys.version_info(major=3, minor=8, micro=10, releaselevel='final', serial=0)
```



Discussion:
Thank you for the report.

> When passed a Generator instance, it should spawn a child and store a deepcopy of it for later use in reset(). When passed seed=None, it should store a deepcopy of the instantiated Generator for the same purpose, not None.

I think this is what we want. I will work on that and improve the tests.





Resolving patch:
diff --git a/scipy/integrate/_quadrature.py b/scipy/integrate/_quadrature.py
--- a/scipy/integrate/_quadrature.py
+++ b/scipy/integrate/_quadrature.py
@@ -1304,7 +1304,7 @@ def qmc_quad(func, a, b, *, n_estimates=8, n_points=1024, qrng=None,
     >>> n_estimates = 8
     >>> res = qmc_quad(func, a, b, n_estimates=n_estimates, qrng=qrng)
     >>> res.integral, res.standard_error
-    (0.00018443143881633162, 4.709434153066518e-08)
+    (0.00018429555666024108, 1.0389431116001344e-07)
 
     A two-sided, 99% confidence interval for the integral may be estimated
     as:
@@ -1312,7 +1312,7 @@ def qmc_quad(func, a, b, *, n_estimates=8, n_points=1024, qrng=None,
     >>> t = stats.t(df=n_estimates-1, loc=res.integral,
     ...             scale=res.standard_error)
     >>> t.interval(0.99)
-    (0.00018426663295474533, 0.0001845962446779179)
+    (0.0001839319802536469, 0.00018465913306683527)
 
     Indeed, the value reported by `scipy.stats.multivariate_normal` is
     within this range.
diff --git a/scipy/stats/_qmc.py b/scipy/stats/_qmc.py
--- a/scipy/stats/_qmc.py
+++ b/scipy/stats/_qmc.py
@@ -25,7 +25,7 @@
     )
 
 import scipy.stats as stats
-from scipy._lib._util import rng_integers
+from scipy._lib._util import rng_integers, _rng_spawn
 from scipy.spatial import distance, Voronoi
 from scipy.special import gammainc
 from ._sobol import (
@@ -748,8 +748,16 @@ def __init__(
             raise ValueError('d must be a non-negative integer value')
 
         self.d = d
-        self.rng = check_random_state(seed)
-        self.rng_seed = copy.deepcopy(seed)
+
+        if isinstance(seed, np.random.Generator):
+            # Spawn a Generator that we can own and reset.
+            self.rng = _rng_spawn(seed, 1)[0]
+        else:
+            # Create our instance of Generator, does not need spawning
+            # Also catch RandomState which cannot be spawned
+            self.rng = check_random_state(seed)
+        self.rng_seed = copy.deepcopy(self.rng)
+
         self.num_generated = 0
 
         config = {
diff --git a/scipy/stats/_sensitivity_analysis.py b/scipy/stats/_sensitivity_analysis.py
--- a/scipy/stats/_sensitivity_analysis.py
+++ b/scipy/stats/_sensitivity_analysis.py
@@ -467,9 +467,9 @@ def sobol_indices(
     ...     random_state=rng
     ... )
     >>> indices.first_order
-    array([3.14996175e-01, 4.40110556e-01, 2.17329918e-04])
+    array([0.31637954, 0.43781162, 0.00318825])
     >>> indices.total_order
-    array([0.55508078, 0.43995732, 0.23803014])
+    array([0.56122127, 0.44287857, 0.24229595])
 
     Confidence interval can be obtained using bootstrapping.
 


Test patch:
diff --git a/scipy/stats/tests/test_multivariate.py b/scipy/stats/tests/test_multivariate.py
--- a/scipy/stats/tests/test_multivariate.py
+++ b/scipy/stats/tests/test_multivariate.py
@@ -2468,7 +2468,7 @@ def test_cdf_against_qsimvtv(self, dim, seed, singular):
     def test_cdf_against_generic_integrators(self):
         # Compare result against generic numerical integrators
         dim = 3
-        rng = np.random.default_rng(413722918996573)
+        rng = np.random.default_rng(41372291899657)
         w = 10 ** rng.uniform(-1, 1, size=dim)
         cov = _random_covariance(dim, w, rng, singular=True)
         mean = rng.random(dim)
diff --git a/scipy/stats/tests/test_qmc.py b/scipy/stats/tests/test_qmc.py
--- a/scipy/stats/tests/test_qmc.py
+++ b/scipy/stats/tests/test_qmc.py
@@ -471,9 +471,11 @@ class QMCEngineTests:
     scramble = [True, False]
     ids = ["Scrambled", "Unscrambled"]
 
-    def engine(self, scramble: bool, seed=None, **kwargs) -> QMCEngine:
-        if seed is None:
-            seed = np.random.default_rng(170382760648021597650530316304495310428)
+    def engine(
+        self, scramble: bool,
+        seed=170382760648021597650530316304495310428,
+        **kwargs
+    ) -> QMCEngine:
         if self.can_scramble:
             return self.qmce(scramble=scramble, seed=seed, **kwargs)
         else:
@@ -533,8 +535,16 @@ def test_continuing(self, scramble):
         assert_allclose(sample, ref_sample[n_half:], atol=1e-1)
 
     @pytest.mark.parametrize("scramble", scramble, ids=ids)
-    def test_reset(self, scramble):
-        engine = self.engine(d=2, scramble=scramble)
+    @pytest.mark.parametrize(
+        "seed",
+        (
+            170382760648021597650530316304495310428,
+            np.random.default_rng(170382760648021597650530316304495310428),
+            None,
+        ),
+    )
+    def test_reset(self, scramble, seed):
+        engine = self.engine(d=2, scramble=scramble, seed=seed)
         ref_sample = engine.random(n=8)
 
         engine.reset()
@@ -912,9 +922,10 @@ def test_validations(self):
     def test_MultinomialBasicDraw(self):
         seed = np.random.default_rng(6955663962957011631562466584467607969)
         p = np.array([0.12, 0.26, 0.05, 0.35, 0.22])
-        expected = np.array([[13, 24, 6, 35, 22]])
-        engine = qmc.MultinomialQMC(p, n_trials=100, seed=seed)
-        assert_array_equal(engine.random(1), expected)
+        n_trials = 100
+        expected = np.atleast_2d(n_trials * p).astype(int)
+        engine = qmc.MultinomialQMC(p, n_trials=n_trials, seed=seed)
+        assert_allclose(engine.random(1), expected, atol=1)
 
     def test_MultinomialDistribution(self):
         seed = np.random.default_rng(77797854505813727292048130876699859000)
@@ -937,11 +948,12 @@ def test_other_engine(self):
         # same as test_MultinomialBasicDraw with different engine
         seed = np.random.default_rng(283753519042773243071753037669078065412)
         p = np.array([0.12, 0.26, 0.05, 0.35, 0.22])
-        expected = np.array([[12, 25, 5, 36, 22]])
+        n_trials = 100
+        expected = np.atleast_2d(n_trials * p).astype(int)
         base_engine = qmc.Sobol(1, scramble=True, seed=seed)
-        engine = qmc.MultinomialQMC(p, n_trials=100, engine=base_engine,
+        engine = qmc.MultinomialQMC(p, n_trials=n_trials, engine=base_engine,
                                     seed=seed)
-        assert_array_equal(engine.random(1), expected)
+        assert_allclose(engine.random(1), expected, atol=1)
 
 
 class TestNormalQMC:
@@ -981,8 +993,8 @@ def test_NormalQMCSeeded(self):
         engine = qmc.MultivariateNormalQMC(
             mean=np.zeros(2), inv_transform=False, seed=seed)
         samples = engine.random(n=2)
-        samples_expected = np.array([[0.446961, -1.243236],
-                                     [-0.230754, 0.21354]])
+        samples_expected = np.array([[-0.932001, -0.522923],
+                                     [-1.477655, 0.846851]])
         assert_allclose(samples, samples_expected, atol=1e-4)
 
         # test odd dimension
@@ -990,8 +1002,8 @@ def test_NormalQMCSeeded(self):
         engine = qmc.MultivariateNormalQMC(
             mean=np.zeros(3), inv_transform=False, seed=seed)
         samples = engine.random(n=2)
-        samples_expected = np.array([[0.446961, -1.243236, 0.324827],
-                                     [-0.997875, 0.399134, 1.032234]])
+        samples_expected = np.array([[-0.932001, -0.522923, 0.036578],
+                                     [-1.778011, 0.912428, -0.065421]])
         assert_allclose(samples, samples_expected, atol=1e-4)
 
         # same test with another engine
@@ -1002,8 +1014,8 @@ def test_NormalQMCSeeded(self):
             engine=base_engine, seed=seed
         )
         samples = engine.random(n=2)
-        samples_expected = np.array([[0.446961, -1.243236, 0.324827],
-                                     [-0.997875, 0.399134, 1.032234]])
+        samples_expected = np.array([[-0.932001, -0.522923, 0.036578],
+                                     [-1.778011, 0.912428, -0.065421]])
         assert_allclose(samples, samples_expected, atol=1e-4)
 
     def test_NormalQMCSeededInvTransform(self):
@@ -1012,8 +1024,8 @@ def test_NormalQMCSeededInvTransform(self):
         engine = qmc.MultivariateNormalQMC(
             mean=np.zeros(2), seed=seed, inv_transform=True)
         samples = engine.random(n=2)
-        samples_expected = np.array([[-0.804472, 0.384649],
-                                     [0.396424, -0.117676]])
+        samples_expected = np.array([[-0.913237, -0.964026],
+                                     [0.255904, 0.003068]])
         assert_allclose(samples, samples_expected, atol=1e-4)
 
         # test odd dimension
@@ -1021,8 +1033,8 @@ def test_NormalQMCSeededInvTransform(self):
         engine = qmc.MultivariateNormalQMC(
             mean=np.zeros(3), seed=seed, inv_transform=True)
         samples = engine.random(n=2)
-        samples_expected = np.array([[-0.804472, 0.384649, 1.583568],
-                                     [0.165333, -2.266828, -1.655572]])
+        samples_expected = np.array([[-0.913237, -0.964026, 0.355501],
+                                     [0.699261, 2.90213 , -0.6418]])
         assert_allclose(samples, samples_expected, atol=1e-4)
 
     def test_other_engine(self):
@@ -1049,7 +1061,7 @@ def test_NormalQMCShapiro(self):
         assert np.abs(cov[0, 1]) < 1e-2
 
     def test_NormalQMCShapiroInvTransform(self):
-        rng = np.random.default_rng(3234455)
+        rng = np.random.default_rng(32344554)
         engine = qmc.MultivariateNormalQMC(
             mean=np.zeros(2), inv_transform=True, seed=rng)
         samples = engine.random(n=256)
@@ -1157,8 +1169,8 @@ def test_MultivariateNormalQMCSeeded(self):
         engine = qmc.MultivariateNormalQMC(np.array([0, 0]), A,
                                            inv_transform=False, seed=rng)
         samples = engine.random(n=2)
-        samples_expected = np.array([[0.479575, 0.934723],
-                                     [1.712571, 0.172699]])
+        samples_expected = np.array([[-0.64419, -0.882413],
+                                     [0.837199, 2.045301]])
         assert_allclose(samples, samples_expected, atol=1e-4)
 
         # test odd dimension
@@ -1168,8 +1180,8 @@ def test_MultivariateNormalQMCSeeded(self):
         engine = qmc.MultivariateNormalQMC(np.array([0, 0, 0]), A,
                                            inv_transform=False, seed=rng)
         samples = engine.random(n=2)
-        samples_expected = np.array([[2.463393, 2.252826, -0.886809],
-                                     [1.252468, 0.029449, -1.126328]])
+        samples_expected = np.array([[-0.693853, -1.265338, -0.088024],
+                                     [1.620193, 2.679222, 0.457343]])
         assert_allclose(samples, samples_expected, atol=1e-4)
 
     def test_MultivariateNormalQMCSeededInvTransform(self):
@@ -1181,8 +1193,8 @@ def test_MultivariateNormalQMCSeededInvTransform(self):
             np.array([0, 0]), A, seed=rng, inv_transform=True
         )
         samples = engine.random(n=2)
-        samples_expected = np.array([[-3.095968, -0.566545],
-                                     [0.603154, 0.222434]])
+        samples_expected = np.array([[0.682171, -3.114233],
+                                     [-0.098463, 0.668069]])
         assert_allclose(samples, samples_expected, atol=1e-4)
 
         # test odd dimension
@@ -1193,13 +1205,13 @@ def test_MultivariateNormalQMCSeededInvTransform(self):
             np.array([0, 0, 0]), A, seed=rng, inv_transform=True
         )
         samples = engine.random(n=2)
-        samples_expected = np.array([[1.427248, -0.338187, -1.560687],
-                                     [-0.357026, 1.662937, -0.29769]])
+        samples_expected = np.array([[0.988061, -1.644089, -0.877035],
+                                     [-1.771731, 1.096988, 2.024744]])
         assert_allclose(samples, samples_expected, atol=1e-4)
 
     def test_MultivariateNormalQMCShapiro(self):
         # test the standard case
-        seed = np.random.default_rng(188960007281846377164494575845971645056)
+        seed = np.random.default_rng(188960007281846377164494575845971640)
         engine = qmc.MultivariateNormalQMC(
             mean=[0, 0], cov=[[1, 0], [0, 1]], seed=seed
         )
@@ -1266,7 +1278,7 @@ def test_MultivariateNormalQMCShapiroInvTransform(self):
 
     def test_MultivariateNormalQMCDegenerate(self):
         # X, Y iid standard Normal and Z = X + Y, random vector (X, Y, Z)
-        seed = np.random.default_rng(163206374175814483578698216542904486209)
+        seed = np.random.default_rng(16320637417581448357869821654290448620)
         engine = qmc.MultivariateNormalQMC(
             mean=[0.0, 0.0, 0.0],
             cov=[[1.0, 0.0, 1.0], [0.0, 1.0, 1.0], [1.0, 1.0, 2.0]],