Repository: dask/dask

Issue:
Unable to replicate random samples between dask and numpy
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:
When the same random seed is used for both dask and numpy, the output of random.normal() do not match each other. 

**What you expected to happen**:
I would have expected the output to match verbatim. When setting a dask random state, and then calling `<random_state>._numpy_state.normal()` the results will match those from numpy. However, this returns a np.ndarray() versus a dask array. I am not sure if this is the intended behavior. 

**Minimal Complete Verifiable Example**:
Using depreciated random.seed():
```python
import dask.array as da
da.random.seed(42)
da.random.normal(0, 1, size=10).compute()

array([-1.31918189,  0.11348963,  1.2427626 , -1.26858024,  2.27840845,
        0.57524507, -1.14656226,  0.90500015, -1.17130718, -0.89184311])

import numpy as np
np.random.seed(42)
np.random.normal(0, 1 ,size=10)

array([ 0.49671415, -0.1382643 ,  0.64768854,  1.52302986, -0.23415337,
       -0.23413696,  1.57921282,  0.76743473, -0.46947439,  0.54256004])
```
Using RandomState():
```python
import dask.array as da
rnd = da.random.RandomState(42)
rnd.normal(0, 1, size=10).compute()

array([-1.31918189,  0.11348963,  1.2427626 , -1.26858024,  2.27840845,
        0.57524507, -1.14656226,  0.90500015, -1.17130718, -0.89184311])

import numpy as np
nprnd = np.random.RandomState(42)
nprnd.normal(0, 1, size=10)

array([ 0.49671415, -0.1382643 ,  0.64768854,  1.52302986, -0.23415337,
       -0.23413696,  1.57921282,  0.76743473, -0.46947439,  0.54256004])
```

Numpy array returned when `_numpy_state` is used (this may be intentional) :
```python
import dask.array as da
rnd = da.random.RandomState(42)
foo = rnd._numpy_state.normal(0, 1, size=10)
type(foo)

numpy.ndarray
```

**Anything else we need to know?**:

**Environment**:

- Dask version: 2021.12.0
- Python version: 3.9
- Operating System: Ubuntu 20.04.3 LTS (Focal Fossa)
- Install method (conda, pip, source): conda

<!-- If you are reporting an issue such as scale stability, cluster deadlock.
Please provide a cluster dump state with this issue, by running client.dump_cluster_state()

https://distributed.dask.org/en/stable/api.html?highlight=dump_cluster_state#distributed.Client.dump_cluster_state

-->

<details>
<summary>Cluster Dump State:</summary>

</details>


Discussion:
Hmm. I'm not sure that I expect dask and numpy to match each other. I just expect dask to match dask and numpy to match numpy. Can you explain what motivates you to want them to match each other?
I'm not the original poster, but I've followed this conversation from the original slack question. I could see someone who has unit tests written for numpy who wants to switch to dask and make sure they get the same results. Obviously if dask and numpy generate random numbers differently then this can't be depended on, but looking at the source code the dask random functions use the same numpy RandomState object that numpy does, but when it comes to actually using the state to generate the numbers dask does it differently and produces different values (if I remember correctly it generates random seeds from a RandomState that then feed into another RandomState).

Also note that this test case can be simplified if you remove the `size=10` argument, a single scalar is also different. Again I don't need this functionality, I'm just curious what dask devs think about the issue.
That makes sense as a scenario. I'll leave this open and see if anyone else chimes in. 
Thanks for the question, there are two areas of interest with this that come to mind. The first is the doc string in the random.py module whose `RandomSate` class says the following:
```
This object contains state to deterministically generate pseudo-random numbers from a variety of 
probability distributions.  It is identical to ``np.random.RandomState`` except that all functions also 
take a ``chunks=``keyword argument.
```
While the it may be identical to `np.random.RandomState` algorithmically, its implementation appears to not be identical due to how the seed is handled in practice. The reader of the comment may assume logically that the behavior of the two classes is also the same given that the exception is out of the scope of the number generation. The simple solution may be to change the language in the doc string so as to not give the impression that parity should/would exist regardless of random seeds between dask and numpy. 

Second, as mentioned above, there is also an issue of data reproducibility when looking at dask as a competitive option to alternatives when compared to numpy as a benchmark within the python ecosystem. Not being able to reproduce results of this nature may cause decision makers to question why, and look elsewhere. I understand these are theoretical concerns. 

However, the actual example in which this came about was trying to improve upon the analysis time of a m x n matrix in the billions of elements from a brownian motion simulation. Not having the reproducibility led to further analysis which was required to determine if the results are likely from the same distribution or not. This is fine during a proof of concept, but doesn't scale well into production when the production matrix would be a size of more billions than during PoC. 

If @djhoese is correct, maybe there is an opportunity to simply not pump the initial random state into a second random state. It may be that this occurrence is accidental vs an intentional design decision. It is unclear from the code comments, and I wasn't able to tell from commit history.  Thank you for the opportunity to provide more context!
I think this might be addressed in part by https://github.com/dask/dask/pull/9038?

Don't want to add noise to the maintainers' notifications but just wanted to link in case anyone else winds up on this issue.

Not clear to me if adding `dask.random.default_rng` will solve any issue with replicability though--the hard part is making it thread safe IIUC
Dask does not and probably should not guarantee random number parity between itself and any other third party library. Numpy does provide an official way to have multiple streams of random number generation from a given seed and using those methods instead of trying to replicate numpy internals does look like the correct choice.

We should also keep the option of using a different ``BitGenerator`` than what is used in numpy  - one that is more suited to [parallel random number generation](https://numpy.org/doc/stable/reference/random/upgrading-pcg64.html) for example - or of being more prone to spawning children or anything else that might come up in the future.

Agree with better docstring and making this explicit

Resolving patch:
diff --git a/dask/array/random.py b/dask/array/random.py
--- a/dask/array/random.py
+++ b/dask/array/random.py
@@ -28,15 +28,16 @@ class Generator:
     Container for the BitGenerators.
 
     ``Generator`` exposes a number of methods for generating random
-    numbers drawn from a variety of probability distributions. In addition to
-    the distribution-specific arguments, each method takes a keyword argument
-    `size` that defaults to ``None``. If `size` is ``None``, then a single
-    value is generated and returned. If `size` is an integer, then a 1-D
-    array filled with generated values is returned. If `size` is a tuple,
-    then an array with that shape is filled and returned.
+    numbers drawn from a variety of probability distributions and serves
+    as a replacement for ``RandomState``. The main difference between the
+    two is that ``Generator`` relies on an additional ``BitGenerator`` to
+    manage state and generate the random bits, which are then transformed
+    into random values from useful distributions. The default ``BitGenerator``
+    used by ``Generator`` is ``PCG64``. The ``BitGenerator`` can be changed
+    by passing an instantiated ``BitGenerator`` to ``Generator``.
 
-    The function :func:`dask.array.random.default_rng` will instantiate
-    a `Generator` with NumPy's default `BitGenerator`.
+    The function :func:`dask.array.random.default_rng` is the recommended way
+    to instantiate a ``Generator``.
 
     .. warning::
 
@@ -45,9 +46,6 @@ class Generator:
        ``Generator`` does not provide a version compatibility guarantee. In
        particular, as better algorithms evolve the bit stream may change.
 
-       This object is identical to ``np.random.Generator`` except that all
-       functions also take a ``chunks=`` keyword argument.
-
     Parameters
     ----------
     bit_generator : BitGenerator
@@ -55,6 +53,13 @@ class Generator:
 
     Notes
     -----
+    In addition to the distribution-specific arguments, each ``Generator``
+    method takes a keyword argument `size` that defaults to ``None``. If
+    `size` is ``None``, then a single value is generated and returned. If
+    `size` is an integer, then a 1-D array filled with generated values is
+    returned. If `size` is a tuple, then an array with that shape is
+    filled and returned.
+
     The Python stdlib module `random` contains pseudo-random number generator
     with a number of methods that are similar to the ones available in
     ``Generator``. It uses Mersenne Twister, and this bit generator can
@@ -62,6 +67,13 @@ class Generator:
     Dask-aware, has the advantage that it provides a much larger number
     of probability distributions to choose from.
 
+    All ``Generator`` methods are identical to ``np.random.Generator`` except
+    that they also take a `chunks=` keyword argument.
+
+    ``Generator`` does not guarantee parity in the generated numbers
+    with any third party library. In particular, numbers generated by
+    `Dask` and `NumPy` will differ even if they use the same seed.
+
     Examples
     --------
     >>> from numpy.random import PCG64
@@ -387,8 +399,8 @@ def default_rng(seed=None):
         A seed to initialize the `BitGenerator`. If None, then fresh,
         unpredictable entropy will be pulled from the OS. If an ``int`` or
         ``array_like[ints]`` is passed, then it will be passed to
-        `SeedSequence` to derive the initial `BitGenerator` state. One may also
-        pass in a `SeedSequence` instance.
+        `SeedSequence` to derive the initial `BitGenerator` state. One may
+        also pass in a `SeedSequence` instance.
         Additionally, when passed a `BitGenerator`, it will be wrapped by
         `Generator`. If passed a `Generator`, it will be returned unaltered.
 
@@ -399,13 +411,14 @@ def default_rng(seed=None):
 
     Notes
     -----
-    If ``seed`` is not a `BitGenerator` or a `Generator`, a new `BitGenerator`
-    is instantiated. This function does not manage a default global instance.
+    If ``seed`` is not a `BitGenerator` or a `Generator`, a new
+    `BitGenerator` is instantiated. This function does not manage a default
+    global instance.
 
     Examples
     --------
-    ``default_rng`` is the recommended constructor for the random number class
-    ``Generator``. Here are several ways we can construct a random
+    ``default_rng`` is the recommended constructor for the random number
+    class ``Generator``. Here are several ways we can construct a random
     number generator using ``default_rng`` and the ``Generator`` class.
 
     Here we use ``default_rng`` to generate a random float:


Test patch: