Repository: dask/dask

Issue:
dask.array.random docs not showing up properly
dask.array.random docs not showing up properly

https://docs.dask.org/en/latest/array-api.html#random

I guess the table is printing the first line of the docstring e.g. https://docs.dask.org/en/latest/array-api.html#dask.array.random.beta

believe the docs are autogenerated in
https://github.com/dask/dask/blob/master/docs/source/array-api.rst#random

Not sure of a workaround.


Discussion:
Thank @raybellwaves. I see the issue -- attaching a photo for others:

<img width="785" alt="Screen Shot 2020-03-29 at 9 47 18 PM" src="https://user-images.githubusercontent.com/1403768/77867830-e674b580-7206-11ea-8dea-35c1de3313c3.png">

@raybellwaves do you know if np.random.RandsomState is differently formatted compared with other NumPy functions?  I suspect not, but thought I would ask.

If someone does have time perhaps looking at the logic in how dask inserts the warning would be good
https://github.com/dask/dask/blob/241027d1e270c2793244d9f16e53903c5ea5bd20/dask/utils.py#L582-L597

For example, the following is from a debug session when inserting the warning into np.random.RandomState.beta.

```
ipdb> doc                                                                                                                                                                                                                   
'\n        beta(a, b, size=None)\n\n        Draw samples from a Beta distribution.\n\n        The Beta distribution is a special case of the Dirichlet distribution,\n        and is related to the Gamma distribution.  It has the probability\n        distribution function\n\n        .. math:: f(x; a,b) = \\frac{1}{B(\\alpha, \\beta)} x^{\\alpha - 1}\n                                                         (1 - x)^{\\beta - 1},\n\n        where the normalization, B, is the beta function,\n\n        .. math:: B(\\alpha, \\beta) = \\int_0^1 t^{\\alpha - 1}\n                                     (1 - t)^{\\beta - 1} dt.\n\n        It is often seen in Bayesian inference and order statistics.\n\n        .. note::\n            New code should use the ``beta`` method of a ``default_rng()``\n            instance instead; see `random-quick-start`.\n\n        Parameters\n        ----------\n        a : float or array_like of floats\n            Alpha, positive (>0).\n        b : float or array_like of floats\n            Beta, positive (>0).\n        size : int or tuple of ints, optional\n            Output shape.  If the given shape is, e.g., ``(m, n, k)``, then\n            ``m * n * k`` samples are drawn.  If size is ``None`` (default),\n            a single value is returned if ``a`` and ``b`` are both scalars.\n            Otherwise, ``np.broadcast(a, b).size`` samples are drawn.\n\n        Returns\n        -------\n        out : ndarray or scalar\n            Drawn samples from the parameterized beta distribution.\n\n        See Also\n        --------\n        Generator.beta: which should be used for new code.\n        '
ipdb> head = doc[: i + 2]                                                                                                                                                                                                   
ipdb> tail = doc[i + 2 :]                                                                                                                                                                                                   
ipdb> head                                                                                                                                                                                                                  
'\n        beta(a, b, size=None)\n\n'
ipdb> tail                                                                                                                                                                                                                  
'        Draw samples from a Beta distribution.\n\n        The Beta distribution is a special case of the Dirichlet distribution,\n        and is related to the Gamma distribution.  It has the probability\n        distribution function\n\n        .. math:: f(x; a,b) = \\frac{1}{B(\\alpha, \\beta)} x^{\\alpha - 1}\n                                                         (1 - x)^{\\beta - 1},\n\n        where the normalization, B, is the beta function,\n\n        .. math:: B(\\alpha, \\beta) = \\int_0^1 t^{\\alpha - 1}\n                                     (1 - t)^{\\beta - 1} dt.\n\n        It is often seen in Bayesian inference and order statistics.\n\n        .. note::\n            New code should use the ``beta`` method of a ``default_rng()``\n            instance instead; see `random-quick-start`.\n\n        Parameters\n        ----------\n        a : float or array_like of floats\n            Alpha, positive (>0).\n        b : float or array_like of floats\n            Beta, positive (>0).\n        size : int or tuple of ints, optional\n            Output shape.  If the given shape is, e.g., ``(m, n, k)``, then\n            ``m * n * k`` samples are drawn.  If size is ``None`` (default),\n            a single value is returned if ``a`` and ``b`` are both scalars.\n            Otherwise, ``np.broadcast(a, b).size`` samples are drawn.\n\n        Returns\n        -------\n        out : ndarray or scalar\n            Drawn samples from the parameterized beta distribution.\n\n        See Also\n        --------\n        Generator.beta: which should be used for new code.\n       
```

@jsignell i recall you looking at some doc issues recently.  If you have time would you be willing to look into what is happening here.  If not, i can probably pick it up some time next week
I can look at this later this week if nobody picks it up.

Resolving patch:
diff --git a/dask/array/random.py b/dask/array/random.py
--- a/dask/array/random.py
+++ b/dask/array/random.py
@@ -196,21 +196,21 @@ def _broadcast_any(ar, shape, chunks):
         graph = HighLevelGraph.from_collections(name, dsk, dependencies=dependencies)
         return Array(graph, name, chunks + extra_chunks, meta=meta)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def beta(self, a, b, size=None, chunks="auto", **kwargs):
         return self._wrap("beta", a, b, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def binomial(self, n, p, size=None, chunks="auto", **kwargs):
         return self._wrap("binomial", n, p, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def chisquare(self, df, size=None, chunks="auto", **kwargs):
         return self._wrap("chisquare", df, size=size, chunks=chunks, **kwargs)
 
     with ignoring(AttributeError):
 
-        @derived_from(np.random.RandomState)
+        @derived_from(np.random.RandomState, skipblocks=1)
         def choice(self, a, size=None, replace=True, p=None, chunks="auto"):
             dependencies = []
             # Normalize and validate `a`
@@ -282,52 +282,52 @@ def choice(self, a, size=None, replace=True, p=None, chunks="auto"):
             )
             return Array(graph, name, chunks, dtype=dtype)
 
-    # @derived_from(np.random.RandomState)
+    # @derived_from(np.random.RandomState, skipblocks=1)
     # def dirichlet(self, alpha, size=None, chunks="auto"):
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def exponential(self, scale=1.0, size=None, chunks="auto", **kwargs):
         return self._wrap("exponential", scale, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def f(self, dfnum, dfden, size=None, chunks="auto", **kwargs):
         return self._wrap("f", dfnum, dfden, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def gamma(self, shape, scale=1.0, size=None, chunks="auto", **kwargs):
         return self._wrap("gamma", shape, scale, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def geometric(self, p, size=None, chunks="auto", **kwargs):
         return self._wrap("geometric", p, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def gumbel(self, loc=0.0, scale=1.0, size=None, chunks="auto", **kwargs):
         return self._wrap("gumbel", loc, scale, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def hypergeometric(self, ngood, nbad, nsample, size=None, chunks="auto", **kwargs):
         return self._wrap(
             "hypergeometric", ngood, nbad, nsample, size=size, chunks=chunks, **kwargs
         )
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def laplace(self, loc=0.0, scale=1.0, size=None, chunks="auto", **kwargs):
         return self._wrap("laplace", loc, scale, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def logistic(self, loc=0.0, scale=1.0, size=None, chunks="auto", **kwargs):
         return self._wrap("logistic", loc, scale, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def lognormal(self, mean=0.0, sigma=1.0, size=None, chunks="auto", **kwargs):
         return self._wrap("lognormal", mean, sigma, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def logseries(self, p, size=None, chunks="auto", **kwargs):
         return self._wrap("logseries", p, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def multinomial(self, n, pvals, size=None, chunks="auto", **kwargs):
         return self._wrap(
             "multinomial",
@@ -338,31 +338,31 @@ def multinomial(self, n, pvals, size=None, chunks="auto", **kwargs):
             extra_chunks=((len(pvals),),),
         )
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def negative_binomial(self, n, p, size=None, chunks="auto", **kwargs):
         return self._wrap("negative_binomial", n, p, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def noncentral_chisquare(self, df, nonc, size=None, chunks="auto", **kwargs):
         return self._wrap(
             "noncentral_chisquare", df, nonc, size=size, chunks=chunks, **kwargs
         )
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def noncentral_f(self, dfnum, dfden, nonc, size=None, chunks="auto", **kwargs):
         return self._wrap(
             "noncentral_f", dfnum, dfden, nonc, size=size, chunks=chunks, **kwargs
         )
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def normal(self, loc=0.0, scale=1.0, size=None, chunks="auto", **kwargs):
         return self._wrap("normal", loc, scale, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def pareto(self, a, size=None, chunks="auto", **kwargs):
         return self._wrap("pareto", a, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def permutation(self, x):
         from .slicing import shuffle_slice
 
@@ -373,83 +373,83 @@ def permutation(self, x):
         self._numpy_state.shuffle(index)
         return shuffle_slice(x, index)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def poisson(self, lam=1.0, size=None, chunks="auto", **kwargs):
         return self._wrap("poisson", lam, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def power(self, a, size=None, chunks="auto", **kwargs):
         return self._wrap("power", a, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def randint(self, low, high=None, size=None, chunks="auto", dtype="l", **kwargs):
         return self._wrap(
             "randint", low, high, size=size, chunks=chunks, dtype=dtype, **kwargs
         )
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def random_integers(self, low, high=None, size=None, chunks="auto", **kwargs):
         return self._wrap(
             "random_integers", low, high, size=size, chunks=chunks, **kwargs
         )
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def random_sample(self, size=None, chunks="auto", **kwargs):
         return self._wrap("random_sample", size=size, chunks=chunks, **kwargs)
 
     random = random_sample
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def rayleigh(self, scale=1.0, size=None, chunks="auto", **kwargs):
         return self._wrap("rayleigh", scale, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def standard_cauchy(self, size=None, chunks="auto", **kwargs):
         return self._wrap("standard_cauchy", size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def standard_exponential(self, size=None, chunks="auto", **kwargs):
         return self._wrap("standard_exponential", size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def standard_gamma(self, shape, size=None, chunks="auto", **kwargs):
         return self._wrap("standard_gamma", shape, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def standard_normal(self, size=None, chunks="auto", **kwargs):
         return self._wrap("standard_normal", size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def standard_t(self, df, size=None, chunks="auto", **kwargs):
         return self._wrap("standard_t", df, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def tomaxint(self, size=None, chunks="auto", **kwargs):
         return self._wrap("tomaxint", size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def triangular(self, left, mode, right, size=None, chunks="auto", **kwargs):
         return self._wrap(
             "triangular", left, mode, right, size=size, chunks=chunks, **kwargs
         )
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def uniform(self, low=0.0, high=1.0, size=None, chunks="auto", **kwargs):
         return self._wrap("uniform", low, high, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def vonmises(self, mu, kappa, size=None, chunks="auto", **kwargs):
         return self._wrap("vonmises", mu, kappa, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def wald(self, mean, scale, size=None, chunks="auto", **kwargs):
         return self._wrap("wald", mean, scale, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def weibull(self, a, size=None, chunks="auto", **kwargs):
         return self._wrap("weibull", a, size=size, chunks=chunks, **kwargs)
 
-    @derived_from(np.random.RandomState)
+    @derived_from(np.random.RandomState, skipblocks=1)
     def zipf(self, a, size=None, chunks="auto", **kwargs):
         return self._wrap("zipf", a, size=size, chunks=chunks, **kwargs)
 
diff --git a/dask/utils.py b/dask/utils.py
--- a/dask/utils.py
+++ b/dask/utils.py
@@ -565,7 +565,7 @@ def extra_titles(doc):
     return "\n".join(lines)
 
 
-def ignore_warning(doc, cls, name, extra=""):
+def ignore_warning(doc, cls, name, extra="", skipblocks=0):
     """Expand docstring by adding disclaimer and extra text"""
     import inspect
 
@@ -584,6 +584,11 @@ def ignore_warning(doc, cls, name, extra=""):
         # Insert our warning
         head = doc[: i + 2]
         tail = doc[i + 2 :]
+        while skipblocks > 0:
+            i = tail.find("\n\n")
+            head = tail[: i + 2]
+            tail = tail[i + 2 :]
+            skipblocks -= 1
         # Indentation of next line
         indent = re.match(r"\s*", tail).group(0)
         # Insert the warning, indented, with a blank line before and after
@@ -612,7 +617,7 @@ def unsupported_arguments(doc, args):
     return "\n".join(lines)
 
 
-def _derived_from(cls, method, ua_args=[], extra=""):
+def _derived_from(cls, method, ua_args=[], extra="", skipblocks=0):
     """ Helper function for derived_from to ease testing """
     # do not use wraps here, as it hides keyword arguments displayed
     # in the doc
@@ -628,7 +633,9 @@ def _derived_from(cls, method, ua_args=[], extra=""):
 
     # Insert disclaimer that this is a copied docstring
     if doc:
-        doc = ignore_warning(doc, cls, method.__name__, extra=extra)
+        doc = ignore_warning(
+            doc, cls, method.__name__, extra=extra, skipblocks=skipblocks
+        )
     elif extra:
         doc += extra.rstrip("\n") + "\n\n"
 
@@ -650,7 +657,7 @@ def _derived_from(cls, method, ua_args=[], extra=""):
     return doc
 
 
-def derived_from(original_klass, version=None, ua_args=[]):
+def derived_from(original_klass, version=None, ua_args=[], skipblocks=0):
     """Decorator to attach original class's docstring to the wrapped method.
 
     The output structure will be: top line of docstring, disclaimer about this
@@ -667,13 +674,20 @@ def derived_from(original_klass, version=None, ua_args=[]):
     ua_args : list
         List of keywords which Dask doesn't support. Keywords existing in
         original but not in Dask will automatically be added.
+    skipblocks : int
+        How many text blocks (paragraphs) to skip from the start of the
+        docstring. Useful for cases where the target has extra front-matter.
     """
 
     def wrapper(method):
         try:
             extra = getattr(method, "__doc__", None) or ""
             method.__doc__ = _derived_from(
-                original_klass, method, ua_args=ua_args, extra=extra
+                original_klass,
+                method,
+                ua_args=ua_args,
+                extra=extra,
+                skipblocks=skipblocks,
             )
             return method
 


Test patch: