Repository: dask/dask

Issue:
Improving docstring wrapping functions
We often reuse NumPy functions' docstrings. This is convenient as we don't take up maintaining docstrings (or more likely allowing them to go stale) when we don't have anything new to add. However there are few downsides to this approach (and possibly others I'm unaware of).

1. NumPy versions show up (very confusing to users). ( https://github.com/ioam/holoviews/pull/2424#issuecomment-371896217 )
2. Arguments show up that are not fully or at all supported (also very confusing to users). ( https://github.com/dask/dask/issues/2946 )
3. Examples use NumPy instead of Dask.

Mainly these sorts of things confuse new users and leave them wondering about what Dask actually supports. This is not a great situation for us to be in. There are a few ways we can improve this.

1. Manually copy and tweak docstrings when they deviate.
2. Scrub things in our wrapper function (e.g. `versionadded`, missing args, examples).
3. Add a note to the docs pointing out how the docstrings are made.
4. Add any missing behavior (though still need to handle version added).

This could be an issue with Pandas too. I don't work with that part of the code enough to know.

Thoughts on this?
Dask dataframe 'describe()' option fails for 'object' dtype 
If I look at the documentation: http://docs.dask.org/en/latest/dataframe-api.html, it seems to be that 'object' dtypes should be accepted with the 'describe()' option by means of this sentence: "For object data (e.g. strings or timestamps), the result’s index will include count, unique, top, and freq", but as I can see in one of my examples, it does not work yet, giving the following error message: "ValueError: DataFrame contains only non-numeric data."
Any news about it so far? Thanks in advance.


Discussion:
This is also related to https://github.com/dask/dask/issues/1815

In general I agree that there is definitely some work that we can do here.  I suggest the following concrete steps:

- [ ] First, we add a disclaimer to the top of the wrapped docstring automatically.  For example this might look like the following:

    ```python
    ORIGINAL TITLE OF DOCSTRING

    Note: this docstring was automatically copied from MODULE.FUNCTION_NAME
    Some discrepancies may exist between that function and this Dask version.
    <this is optional> In particular the parameters X, Y, and Z are unsupported </this is optional>

    ORIGINAL BODY OF DOCSTRING
    ```

    for example

    ```
    Compute the arithmetic mean along the specified axis.

    Note: this docstring was automatically copied from numpy.mean
    Some discrepancies may exist between that function and this Dask version.

    Returns the average of the array elements.  The average is taken over
    the flattened array by default, otherwise over the specified axis.
    `float64` intermediate and return values are used for integer inputs.
    ```

    The unsupported paramaters are made available if the person wrapping the function uses the `ua_args` keyword to the derived_from function, where all of this is done.

    https://github.com/dask/dask/blob/7247ce666f7c277a231f59b175761d05b881884d/dask/utils.py#L470

- [ ] If `ua_args` is not specified, and if the function signature uses neither `*args` nor `**kwargs`, then we inspect the function signatures of both the wrapped and wrapping functions and note any paramters that are missing automatically.

This issue should be doable by someone with relatively little direct experience with Dask, but would be a very meaningful contribution.  I'm marking this as a good first issue.
Ah did not see that issue when I opened this one. Happy to consolidate as needed.
Lets just stay here for now.  The other one is more broad and suggests rewriting docstrings.  This one focuses on the (I think) more likely solution of handling things systematically.
It looks like we don't currently do anything for object dtype data

```python
In [1]: import dask                                                                                                

In [2]: df = dask.datasets.timeseries()                                                                            

In [3]: df.head()                                                                                                  
Out[3]: 
                       id    name         x         y
timestamp                                            
2000-01-01 00:00:00  1026  Oliver  0.857227  0.602934
2000-01-01 00:00:01   990  George  0.542299  0.947545
2000-01-01 00:00:02   982  Victor  0.034715 -0.871700
2000-01-01 00:00:03  1013     Ray  0.923782  0.592789
2000-01-01 00:00:04   992  Oliver  0.907002  0.016452

In [4]: df.describe()                                                                                              
Out[4]: 
Dask DataFrame Structure:
                  id        x        y
npartitions=1                         
               int64  float64  float64
                 ...      ...      ...
Dask Name: describe, 525 tasks

In [5]: df.describe().compute()                                                                                    
Out[5]: 
                 id             x             y
count  2.592000e+06  2.592000e+06  2.592000e+06
mean   9.999788e+02  7.694701e-04 -1.430254e-04
std    3.162041e+01  5.771207e-01  5.773583e-01
min    8.350000e+02 -9.999995e-01 -9.999996e-01
25%    9.790000e+02 -4.935266e-01 -4.926832e-01
50%    1.000000e+03  5.384249e-03  7.559225e-03
75%    1.022000e+03  5.054279e-01  5.048796e-01
max    1.166000e+03  9.999998e-01  9.999999e-01
```

docstrings are copied from pandas.  See https://github.com/dask/dask/issues/1815

If you have any interest in improving the situation that would be very welcome.
This should be a quick fix to the docs

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -1537,7 +1537,7 @@ def quantile(self, q=0.5, axis=0, method='default'):
 
     @derived_from(pd.DataFrame)
     def describe(self, split_every=False, percentiles=None, percentiles_method='default'):
-        # currently, only numeric describe is supported
+        """Currently, only numeric describe is supported """
         num = self._get_numeric_data()
         if self.ndim == 2 and len(num.columns) == 0:
             raise ValueError("DataFrame contains only non-numeric data.")
diff --git a/dask/utils.py b/dask/utils.py
--- a/dask/utils.py
+++ b/dask/utils.py
@@ -474,8 +474,9 @@ def extra_titles(doc):
     return '\n'.join(lines)
 
 
-def ignore_warning(doc, cls, name):
-    l1 = "This docstring was copied from %s.%s.%s\n" % (cls.__module__, cls.__name__, name)
+def ignore_warning(doc, cls, name, extra=""):
+    """Expand docstring by adding disclaimer and extra text"""
+    l1 = "This docstring was copied from %s.%s.%s. \n\n" % (cls.__module__, cls.__name__, name)
     l2 = "Some inconsistencies with the Dask version may exist."
 
     i = doc.find('\n\n')
@@ -486,12 +487,16 @@ def ignore_warning(doc, cls, name):
         # Indentation of next line
         indent = re.match(r'\s*', tail).group(0)
         # Insert the warning, indented, with a blank line before and after
-        doc = ''.join([
+        if extra:
+            more = [indent, extra.rstrip('\n') + '\n\n']
+        else:
+            more = []
+        bits = [
             head,
             indent, l1,
-            indent, l2, '\n\n',
-            tail
-        ])
+            indent, l2, '\n\n'
+        ] + more + [tail]
+        doc = ''.join(bits)
 
     return doc
 
@@ -507,7 +512,7 @@ def unsupported_arguments(doc, args):
     return '\n'.join(lines)
 
 
-def _derived_from(cls, method, ua_args=[]):
+def _derived_from(cls, method, ua_args=[], extra=""):
     """ Helper function for derived_from to ease testing """
     # do not use wraps here, as it hides keyword arguments displayed
     # in the doc
@@ -518,7 +523,9 @@ def _derived_from(cls, method, ua_args=[]):
 
     # Insert disclaimer that this is a copied docstring
     if doc:
-        doc = ignore_warning(doc, cls, method.__name__)
+        doc = ignore_warning(doc, cls, method.__name__, extra=extra)
+    elif extra:
+        doc += extra.rstrip('\n') + '\n\n'
 
     # Mark unsupported arguments
     try:
@@ -541,6 +548,11 @@ def _derived_from(cls, method, ua_args=[]):
 def derived_from(original_klass, version=None, ua_args=[]):
     """Decorator to attach original class's docstring to the wrapped method.
 
+    The output structure will be: top line of docstring, disclaimer about this
+    being auto-derived, any extra text associated with the method being patched,
+    the body of the docstring and finally, the list of keywords that exist in
+    the original method but not in the dask version.
+
     Parameters
     ----------
     original_klass: type
@@ -553,7 +565,10 @@ def derived_from(original_klass, version=None, ua_args=[]):
     """
     def wrapper(method):
         try:
-            method.__doc__ = _derived_from(original_klass, method, ua_args=ua_args)
+            extra = (getattr(method, '__doc__', None) or "")
+            method.__doc__ = _derived_from(
+                original_klass, method, ua_args=ua_args, extra=extra
+            )
             return method
 
         except AttributeError:


Test patch:
diff --git a/dask/bag/tests/test_avro.py b/dask/bag/tests/test_avro.py
--- a/dask/bag/tests/test_avro.py
+++ b/dask/bag/tests/test_avro.py
@@ -19,6 +19,7 @@
 
 
 def test_onefile_oneblock(tmpdir):
+    tmpdir = str(tmpdir)
     fn = os.path.join(tmpdir, 'one.avro')
     with open(fn, 'wb') as f:
         fastavro.writer(f, records=expected, schema=schema)
@@ -28,6 +29,7 @@ def test_onefile_oneblock(tmpdir):
 
 
 def test_twofile_oneblock(tmpdir):
+    tmpdir = str(tmpdir)
     fn1 = os.path.join(tmpdir, 'one.avro')
     fn2 = os.path.join(tmpdir, 'two.avro')
     with open(fn1, 'wb') as f:
@@ -40,6 +42,7 @@ def test_twofile_oneblock(tmpdir):
 
 
 def test_twofile_multiblock(tmpdir):
+    tmpdir = str(tmpdir)
     fn1 = os.path.join(tmpdir, 'one.avro')
     fn2 = os.path.join(tmpdir, 'two.avro')
     with open(fn1, 'wb') as f:
@@ -59,6 +62,7 @@ def test_twofile_multiblock(tmpdir):
 
 def test_roundtrip_simple(tmpdir):
     from dask.delayed import Delayed
+    tmpdir = str(tmpdir)
     fn = os.path.join(tmpdir, 'out*.avro')
     b = db.from_sequence([{'a': i} for i in [1, 2, 3, 4, 5]], npartitions=2)
     schema = {
@@ -76,6 +80,7 @@ def test_roundtrip_simple(tmpdir):
 
 @pytest.mark.parametrize('codec', ['null', 'deflate', 'snappy'])
 def test_roundtrip(tmpdir, codec):
+    tmpdir = str(tmpdir)
     if codec == 'snappy':
         pytest.importorskip('snappy')
     fn = os.path.join(tmpdir, 'out*.avro')
@@ -86,6 +91,7 @@ def test_roundtrip(tmpdir, codec):
 
 
 def test_invalid_schema(tmpdir):
+    tmpdir = str(tmpdir)
     b = db.from_sequence(expected, npartitions=3)
     fn = os.path.join(tmpdir, 'out*.avro')
     with pytest.raises(AssertionError):
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -3631,6 +3631,14 @@ def test_str_expand_more_columns():
     s.str.split(n=10, expand=True).compute()
 
 
+@pytest.mark.skipif(PY2,
+                    reason="docstring not rendered on py2 build")
+def test_describe_doc():
+    # tests the addition of text into a dataframe method
+    assert ("Currently, only numeric describe is supported"
+            in dd.DataFrame.describe.__doc__)
+
+
 def test_dtype_cast():
     df = pd.DataFrame({
         'A': np.arange(10, dtype=np.int32),
diff --git a/dask/tests/test_utils.py b/dask/tests/test_utils.py
--- a/dask/tests/test_utils.py
+++ b/dask/tests/test_utils.py
@@ -401,6 +401,12 @@ class Bar:
         def f(a, c):
             pass
 
+    class Zap:
+        @derived_from(Foo)
+        def f(a, c):
+            "extra docstring"
+            pass
+
     assert Bar.f.__doc__.strip().startswith('A super docstring')
     assert "Foo.f" in Bar.f.__doc__
     assert any("inconsistencies" in line for line in Bar.f.__doc__.split('\n')[:7])
@@ -409,6 +415,8 @@ def f(a, c):
     assert "not supported" in b_arg.lower()
     assert "dask" in b_arg.lower()
 
+    assert '  extra docstring\n\n' in Zap.f.__doc__
+
 
 @pytest.mark.skipif(PY2, reason="Docstrings not as easy to manipulate in Py2")
 def test_derived_from_dask_dataframe():