Repository: dask/dask
Pull request: Fix docs build warnings
- [X] Fixes #5610
- [X] Tests added / passed
- [X] Passes `pre-commit run --all-files`

Fixes all remaining docs build warnings via the following changes:

- Change `array` to `Array` in Array API docs page
- Set `:noindex:` option on `.. autofunction:: map_overlap` and `.. autoclass:: Pub` to avoid duplicate object descriptions
- Add missing contributor / reference target in changelog
- Exclude `HighLevelGraph.visualize` from API docs due to missing build dep (added TODO for future work)
- Treat docs warnings as errors (to avoid introducing new warnings in the future)

Note that some docs build warnings in Dask were a result of some warnings/docs issues in `distributed`. I opened the following PR in `distributed` that addresses those: https://github.com/dask/distributed/pull/5549. That PR will need to be merged upstream before getting a clean docs build in this PR.

Problem statement:
Clean up warnings in doc build
Right now our doc build has many warnings. This makes it hard to catch newly introduced errors in the doc build. Sphinx has an option to elevate warnings to errors.

In the details, I've included some warnings from a recent local build

<details>

```
/Users/taugspurger/miniconda3/envs/dask-dev/lib/python3.7/site-packages/numpydoc/docscrape.py:377: UserWarning: Unknown section Paramters in the docstring of None in None.
  warn(msg)
/Users/taugspurger/miniconda3/envs/dask-dev/lib/python3.7/site-packages/numpydoc/docscrape.py:377: UserWarning: Unknown section Paramters in the docstring of <function make_people at 0x1053d3c80> in /Users/taugspurger/sandbox/dask/dask/datasets.py.
  warn(msg)
/Users/taugspurger/miniconda3/envs/dask-dev/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
  return f(*args, **kwds)
/Users/taugspurger/miniconda3/envs/dask-dev/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
  return f(*args, **kwds)
/Users/taugspurger/miniconda3/envs/dask-dev/lib/python3.7/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject
  return f(*args, **kwds)
/Users/taugspurger/sandbox/dask/dask/array/core.py:docstring of dask.array.from_array:16: WARNING: Unexpected indentation.
/Users/taugspurger/sandbox/dask/dask/array/core.py:docstring of dask.array.from_array:17: WARNING: Block quote ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/dask/array/routines.py:docstring of dask.array.apply_along_axis:13: WARNING: Unexpected section title.

Parameters
__________
/Users/taugspurger/sandbox/dask/dask/array/creation.py:docstring of dask.array.indices:26: WARNING: Bullet list ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/dask/array/percentile.py:docstring of dask.array.percentile:19: WARNING: Bullet list ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/dask/array/core.py:docstring of dask.array.from_array:16: WARNING: Unexpected indentation.
/Users/taugspurger/sandbox/dask/dask/array/core.py:docstring of dask.array.from_array:17: WARNING: Block quote ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/dask/array/core.py:docstring of dask.array.to_npy_stack:26: WARNING: Inline substitution_reference start-string without end-string.
/Users/taugspurger/sandbox/dask/dask/array/core.py:docstring of dask.array.to_npy_stack:26: WARNING: Inline substitution_reference start-string without end-string.
/Users/taugspurger/sandbox/dask/dask/array/core.py:docstring of dask.array.to_npy_stack:26: WARNING: Inline substitution_reference start-string without end-string.
/Users/taugspurger/sandbox/dask/dask/array/core.py:docstring of dask.array.to_npy_stack:26: WARNING: Inline substitution_reference start-string without end-string.
/Users/taugspurger/sandbox/dask/dask/array/core.py:docstring of dask.array.Array.moment:44: WARNING: Explicit markup ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/dask/array/core.py:docstring of dask.array.Array.vindex:12: WARNING: Unknown target name: "1".
/Users/taugspurger/sandbox/dask/docs/source/array-creation.rst:221: WARNING: Duplicate explicit target name: "zarr".
/Users/taugspurger/sandbox/dask/docs/source/custom-collections.rst:46: WARNING: Unexpected section title.

Returns
-------
/Users/taugspurger/sandbox/dask/docs/source/custom-collections.rst:58: WARNING: Unexpected section title.

Returns
-------
/Users/taugspurger/sandbox/dask/docs/source/custom-collections.rst:79: WARNING: Unexpected section title.

Parameters
----------
/Users/taugspurger/sandbox/dask/docs/source/custom-collections.rst:91: WARNING: Unexpected section title.

Returns
-------
/Users/taugspurger/sandbox/dask/docs/source/custom-collections.rst:116: WARNING: Unexpected section title.

Returns
-------
/Users/taugspurger/sandbox/dask/docs/source/custom-collections.rst:139: WARNING: Unexpected section title.

Returns
-------
WARNING: failed to import to_dask_array
WARNING: failed to import to_delayed
/Users/taugspurger/sandbox/dask/dask/dataframe/core.py:docstring of dask.dataframe.DataFrame.merge:17: WARNING: Unexpected indentation.
/Users/taugspurger/sandbox/dask/dask/dataframe/core.py:docstring of dask.dataframe.DataFrame.merge:19: WARNING: Block quote ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/dask/dataframe/core.py:docstring of dask.dataframe.DataFrame.where:141: WARNING: Duplicate explicit target name: "user guide".
/Users/taugspurger/sandbox/dask/dask/dataframe/core.py:docstring of dask.dataframe.Series.where:141: WARNING: Duplicate explicit target name: "user guide".
/Users/taugspurger/sandbox/dask/dask/dataframe/core.py:docstring of dask.dataframe.Series.where:141: WARNING: Duplicate explicit target name: "user guide".
/Users/taugspurger/sandbox/dask/dask/dataframe/io/parquet/core.py:docstring of dask.dataframe.read_parquet:25: WARNING: Definition list ends without a blank line; unexpected unindent.
WARNING: autodoc: failed to import function 'make_meta' from module 'utils'; the following exception was raised:
No module named 'utils'
/Users/taugspurger/sandbox/dask/docs/source/institutional-faq.rst:158: WARNING: Bullet list ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/docs/source/institutional-faq.rst:160: WARNING: Block quote ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/docs/source/institutional-faq.rst:161: WARNING: Bullet list ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/docs/source/institutional-faq.rst:171: WARNING: Bullet list ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/docs/source/institutional-faq.rst:174: WARNING: Bullet list ends without a blank line; unexpected unindent.
/Users/taugspurger/sandbox/dask/docs/source/remote-data-services.rst:262: WARNING: Explicit markup ends without a blank line; unexpected unindent.
WARNING: duplicate object description of distributed.Client.register_worker_plugin, other instance in futures, use :noindex: for one of them
/Users/taugspurger/sandbox/distributed/distributed/deploy/ssh.py:docstring of distributed.deploy.ssh.SSHCluster:30: WARNING: Unexpected indentation.
looking for now-outdated files... none found
pickling environment... done
checking consistency... /Users/taugspurger/sandbox/dask/docs/source/phases-of-computation.rst: WARNING: document isn't included in any toctree
/Users/taugspurger/sandbox/dask/docs/source/scheduler-choice.rst: WARNING: document isn't included in any toctree
/Users/taugspurger/sandbox/dask/docs/source/scheduler-overview.rst: WARNING: document isn't included in any toctree
/Users/taugspurger/sandbox/dask/docs/source/scheduling-policy.rst: WARNING: document isn't included in any toctree
/Users/taugspurger/sandbox/dask/docs/source/shared.rst: WARNING: document isn't included in any toctree

/Users/taugspurger/miniconda3/envs/dask-dev/lib/python3.7/site-packages/dask_sphinx_theme/layout.html:22: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
  <ul>
/Users/taugspurger/sandbox/dask/docs/source/dataframe-create.rst:10: WARNING: unknown document: dataframe-overview
/Users/taugspurger/sandbox/distributed/distributed/client.py:docstring of distributed.Client.compute:44: WARNING: unknown document: resources
/Users/taugspurger/sandbox/distributed/distributed/client.py:docstring of distributed.Client.compute:51: WARNING: unknown document: actors
/Users/taugspurger/sandbox/distributed/distributed/client.py:docstring of distributed.Client.map:40: WARNING: unknown document: resources
/Users/taugspurger/sandbox/distributed/distributed/client.py:docstring of distributed.Client.map:46: WARNING: unknown document: actors
/Users/taugspurger/sandbox/distributed/distributed/client.py:docstring of distributed.Client.persist:38: WARNING: unknown document: resources
/Users/taugspurger/sandbox/distributed/distributed/client.py:docstring of distributed.Client.persist:45: WARNING: unknown document: actors
/Users/taugspurger/sandbox/distributed/distributed/client.py:docstring of distributed.Client.submit:42: WARNING: unknown document: resources
/Users/taugspurger/sandbox/distributed/distributed/client.py:docstring of distributed.Client.submit:47: WARNING: unknown document: actors
/Users/taugspurger/sandbox/dask/docs/source/setup/custom-startup.rst:30: WARNING: unknown document: plugins

copying static files... done
copying extra files... /Users/taugspurger/miniconda3/envs/dask-dev/lib/python3.7/site-packages/sphinx_rtd_theme/search.html:20: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
  {{ super() }}
WARNING: html_extra_path entry '/Users/taugspurger/sandbox/dask/docs/source/index.html' does not exist

```

</details>

Hints from issue discussion:
Just wanted to drop a note here that there are still many docs warnings https://readthedocs.org/api/v2/build/13290377.txt for example
There are a few warnings, such as:

> /home/docs/checkouts/readthedocs.org/user_builds/dask/checkouts/7419/docs/source/scheduler-overview.rst: WARNING: document isn't included in any toctree
> /home/docs/checkouts/readthedocs.org/user_builds/dask/checkouts/7419/docs/source/scheduling-policy.rst: WARNING: document isn't included in any toctree
> /home/docs/checkouts/readthedocs.org/user_builds/dask/checkouts/7419/docs/source/shared.rst: WARNING: document isn't included in any toctree

see https://readthedocs.org/api/v2/build/13290827.txt

In which toctree, should these .rst files be included?

I think I've addressed those in #7426 @mesejo, sorry I forgot to link back to this issue. There are still some more to tackle though :)
is this issue still open for work??

This issue is still open. Feel free to contribute @rajpratyush

Commit message: Avoid duplicate reference

Patch to review:
diff --git a/docs/Makefile b/docs/Makefile
index 2860b0f8bbd..7e91843cec3 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -2,7 +2,7 @@
 #
 
 # You can set these variables from the command line.
-SPHINXOPTS    =
+SPHINXOPTS    = -W --keep-going
 SPHINXBUILD   = sphinx-build
 PYTEST        = pytest
 PAPER         =
diff --git a/docs/source/array-api.rst b/docs/source/array-api.rst
index 1e9c0f765ac..79cea93d485 100644
--- a/docs/source/array-api.rst
+++ b/docs/source/array-api.rst
@@ -30,7 +30,7 @@ Top level functions
    argtopk
    argwhere
    around
-   array
+   Array
    asanyarray
    asarray
    atleast_1d
diff --git a/docs/source/array-overlap.rst b/docs/source/array-overlap.rst
index d3465780724..fec02662889 100644
--- a/docs/source/array-overlap.rst
+++ b/docs/source/array-overlap.rst
@@ -25,6 +25,7 @@ below:
    map_overlap
 
 .. autofunction:: map_overlap
+   :noindex:
 
 
 Explanation
@@ -120,7 +121,7 @@ that is not stored locally in each block:
 
    >>> filt = g.map_blocks(func)
 
-While in this case we used a SciPy function, any arbitrary function could have been 
+While in this case we used a SciPy function, any arbitrary function could have been
 used instead. This is a good interaction point with Numba_.
 
 If your function does not preserve the shape of the block, then you will need to
diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst
index 6b907793a12..eeb4589f690 100644
--- a/docs/source/changelog.rst
+++ b/docs/source/changelog.rst
@@ -4666,6 +4666,7 @@ Other
 .. _`Martin Fleischmann`: https://github.com/martinfleis
 .. _`Robert Hales`: https://github.com/robalar
 .. _`João Paulo Lacerda`: https://github.com/jopasdev
+.. _`neel iyer`: https://github.com/spiyer99
 .. _`SnkSynthesis`: https://github.com/SnkSynthesis
 .. _`JoranDox`: https://github.com/JoranDox
 .. _`Kinshuk Dua`: https://github.com/kinshukdua
diff --git a/docs/source/futures.rst b/docs/source/futures.rst
index 5096a93734b..d90bd873ac8 100644
--- a/docs/source/futures.rst
+++ b/docs/source/futures.rst
@@ -721,6 +721,7 @@ providing an additional channel of communication between ongoing tasks.
 
 .. autoclass:: Pub
    :members:
+   :noindex:
 
 Actors
 ------
diff --git a/docs/source/high-level-graphs.rst b/docs/source/high-level-graphs.rst
index 65264bcd00d..ce9d01dd343 100644
--- a/docs/source/high-level-graphs.rst
+++ b/docs/source/high-level-graphs.rst
@@ -190,3 +190,7 @@ API
 .. autoclass:: HighLevelGraph
    :members:
    :inherited-members:
+   :exclude-members: visualize
+
+.. TODO: Fix graphviz dependency in docs build and remove ``visualize`` from
+   exclude-members in the above directive


Code review comments:
diff_hunk: @@ -30,7 +30,7 @@ Top level functions
    argtopk
    argwhere
    around
-   array
+   Array
path: docs/source/array-api.rst
text: @user1:
Was this giving a warning before? I think the intention of this entry is to document the top-level `dask.array.array` function, which is different from the `dask.array.Array` class constructor:

```python
In [1]: import dask.array as da

In [2]: da.array
Out[2]: <function dask.array.routines.array(x, dtype=None, ndmin=None, *, like=None)>

In [3]: da.Array
Out[3]: dask.array.core.Array
```

@author:
Thanks for clarifying, I wasn't sure about the intention. Yes, the warning that it was throwing was:

```
/Users/koverholt/Repos/dask/docs/source/array-api.rst:9:
WARNING: autosummary: stub file not found 'dask.array.array'.
Check your autosummary_generate setting.
```

I thought it was unintentional, but with your comment I am not sure why this stub is the only one complaining. I'll look into this a bit more, but let me know if you have any insight based on the above warning.

@author:
It looks like there is no `docs/build/html/generated/dask.array.array.html` where the other stubs are generated (at least in my local build).

Is there something different about https://github.com/dask/dask/blob/a5aecac8313fea30c5503f534c71f325b1775b9c/dask/array/routines.py#L44-L53 that would result in `dask.array.array` being the only function with no generated stub?

Although this is happening in my local build, and the stub exists in the Dask docs: https://docs.dask.org/en/stable/generated/dask.array.array.html.

@author:
I reverted my change and set it back to `array`, and although it throws a warning on my machine, it seems to be working fine in RTD docs build:

- https://readthedocs.org/api/v2/build/15413948.txt (no warning in there about `dask.array.array`, and the other warnings shown in that build log will go away after https://github.com/dask/distributed/pull/5549 is merged)
- https://dask--8432.org.readthedocs.build/en/8432/generated/dask.array.array.html

So this PR should be ready to go now.