Repository: scverse/scanpy

Issue:
Sphinx 4.1.0 doesn't like ScanpyConfig
Update:

Docs don't build with sphinx 4.1.0 due to a error triggered by `scanpydoc`. Sphinx will be pinned until this is solved (which is when this issue should be closed). It's not obvious to me at the moment whether sphinx or scanpydoc is at fault.

---------------

Trying to build the docs with Sphinx 4.1.0 fails with the following output:

<details>
<summary> </summary>

```sh
$ make html
Running Sphinx v4.1.0
loading intersphinx inventory from https://anndata.readthedocs.io/en/stable/objects.inv...
loading intersphinx inventory from https://bbknn.readthedocs.io/en/latest/objects.inv...
loading intersphinx inventory from https://matplotlib.org/cycler/objects.inv...
loading intersphinx inventory from http://docs.h5py.org/en/stable/objects.inv...
loading intersphinx inventory from https://ipython.readthedocs.io/en/stable/objects.inv...
loading intersphinx inventory from https://leidenalg.readthedocs.io/en/latest/objects.inv...
loading intersphinx inventory from https://louvain-igraph.readthedocs.io/en/latest/objects.inv...
loading intersphinx inventory from https://matplotlib.org/objects.inv...
loading intersphinx inventory from https://networkx.github.io/documentation/networkx-1.10/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/numpy/objects.inv...
loading intersphinx inventory from https://pandas.pydata.org/pandas-docs/stable/objects.inv...
loading intersphinx inventory from https://docs.pytest.org/en/latest/objects.inv...
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/scipy/reference/objects.inv...
loading intersphinx inventory from https://seaborn.pydata.org/objects.inv...
loading intersphinx inventory from https://scikit-learn.org/stable/objects.inv...
loading intersphinx inventory from https://scanpy-tutorials.readthedocs.io/en/latest/objects.inv...
intersphinx inventory has moved: https://networkx.github.io/documentation/networkx-1.10/objects.inv -> https://networkx.org/documentation/networkx-1.10/objects.inv
intersphinx inventory has moved: https://docs.scipy.org/doc/numpy/objects.inv -> https://numpy.org/doc/stable/objects.inv
intersphinx inventory has moved: http://docs.h5py.org/en/stable/objects.inv -> https://docs.h5py.org/en/stable/objects.inv
[autosummary] generating autosummary for: _key_contributors.rst, api.rst, basic_usage.rst, community.rst, contributors.rst, dev/ci.rst, dev/code.rst, dev/documentation.rst, dev/external-tools.rst, dev/getting-set-up.rst, ..., release-notes/1.7.1.rst, release-notes/1.7.2.rst, release-notes/1.8.0.rst, release-notes/1.8.1.rst, release-notes/1.8.2.rst, release-notes/1.9.0.rst, release-notes/index.rst, release-notes/release-latest.rst, tutorials.rst, usage-principles.rst
Error in github_url('scanpy._settings.ScanpyConfig.N_PCS'):

Extension error (sphinx.ext.autosummary):
Handler <function process_generate_options at 0x139c4a940> for event 'builder-inited' threw an exception (exception: type object 'ScanpyConfig' has no attribute 'N_PCS')
make: *** [html] Error 2
```

</details>

However, I'm entirely sure if this is Sphinx's fault, or our own. Currently the [N_PCS parameter isn't in the rendered documentation](https://scanpy.readthedocs.io/en/stable/generated/scanpy._settings.ScanpyConfig.html#scanpy._settings.ScanpyConfig). I think it should be, and am not sure why it's not showing up here.

To summarize:

* Previous versions of our doc builds didn't seem to be including attribute docstrings for `ScanpyConfig`.
* Sphinx 4.1.0 raises an error when it hits this attribute


Discussion:
Changing it to a property throws a different error:

<details>
<summary> from make html </summary>

```sh
reading sources... [  5%] generated/classes/scanpy.pl.DotPlot                                                           
Exception occurred:
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/docfields.py", line 369, in transform
    new_list += fieldtype.make_field(fieldtypes, self.directive.domain, items,
TypeError: make_field() got an unexpected keyword argument 'inliner'
The full traceback has been saved in /var/folders/bd/43q20k0n6z15tdfzxvd22r7c0000gn/T/sphinx-err-qbzn5se8.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make: *** [html] Error 2
```

</details>


<details>
<summary> contents of the referenced log file </summary>

```python
# Sphinx version: 4.1.0
# Python version: 3.8.10 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 2.11.2
# Last messages:
#   reading sources... [  2%] dev/documentation
#   reading sources... [  2%] dev/external-tools
#   reading sources... [  3%] dev/getting-set-up
#   reading sources... [  3%] dev/index
#   reading sources... [  3%] dev/release
#   reading sources... [  4%] dev/testing
#   reading sources... [  4%] dev/versioning
#   reading sources... [  4%] ecosystem
#   reading sources... [  5%] external
#   reading sources... [  5%] generated/classes/scanpy.pl.DotPlot
# Loaded extensions:
#   sphinx.ext.mathjax (4.1.0) from /usr/local/lib/python3.8/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /usr/local/lib/python3.8/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /usr/local/lib/python3.8/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (2.0.0) from /usr/local/lib/python3.8/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.5) from /usr/local/lib/python3.8/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /usr/local/lib/python3.8/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /usr/local/lib/python3.8/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc.preserve_defaults (1.0) from /usr/local/lib/python3.8/site-packages/sphinx/ext/autodoc/preserve_defaults.py
#   sphinx.ext.autodoc.type_comment (4.1.0) from /usr/local/lib/python3.8/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (4.1.0) from /usr/local/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.intersphinx (4.1.0) from /usr/local/lib/python3.8/site-packages/sphinx/ext/intersphinx.py
#   sphinx.ext.doctest (4.1.0) from /usr/local/lib/python3.8/site-packages/sphinx/ext/doctest.py
#   sphinx.ext.coverage (4.1.0) from /usr/local/lib/python3.8/site-packages/sphinx/ext/coverage.py
#   sphinx.ext.napoleon (4.1.0) from /usr/local/lib/python3.8/site-packages/sphinx/ext/napoleon/__init__.py
#   sphinx.ext.autosummary (4.1.0) from /usr/local/lib/python3.8/site-packages/sphinx/ext/autosummary/__init__.py
#   matplotlib.sphinxext.plot_directive (3.4.2) from /usr/local/lib/python3.8/site-packages/matplotlib/sphinxext/plot_directive.py
#   sphinx_autodoc_typehints (unknown version) from /usr/local/lib/python3.8/site-packages/sphinx_autodoc_typehints.py
#   scanpydoc.autosummary_generate_imported (0.7.2) from /usr/local/lib/python3.8/site-packages/scanpydoc/autosummary_generate_imported.py
#   scanpydoc.definition_list_typed_field (0.7.2) from /usr/local/lib/python3.8/site-packages/scanpydoc/definition_list_typed_field.py
#   scanpydoc.elegant_typehints (0.7.2) from /usr/local/lib/python3.8/site-packages/scanpydoc/elegant_typehints/__init__.py
#   scanpydoc.rtd_github_links (0.7.2) from /usr/local/lib/python3.8/site-packages/scanpydoc/rtd_github_links.py
#   scanpydoc.theme (unknown version) from /usr/local/lib/python3.8/site-packages/scanpydoc/theme/__init__.py
#   scanpydoc (0.7.2) from /usr/local/lib/python3.8/site-packages/scanpydoc/__init__.py
#   typed_returns (unknown version) from /Users/isaac/github/scanpy/docs/extensions/typed_returns.py
#   function_images (unknown version) from /Users/isaac/github/scanpy/docs/extensions/function_images.py
#   github_links (unknown version) from /Users/isaac/github/scanpy/docs/extensions/github_links.py
#   param_police (unknown version) from /Users/isaac/github/scanpy/docs/extensions/param_police.py
#   debug_docstrings (unknown version) from /Users/isaac/github/scanpy/docs/extensions/debug_docstrings.py
#   sphinx_rtd_theme (unknown version) from /usr/local/lib/python3.8/site-packages/sphinx_rtd_theme/__init__.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/usr/local/lib/python3.8/site-packages/sphinx/application.py", line 343, in build
    self.builder.build_update()
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 293, in build_update
    self.build(to_build,
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 307, in build
    updated_docnames = set(self.read())
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 414, in read
    self._read_serial(docnames)
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 435, in _read_serial
    self.read_doc(docname)
  File "/usr/local/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 475, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/usr/local/lib/python3.8/site-packages/sphinx/io.py", line 188, in read_doc
    pub.publish()
  File "/usr/local/lib/python3.8/site-packages/docutils/core.py", line 217, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "/usr/local/lib/python3.8/site-packages/sphinx/io.py", line 108, in read
    self.parse()
  File "/usr/local/lib/python3.8/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/usr/local/lib/python3.8/site-packages/sphinx/parsers.py", line 100, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 170, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
  File "/usr/local/lib/python3.8/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/local/lib/python3.8/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 393, in new_subsection
    newabsoffset = self.nested_parse(
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/local/lib/python3.8/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/local/lib/python3.8/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2344, in explicit_markup
    self.explicit_list(blank_finish)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2369, in explicit_list
    newline_offset, blank_finish = self.nested_list_parse(
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 318, in nested_list_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/local/lib/python3.8/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/local/lib/python3.8/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2647, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
    return self.run_directive(
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/usr/local/lib/python3.8/site-packages/sphinx/ext/autodoc/directive.py", line 173, in run
    result = parse_generated_content(self.state, params.result, documenter)
  File "/usr/local/lib/python3.8/site-packages/sphinx/ext/autodoc/directive.py", line 120, in parse_generated_content
    state.nested_parse(content, 0, node)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 281, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/local/lib/python3.8/site-packages/docutils/statemachine.py", line 241, in run
    context, next_state, result = self.check_line(
  File "/usr/local/lib/python3.8/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
    return self.run_directive(
  File "/usr/local/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/usr/local/lib/python3.8/site-packages/sphinx/domains/__init__.py", line 286, in run
    return super().run()
  File "/usr/local/lib/python3.8/site-packages/sphinx/directives/__init__.py", line 212, in run
    DocFieldTransformer(self).transform_all(contentnode)
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/docfields.py", line 251, in transform_all
    self.transform(child)
  File "/usr/local/lib/python3.8/site-packages/sphinx/util/docfields.py", line 369, in transform
    new_list += fieldtype.make_field(fieldtypes, self.directive.domain, items,
TypeError: make_field() got an unexpected keyword argument 'inliner'

```

</details>
Looking into it a bit further, it seems like an incompatibility with scanpydoc's `DLTypedField`. The API for `sphinx.util.docfield.Field.make_field` now has kwargs `inliner` and `location` which `DocFieldTransformer` uses.

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -155,6 +155,9 @@ def setup(app):
     ('py:class', 'scanpy.plotting._utils._AxesSubplot'),
     ('py:class', 'scanpy._utils.Empty'),
     ('py:class', 'numpy.random.mtrand.RandomState'),
+    # Will work once scipy 1.8 is released
+    ('py:class', 'scipy.sparse.base.spmatrix'),
+    ('py:class', 'scipy.sparse.csr.csr_matrix'),
 ]
 
 # Options for plot examples


Test patch:
diff --git a/docs/extensions/has_attr_test.py b/docs/extensions/has_attr_test.py
new file mode 100644
--- /dev/null
+++ b/docs/extensions/has_attr_test.py
@@ -0,0 +1,13 @@
+from jinja2.defaults import DEFAULT_NAMESPACE
+from jinja2.utils import import_string
+from sphinx.application import Sphinx
+
+
+def has_attr(obj_path: str, attr: str) -> bool:
+    # https://jinja.palletsprojects.com/en/3.0.x/api/#custom-tests
+    obj = import_string(obj_path)
+    return hasattr(obj, attr)
+
+
+def setup(app: Sphinx):
+    DEFAULT_NAMESPACE["has_attr"] = has_attr