Repository: astropy/sphinx-automodapi

Issue:
Incompatible with sphinx>4 (submodules of submodules)
Not sure what is happening but we need to make sure it is not a bug here before reporting it upstream to `sphinx`. This is blocking astropy/astropy#11763 .

@larrybradley narrowed down the issue in https://github.com/astropy/astropy/issues/11723#issuecomment-838733176

<hr/>

I think this is a real issue.  I'm getting this same error in `photutils`.  It appears to be due to submodules of submodules:


```
mod1
    __init__.py  (from mod2 import *)
    mod2
       __init__.py (from a import *)
       a.py (defines myfunc, and __all__ = ['myfunc'])
```

In this case, Sphinx is now defining duplicate references as:
`package.mod1.myfunc` and `package.mod1.mod2.myfunc` and raising the
"duplicate object descriptions".

This started with Sphinx 4 release.  Not sure if this was an unintended bug there or if something needs to be fixed in `astropy-sphinx` for compatiblity with Sphinx 4.

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/astropy/sphinx-automodapi/pull/150?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astropy) Report
> Merging [#150](https://codecov.io/gh/astropy/sphinx-automodapi/pull/150?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astropy) (0b85680) into [main](https://codecov.io/gh/astropy/sphinx-automodapi/commit/b602fdcee48f6f5c597bcc93340e34558867af9b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astropy) (b602fdc) will **decrease** coverage by `0.23%`.
> The diff coverage is `71.42%`.

```diff
@@            Coverage Diff             @@
##             main     #150      +/-   ##
==========================================
- Coverage   89.43%   89.19%   -0.24%     
==========================================
  Files          23       23              
  Lines        1098     1111      +13     
==========================================
+ Hits          982      991       +9     
- Misses        116      120       +4     
```


| [Impacted Files](https://codecov.io/gh/astropy/sphinx-automodapi/pull/150?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astropy) | Coverage Δ | |
|---|---|---|
| [sphinx\_automodapi/automodapi.py](https://codecov.io/gh/astropy/sphinx-automodapi/pull/150/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astropy#diff-c3BoaW54X2F1dG9tb2RhcGkvYXV0b21vZGFwaS5weQ==) | `92.72% <60.00%> (-1.03%)` | :arrow_down: |
| [sphinx\_automodapi/utils.py](https://codecov.io/gh/astropy/sphinx-automodapi/pull/150/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astropy#diff-c3BoaW54X2F1dG9tb2RhcGkvdXRpbHMucHk=) | `89.56% <66.66%> (-1.27%)` | :arrow_down: |
| [sphinx\_automodapi/automodsumm.py](https://codecov.io/gh/astropy/sphinx-automodapi/pull/150/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astropy#diff-c3BoaW54X2F1dG9tb2RhcGkvYXV0b21vZHN1bW0ucHk=) | `85.71% <100.00%> (+0.08%)` | :arrow_up: |

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. [Learn more](https://about.codecov.io/product/feature/runtime-insights/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astropy)

saimn: Wasn't easy but I managed to add a test :)
Who wants to review ?
saimn: Oh, of course tests fail... maybe for old sphinx versions though
saimn: All green now :)
pllim: Ooooo... how do we use this in `astropy`? Does it just magically work?
saimn: No magic :D, we must add `:noindex:` to the duplicated directives, and this PR just propagates `:noindex:` to autodoc.
pllim: @saimn , is there an `astropy` PR that uses this feature? Would be nice to see if it really works before we merge.
saimn: > is there an astropy PR that uses this feature? Would be nice to see if it really works before we merge.

@pllim - no, I could do that indeed :+1: 
pllim: @saimn , any chance we can push this forward? Did you say there is something that not quite working still using this with `astropy`? 

@WilliamJamieson is unable to build `astropy` docs on Python 3.10 because of the Sphinx maxversion pin.
saimn: Within another round of testing/debugging on Astropy I'm pretty convinced that the changes here are correct and are doing what is supposed. Adding `:noindex:` should help fixing duplicate warning for simple cases such as https://github.com/maxnoe/sphinx-automodapi-duplicated-warning, and probably photutils. But for astropy it's not enough because of the complicated structure of the docs and code, using `:noindex:` there then creates some other warnings about missing references.
pllim: Is it time to make astropy less complicated?! We cannot pin Sphinx forever.
Cadair: > Is it time to make astropy less complicated?!

I have been working on this for years :stuck_out_tongue: 
Cadair: Even though astropy core doesn't need this, we might as well merge it?
pllim: Gah, CI got disabled. I have to close/reopen.

Patch:
diff --git a/setup.cfg b/setup.cfg
index 0dfbcc6..4886f2c 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -32,7 +32,7 @@ test =
 
 [options.package_data]
 sphinx_automodapi = templates/*/*.rst
-sphinx_automodapi.tests = cases/*/*.*, cases/*/*/*.*, cases/*/*/*/*.*, cases/*/*/*/*/*.*
+sphinx_automodapi.tests = cases/*/*.*, cases/*/*/*.*, cases/*/*/*/*.*, cases/*/*/*/*/*.*, duplicated_warning/docs/*
 
 [tool:pytest]
 minversion = 4.6
diff --git a/sphinx_automodapi/automodapi.py b/sphinx_automodapi/automodapi.py
index f993cf5..a6cd719 100644
--- a/sphinx_automodapi/automodapi.py
+++ b/sphinx_automodapi/automodapi.py
@@ -63,6 +63,10 @@ It accepts the following options:
         documentation. The option ``:inherited-members:`` or ``:no-inherited-members:``
         allows the user to overrride the global setting.
 
+    * ``:noindex:``
+        Propagates the ``noindex`` flag to autodoc. Use it to avoid duplicate
+        objects warnings.
+
 
 This extension also adds four sphinx configuration options:
 
@@ -239,6 +243,7 @@ def automodapi_replace(sourcestr, app, dotoctree=True, docname=None,
             hds = '-^'
             allowedpkgnms = []
             allowothers = False
+            noindex = False
 
             # look for actual options
             unknownops = []
@@ -266,6 +271,8 @@ def automodapi_replace(sourcestr, app, dotoctree=True, docname=None,
                     inherited_members = False
                 elif opname == 'include-all-objects':
                     allowothers = True
+                elif opname == 'noindex':
+                    noindex = True
                 else:
                     unknownops.append(opname)
 
@@ -321,6 +328,8 @@ def automodapi_replace(sourcestr, app, dotoctree=True, docname=None,
             clsfuncoptions = []
             if toctreestr:
                 clsfuncoptions.append(toctreestr)
+            if noindex:
+                clsfuncoptions.append(':noindex:')
             if toskip:
                 clsfuncoptions.append(':skip: ' + ','.join(toskip))
             if allowedpkgnms:
diff --git a/sphinx_automodapi/automodsumm.py b/sphinx_automodapi/automodsumm.py
index dd2dfc2..10a71f6 100644
--- a/sphinx_automodapi/automodsumm.py
+++ b/sphinx_automodapi/automodsumm.py
@@ -124,6 +124,7 @@ class Automodsumm(Autosummary):
     option_spec['allowed-package-names'] = _str_list_converter
     option_spec['inherited-members'] = flag
     option_spec['no-inherited-members'] = flag
+    option_spec['noindex'] = flag
 
     def run(self):
         env = self.state.document.settings.env
@@ -457,7 +458,7 @@ def generate_automodsumm_docs(lines, srcfn, app=None, suffix='.rst',
     new_files = []
 
     # write
-    for name, path, template_name, inherited_mem in sorted(items):
+    for name, path, template_name, inherited_mem, noindex in sorted(items):
 
         if path is None:
             # The corresponding autosummary:: directive did not have
@@ -601,6 +602,7 @@ def generate_automodsumm_docs(lines, srcfn, app=None, suffix='.rst',
             else:
                 mod_name, obj_name = '.'.join(parts[:-1]), parts[-1]
 
+            ns['noindex'] = noindex
             ns['fullname'] = name
             ns['module'] = mod_name
             ns['objname'] = obj_name
diff --git a/sphinx_automodapi/templates/autosummary_core/class.rst b/sphinx_automodapi/templates/autosummary_core/class.rst
index 85105fa..198d04b 100644
--- a/sphinx_automodapi/templates/autosummary_core/class.rst
+++ b/sphinx_automodapi/templates/autosummary_core/class.rst
@@ -9,6 +9,9 @@
 
 .. autoclass:: {{ objname }}
    :show-inheritance:
+   {% if noindex -%}
+   :noindex:
+   {%- endif %}
 
    {% if '__init__' in methods %}
      {% set caught_result = methods.remove('__init__') %}
diff --git a/sphinx_automodapi/utils.py b/sphinx_automodapi/utils.py
index f0b1fff..2cab3ad 100644
--- a/sphinx_automodapi/utils.py
+++ b/sphinx_automodapi/utils.py
@@ -108,7 +108,7 @@ def find_mod_objs(modname, onlylocals=False):
 def find_autosummary_in_lines_for_automodsumm(lines, module=None, filename=None):
     """Find out what items appear in autosummary:: directives in the
     given lines.
-    Returns a list of (name, toctree, template, inherited_members)
+    Returns a list of (name, toctree, template, inherited_members, noindex)
     where *name* is a name
     of an object and *toctree* the :toctree: path of the corresponding
     autosummary directive (relative to the root of the file name),
@@ -133,12 +133,14 @@ def find_autosummary_in_lines_for_automodsumm(lines, module=None, filename=None)
     template_arg_re = re.compile(r'^\s+:template:\s*(.*?)\s*$')
     inherited_members_arg_re = re.compile(r'^\s+:inherited-members:\s*$')
     no_inherited_members_arg_re = re.compile(r'^\s+:no-inherited-members:\s*$')
+    noindex_arg_re = re.compile(r'^\s+:noindex:\s*$')
 
     documented = []
 
     toctree = None
     template = None
     inherited_members = None
+    noindex = None
     current_module = module
     in_autosummary = False
     base_indent = ""
@@ -168,6 +170,11 @@ def find_autosummary_in_lines_for_automodsumm(lines, module=None, filename=None)
                 inherited_members = False
                 continue
 
+            m = noindex_arg_re.match(line)
+            if m:
+                noindex = True
+                continue
+
             if line.strip().startswith(':'):
                 warn(line)
                 continue  # skip options
@@ -181,7 +188,7 @@ def find_autosummary_in_lines_for_automodsumm(lines, module=None, filename=None)
                    not name.startswith(current_module + '.'):
                     name = "%s.%s" % (current_module, name)
                 documented.append((name, toctree, template,
-                                   inherited_members))
+                                   inherited_members, noindex))
                 continue
 
             if not line.strip() or line.startswith(base_indent + " "):


Test patch:
diff --git a/sphinx_automodapi/tests/duplicated_warning/__init__.py b/sphinx_automodapi/tests/duplicated_warning/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/sphinx_automodapi/tests/duplicated_warning/docs/conf.py b/sphinx_automodapi/tests/duplicated_warning/docs/conf.py
new file mode 100644
index 0000000..b3a40ef
--- /dev/null
+++ b/sphinx_automodapi/tests/duplicated_warning/docs/conf.py
@@ -0,0 +1,11 @@
+project = 'duplicated'
+copyright = '2022, Maximilian Linhoff'
+author = 'Maximilian Linhoff'
+release = '0.1'
+
+
+extensions = [
+    "sphinx_automodapi.automodapi",
+]
+
+html_theme = 'alabaster'
diff --git a/sphinx_automodapi/tests/duplicated_warning/docs/foo.rst b/sphinx_automodapi/tests/duplicated_warning/docs/foo.rst
new file mode 100644
index 0000000..15c9792
--- /dev/null
+++ b/sphinx_automodapi/tests/duplicated_warning/docs/foo.rst
@@ -0,0 +1,9 @@
+Foo Submodule
+=============
+
+
+API Reference
+-------------
+
+.. automodapi:: sphinx_automodapi.tests.duplicated_warning.duplicated.foo
+   :noindex:
diff --git a/sphinx_automodapi/tests/duplicated_warning/docs/index.rst b/sphinx_automodapi/tests/duplicated_warning/docs/index.rst
new file mode 100644
index 0000000..55140c9
--- /dev/null
+++ b/sphinx_automodapi/tests/duplicated_warning/docs/index.rst
@@ -0,0 +1,19 @@
+.. duplicated documentation master file, created by
+   sphinx-quickstart on Tue Mar 29 17:11:23 2022.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to duplicated's documentation!
+======================================
+
+.. toctree::
+    :maxdepth: 2
+    :caption: Contents:
+
+    foo
+
+
+API Reference
+-------------
+
+.. automodapi:: sphinx_automodapi.tests.duplicated_warning.duplicated
diff --git a/sphinx_automodapi/tests/duplicated_warning/duplicated/__init__.py b/sphinx_automodapi/tests/duplicated_warning/duplicated/__init__.py
new file mode 100644
index 0000000..41d8653
--- /dev/null
+++ b/sphinx_automodapi/tests/duplicated_warning/duplicated/__init__.py
@@ -0,0 +1,6 @@
+from .foo import Foo
+
+
+__all__ = [
+    'Foo',
+]
diff --git a/sphinx_automodapi/tests/duplicated_warning/duplicated/foo/__init__.py b/sphinx_automodapi/tests/duplicated_warning/duplicated/foo/__init__.py
new file mode 100644
index 0000000..a15a7ef
--- /dev/null
+++ b/sphinx_automodapi/tests/duplicated_warning/duplicated/foo/__init__.py
@@ -0,0 +1,5 @@
+from .foo import Foo
+
+__all__ = [
+    "Foo",
+]
diff --git a/sphinx_automodapi/tests/duplicated_warning/duplicated/foo/foo.py b/sphinx_automodapi/tests/duplicated_warning/duplicated/foo/foo.py
new file mode 100644
index 0000000..c706ea3
--- /dev/null
+++ b/sphinx_automodapi/tests/duplicated_warning/duplicated/foo/foo.py
@@ -0,0 +1,17 @@
+__all__ = [
+    'Foo',
+    'Bar',
+    'baz',
+]
+
+
+class Foo:
+    '''Awesome Foo class'''
+
+
+class Bar:
+    '''Awesome Bar class'''
+
+
+def baz():
+    '''Awesome baz function'''
diff --git a/sphinx_automodapi/tests/example_module/abstract_classes.py b/sphinx_automodapi/tests/example_module/abstract_classes.py
index a5893c3..af1102d 100644
--- a/sphinx_automodapi/tests/example_module/abstract_classes.py
+++ b/sphinx_automodapi/tests/example_module/abstract_classes.py
@@ -1,10 +1,4 @@
-try:
-    # Python 3
-    from collections.abc import Sequence
-except ImportError:
-    # Python 2 (this import also works in Python <= 3.7, but will be removed in
-    # Python 3.8)
-    from collections import Sequence
+from collections.abc import Sequence
 
 __all__ = ['SequenceSubclass']
 
diff --git a/sphinx_automodapi/tests/test_cases.py b/sphinx_automodapi/tests/test_cases.py
index 351e388..63b0171 100644
--- a/sphinx_automodapi/tests/test_cases.py
+++ b/sphinx_automodapi/tests/test_cases.py
@@ -115,3 +115,28 @@ def test_run_full_case(tmpdir, case_dir, parallel):
             with open(path_reference, encoding='utf8') as f:
                 reference = f.read()
             assert actual.strip() == reference.strip()
+
+
+def test_duplicated_warning(tmpdir):
+    input_dir = os.path.join(os.path.dirname(__file__), 'duplicated_warning', 'docs')
+    docs_dir = tmpdir.mkdir('docs').strpath
+
+    start_dir = os.path.abspath('.')
+    src_dir = '.'
+
+    for root, dirnames, filenames in os.walk(input_dir):
+        for filename in filenames:
+            root_dir = os.path.join(docs_dir, os.path.relpath(root, input_dir))
+            ensuredir(root_dir)
+            input_file = os.path.join(root, filename)
+            shutil.copy(input_file, root_dir)
+
+    argv = ['-W', '-b', 'html', src_dir, '_build/html']
+
+    try:
+        os.chdir(docs_dir)
+        status = build_main(argv=argv)
+    finally:
+        os.chdir(start_dir)
+
+    assert status == 0