Repository: ros-infrastructure/rosdoc2 (language: python)

Issue:
Packages that define conf.py in standrd locations usually fail
I scanned rolling packages, and looked for packages with a conf.py in a standard location: doc/ or doc/source/. I found 18. Of these, 10 had serious failures (either no rosdoc2 result at all, or failure to show intended documentation:

conf.py failures:
1. geodesy
2. naoqi_driver
3. cv_bridge
4. mrpt_path_planning
5. robot_localization
6. ffmpeg_image_transport (readme_include fails)
7. ffmpeg_image_transport_tools (just shows package name)
8. quaternion_operation (just shows package name)
9. event_camera_py (rst files not shown; failed READMR include)
10. mrpt2

The reasons were varied, but some common reasons:
- error in user conf.py caused abort
- .rst files has ```.. include``` sections that referenced files not available in our wrapping

Of those who define conf.py, several would be better off with our newer rosdoc2 implementation rather than trying to roll their own. In others, the result would be better under our new *documentation* section rather than replacing all of the rosdoc2 output.



Discussion / hints:
Just as a FYI, I've been working recently on trying to understand and overcome issues of rosdoc2 failures from a variety of sources, including this one. So this issue is rather stale, but eventually I'll propose PRs for these issues.

Fix patch:
diff --git a/rosdoc2/verbs/build/builders/sphinx_builder.py b/rosdoc2/verbs/build/builders/sphinx_builder.py
index 73a3bc8..348889c 100644
--- a/rosdoc2/verbs/build/builders/sphinx_builder.py
+++ b/rosdoc2/verbs/build/builders/sphinx_builder.py
@@ -47,6 +47,8 @@ def esc_backslash(path):
 
 import os
 import sys
+if '{python_src_directory}' != 'None':
+    sys.path.insert(0, os.path.abspath(os.path.join('{python_src_directory}', '..')))
 
 ## exec the user's conf.py to bring all of their settings into this file.
 exec(open("{user_conf_py_filename}").read())
@@ -227,14 +229,19 @@ def ensure_global(name, default):
 
 # -- Path setup --------------------------------------------------------------
 
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
+# rosdoc2 runs sphinx in a wrapping directory so that output does not contaminate
+# the source repository. But that can make figuring out the proper path to
+# python files tricky in conf.py. Normally you do not have to set this in a custom
+# conf.py, as the proper directory is set in a wrapping conf.py (based on the project's
+# 'python_source' which by default is the package name). Unfortunately there is no general
+# way to do that in a custom conf.py (as it depends on where docs_build is located), so we do
+# not recommend sys.path be modified here.
 #
-import os
-import sys
-sys.path.insert(0, os.path.abspath(os.path.join('{python_src_directory}', '..')))
-
+# If for some reason you must set a path here, follow this pattern:
+#
+#import os
+#import sys
+#sys.path.insert(0, '<absolute path to python source directory>/..')
 
 # -- Project information -----------------------------------------------------
 


Test patch:
diff --git a/test/packages/basic_cpp/doc/conf.py b/test/packages/basic_cpp/doc/conf.py
index d5b595b..b3b9fde 100644
--- a/test/packages/basic_cpp/doc/conf.py
+++ b/test/packages/basic_cpp/doc/conf.py
@@ -6,17 +6,6 @@
 # list see the documentation:
 # https://www.sphinx-doc.org/en/master/usage/configuration.html
 
-# -- Path setup --------------------------------------------------------------
-
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-import os
-import sys
-sys.path.insert(0, os.path.abspath(os.path.join('/home/kent/github/rkent/rosdoc2/test/packages/basic_cpp/basic_cpp', '..')))
-
-
 # -- Project information -----------------------------------------------------
 
 project = 'basic_cpp_and_more'
diff --git a/test/packages/src_python/doc/source/conf.py b/test/packages/src_python/doc/source/conf.py
new file mode 100644
index 0000000..f5ce34c
--- /dev/null
+++ b/test/packages/src_python/doc/source/conf.py
@@ -0,0 +1,145 @@
+## Generated by rosdoc2.verbs.build.builders.SphinxBuilder.
+## Based on a recent output from Sphinx-quickstart.
+
+# Configuration file for the Sphinx documentation builder.
+#
+# This was copied from a generated conf.py, and flake8 does not like it.
+# flake8: noqa
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# rosdoc2 runs sphinx in wrapping directory so that output does not contaminate
+# the source repository. But that can make figuring out the proper path to
+# python files tricky in conf.py. Normally you do not have to set this in a custom
+# conf.py, as the proper directory is set in the a wrapping conf.py (based on the project's
+# 'python_src_directory' which by default is the package name). If for some reason you must
+# set a path there, as an example where your directory structure is:
+#
+#   some_package_name/
+#     package.xml
+#     doc/
+#       conf.py
+#     some_package_name/
+#       some_python_file.py
+#
+#  then the correct entry for sys.path would be:
+#
+#import os
+#import sys
+#sys.path.insert(0, os.path.abspath(os.path.join('..', '..', '..', 'some_package_name)))
+
+# Note we no longer have to adjust the sys.path in custom conf.py.
+
+# -- Project information -----------------------------------------------------
+
+project = 'src_python'
+# TODO(tfoote) The docs say year and author but we have this and it seems more relevant.
+copyright = 'The <src_python> Contributors. License: Apache-2.0'
+author = """Ye ol' Python Pro"""
+
+# The full version, including alpha/beta/rc tags
+release = '0.0.0'
+
+version = '0.0'
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+## rosdoc2 will extend the extensions to enable Breathe and Exhale if you
+## do not add them here, as well as others, perhaps.
+## If you add them manually rosdoc2 may still try to configure them.
+## See the rosdoc2_settings below for some options on avoiding that.
+extensions = [
+    'sphinx_rtd_theme',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = []
+
+master_doc = 'index'
+
+source_suffix = {
+    '.rst': 'restructuredtext',
+    '.md': 'markdown',
+    '.markdown': 'markdown',
+}
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+## rosdoc2 will override the theme, but you may set one here for running Sphinx
+## without the rosdoc2 tool.
+html_theme = 'sphinx_rtd_theme'
+
+html_theme_options = {
+    # Toc options
+    'collapse_navigation': False,
+    'sticky_navigation': True,
+    'navigation_depth': 4,
+    'includehidden': True,
+    'titles_only': False,
+}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+## rosdoc2 comments this out by default because we're not creating it.
+# html_static_path = ['_static']
+
+# -- Options for rosdoc2 -----------------------------------------------------
+
+## These settings are specific to rosdoc2, and if Sphinx is run without rosdoc2
+## they will be safely ignored.
+## None are required by default, so the lines below show the default values,
+## therefore you will need to uncomment the lines and change their value
+## if you want change the behavior of rosdoc2.
+rosdoc2_settings = {
+    ## This setting, if True, will ensure breathe is part of the 'extensions',
+    ## and will set all of the breathe configurations, if not set, and override
+    ## settings as needed if they are set by this configuration.
+    # 'enable_breathe': True,
+
+    ## This setting, if True, will ensure exhale is part of the 'extensions',
+    ## and will set all of the exhale configurations, if not set, and override
+    ## settings as needed if they are set by this configuration.
+    # 'enable_exhale': True,
+
+    ## This setting, if provided, allows option specification for breathe
+    ## directives through exhale. If not set, exhale defaults will be used.
+    ## If an empty dictionary is provided, breathe defaults will be used.
+    # 'exhale_specs_mapping': {},
+
+    ## This setting, if True, will ensure autodoc is part of the 'extensions'.
+    # 'enable_autodoc': True,
+
+    ## This setting, if True, will ensure intersphinx is part of the 'extensions'.
+    # 'enable_intersphinx': True,
+
+    ## This setting, if True, will have the 'html_theme' overridden to provide
+    ## a consistent style across all of the ROS documentation.
+    # 'override_theme': True,
+
+    ## This setting, if True, will automatically extend the intersphinx mapping
+    ## using inventory files found in the cross-reference directory.
+    ## If false, the `found_intersphinx_mappings` variable will be in the global
+    ## scope when run with rosdoc2, and could be conditionally used in your own
+    ## Sphinx conf.py file.
+    # 'automatically_extend_intersphinx_mapping': True,
+
+    ## Support markdown
+    # 'support_markdown': True,
+}
diff --git a/test/test_builder.py b/test/test_builder.py
index 3002865..e70fc5f 100644
--- a/test/test_builder.py
+++ b/test/test_builder.py
@@ -120,7 +120,7 @@ def test_src_python(module_dir):
         'src_python package',
         'documentation in source',  # We found the documentation in doc/source
     ]
-    links_exist = ['src_python.html']
+    links_exist = ['src_python.html', 'src_python.python_node.html']
 
     do_build_package(DATAPATH / PKG_NAME, module_dir)