Repository: ros-infrastructure/rosdoc2 (language: python)

Issue:
Add scan option to build packages in subdirectories
Replaces #81

This PR implements a new top-level option, 'scan', which scans subdirectories, and builds any found packages. It uses the same options as ```build``` with a couple of additions: a per-package timeout, and a maximum number of packages.

I've been running variations of this for a few weeks now in a nightly Jenkins build of all ros2:iron packages, so the available output has been tweaked to focus on that environment.

It captures output from each package build in a file stdout.txt in the docs_build folder. Summary information is printed in the more typical console locations. Looking at the notes from #81, I believe this implements most of those requests, with the exception of the request to fail on first error. Plenty of existing packages have errors, so I don't really see the value of this if you are running a full distro. If you are only running a repository, it runs quickly enough that I don't see the value of that option either. I do include though a maximum number of packages options which is useful to limit time when testing. If you really want that feature, let's do that as a followup, it would not be very hard. But please state the use case.

On my system, if I set a ```-t 1``` timeout then about a third of the test packages time out, which can give some idea of what error output looks like. So I can run ```rosdoc2 scan -p test -t 1``` from the rosdoc2 directory as a simple test. Or you could do that on a repository with multiple packages instead.

I used colcon as requested, but I have some comments on that. First, the documentation on colcon that I could find is all from the perspective of using the resulting commands that are generated using colcon. I could find no hints of how to use it as a python library. Second, the code has multiple levels of abstraction, with key features being added through extensions. Examining the code for, say, colcon_core.package_discovery give no clue on what options are supported, as all of that is added in extensions. What extensions? I have no idea. Basically through trial and error and guesswork, I was able to discover that colcon-common-extensions give the functionality that I need, with ```--base-paths``` as the option I needed to set. Third, using colcon as a package discovery library requires 7 lines of obscure setup code. Yes this was partly written as a demo for me, but this is not at all the expectation one has when using library code. In summary, I would not recommend to anyone else that they attempt to use colcon as library code like this.




Discussion / hints:
Here is the list of failed packages from a recent ros2:iron run of this with the default 15 minute timeout (this is a 10 year old i7 machine using a physical hard disk):
```
37 packages failed:
rviz_common: retval=1: RuntimeError Sphinx-build exited with return code '2'
azure_iot_sdks: retval=1: RuntimeError Failed to find a ROS package at given path 'rosindex/_plugins_data/checkout/azure-iot-sdk-c/github-Azure-azure-iot-sdk-c/azure-iot-sdk-c'
ecl_geometry: retval=1: RuntimeError Sphinx-build exited with return code '2'
rotconv: retval=1: RuntimeError Failed to find a ROS package at given path 'rosindex/_plugins_data/checkout/rot_conv_lib/github-AIS-Bonn-rot_conv_lib/rot_conv_lib'
proxsuite: retval=2: TimeoutError runtime 900.351 seconds
gazebo_plugins: retval=1: RuntimeError Sphinx-build exited with return code '2'
magic_enum: retval=1: RuntimeError Sphinx-build exited with return code '2'
avt_vimba_camera: retval=1: RuntimeError Sphinx-build exited with return code '1'
fastcdr: retval=1: RuntimeError Failed to find a ROS package at given path 'rosindex/_plugins_data/checkout/fastcdr/github-eProsima-Fast-CDR/fastcdr'
octomap-distribution: retval=1: RuntimeError Failed to find a ROS package at given path 'rosindex/_plugins_data/checkout/octomap/github-octomap-octomap/octomap'
mrpt2: retval=1: RuntimeError Sphinx-build exited with return code '2'
libcaer: retval=2: TimeoutError runtime 900.348 seconds
mrpt_path_planning: retval=1: RuntimeError Sphinx-build exited with return code '2'
vitis_common: retval=2: TimeoutError runtime 900.349 seconds
sm_multi_stage_1: retval=2: TimeoutError runtime 900.349 seconds
sm_pack_ml: retval=2: TimeoutError runtime 900.349 seconds
VRPN: retval=1: RuntimeError Failed to find a ROS package at given path 'rosindex/_plugins_data/checkout/vrpn/github-vrpn-vrpn/vrpn'
VISP: retval=1: RuntimeError Failed to find a ROS package at given path 'rosindex/_plugins_data/checkout/visp/github-lagadic-visp/visp'
urdfdom_headers: retval=1: RuntimeError Failed to find a ROS package at given path 'rosindex/_plugins_data/checkout/urdfdom_headers/github-ros-urdfdom_headers/urdfdom_headers'
nav2_costmap_2d: retval=1: RuntimeError Sphinx-build exited with return code '2'
ecl_containers: retval=1: RuntimeError Sphinx-build exited with return code '2'
sick_scan_xd: retval=2: TimeoutError runtime 900.347 seconds
robot_localization: retval=1: RuntimeError Sphinx-build exited with return code '2'
naoqi_driver: retval=1: RuntimeError Sphinx-build exited with return code '2'
ecl_manipulators: retval=1: RuntimeError Sphinx-build exited with return code '2'
pinocchio: retval=2: TimeoutError runtime 900.351 seconds
ecl_converters: retval=1: RuntimeError Sphinx-build exited with return code '2'
fastrtps: retval=2: TimeoutError runtime 900.357 seconds
geodesy: retval=1: RuntimeError Sphinx-build exited with return code '2'
ign_rviz: retval=1: RuntimeError Sphinx-build exited with return code '2'
cv_bridge: retval=1: RuntimeError Sphinx-build exited with return code '2'
camera_calibration: retval=1: RuntimeError Sphinx-build exited with return code '2'
transmission_interface: retval=1: RuntimeError Sphinx-build exited with return code '1'
ur_client_library: retval=1: RuntimeError Sphinx-build exited with return code '2'
IFM3D: retval=1: RuntimeError Failed to find a ROS package at given path 'rosindex/_plugins_data/checkout/ifm3d_core/github-ifm-ifm3d/ifm3d_core'
ign_rviz_plugins: retval=1: RuntimeError Sphinx-build exited with return code '2'
rmf_visualization_schedule: retval=1: RuntimeError Sphinx-build exited with return code '2'
```
I can look at the logs on docs_build, for example, for rviz_common to see that the error was:
```
Exception occurred:
  File "/rosdoc2/venv/lib/python3.10/site-packages/breathe/renderer/sphinxrenderer.py", line 334, in create
    cls, name = DomainDirectiveFactory.cpp_classes[args[0]]  # type: ignore
KeyError: 'property'
```

During the active run, the console (Jenkins) output looks like this excerpt:
```
21:53:08 Begin processing geometry2
21:53:13 Package joint_state_publisher_gui succeeded in 10.981 seconds
[rosdoc2_scan] [INFO] joint_state_publisher_gui successful (1412/1440)
21:53:13 Begin processing system_modes_msgs
21:53:17 Package geometry2 succeeded in 8.858 seconds
[rosdoc2_scan] [INFO] geometry2 successful (1413/1440)
21:53:17 Begin processing mola_test_datasets
21:53:31 Package system_modes_msgs succeeded in 17.519 seconds
[rosdoc2_scan] [INFO] system_modes_msgs successful (1414/1440)
21:53:31 Begin processing ament_cmake_test
21:53:32 Package mola_test_datasets succeeded in 14.247 seconds
[rosdoc2_scan] [INFO] mola_test_datasets successful (1415/1440)
```
A few notes on the code:

- I had to abandon running sphinx-build directly as a python module rather than in a subprocess, because exhale when it dies does an abrupt system exit without cleanup, which tends to hang the process pool since I cannot catch it.
- I replaced the pipx tests with a venv test. There does not seem to be any interest in promoting pipx in the ros2 community, while venv is important for development, and the install requirements can be different if not starting from a ros2 enabled system.

I've done additional work on the whole area of locating packages to document in the last couple of weeks. I am now firmly convinced that colcon is the wrong tool for this use in rosdoc2, and I have [replaced it](https://github.com/rkent/rosdoc2/commit/b47e3a6f699618bc0d9a613a5d131327fec66588) with catkin's ```find_packages_allowing_duplicates``` in my fork. I'll eventually update this PR with that change, but for now please hold this PR.

Here's some background. colcon seems to be designed to find packages for building. Fine, but that is not what we are doing. Specifically (in addition to the complications mentioned above) it additionally scans CMakeLists.txt to try to find what it thinks is a package. Fine for building, not good for us since we need a real package.xml to process. So colcon is finding packages that we can't actually process, so it just ends up causing errors (like the ```urdfdom_headers: retval=1: RuntimeError Failed to find a ROS package at given path 'rosindex/_plugins_data/checkout/urdfdom_headers/github-ros-urdfdom_headers/urdfdom_headers'``` above).

I've fixed that in [my fork](http://rosdoc2.rosdabbler.com/urdfdom_headers/), but that fix (which would partially be at the buildfarm level) needs more tweaking. In any case, you have to replace colcon with catkin to eliminate nuisance errors.


Fix patch:
diff --git a/rosdoc2/verbs/build/builders/sphinx_builder.py b/rosdoc2/verbs/build/builders/sphinx_builder.py
index b6282ea8..e72d6b94 100644
--- a/rosdoc2/verbs/build/builders/sphinx_builder.py
+++ b/rosdoc2/verbs/build/builders/sphinx_builder.py
@@ -65,6 +65,42 @@ def ensure_global(name, default):
 ensure_global('release', "{package.version}")
 ensure_global('version', "{package_version_short}")
 
+# Remove any unsupported extensions
+allowed_extensions = set((
+    # Shipped with sphinx
+    'sphinx.ext.autodoc',
+    'sphinx.ext.autosectionlabel',
+    'sphinx.ext.autosummary',
+    'sphinx.ext.coverage',
+    'sphinx.ext.doctest',
+    'sphinx.ext.duration',
+    'sphinx.ext.extlinks',
+    'sphinx.ext.githubpages',
+    'sphinx.ext.graphviz',
+    'sphinx.ext.ifconfig',
+    'sphinx.ext.imgconverter',
+    'sphinx.ext.inheritance_diagram',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.linkcode',
+    'sphinx.ext.napoleon',
+    'sphinx.ext.todo',
+    'sphinx.ext.viewcode',
+    # Sphinx-included math extensions
+    'sphinx.ext.imgmath',
+    'sphinx.ext.mathjax',
+    # Installed by us
+    'breathe',
+    'exhale',
+    'myst_parser',
+    'sphinx_rtd_theme',
+))
+for extension in extensions[:]:
+    if extension not in allowed_extensions:
+        print(f'[rosdoc2] *** Warning *** removing extension "{{extension}}", not supported')
+        extensions.remove(extension)
+if extensions:
+    print(f'[rosdoc2] user conf.py specified allowed extensions: {{extensions}}')
+
 if rosdoc2_settings.get('enable_autodoc', True):
     print('[rosdoc2] enabling autodoc', file=sys.stderr)
     extensions.append('sphinx.ext.autodoc')


Test patch:
diff --git a/test/packages/basic_cpp/doc/conf.py b/test/packages/basic_cpp/doc/conf.py
index a6c7e47b..d5b595b9 100644
--- a/test/packages/basic_cpp/doc/conf.py
+++ b/test/packages/basic_cpp/doc/conf.py
@@ -41,6 +41,29 @@
 ## See the rosdoc2_settings below for some options on avoiding that.
 extensions = [
     'sphinx_rtd_theme',
+    'i_do_not_exist',
+    # Shipped with sphinx
+    'sphinx.ext.autodoc',
+    'sphinx.ext.autosectionlabel',
+    'sphinx.ext.autosummary',
+    'sphinx.ext.coverage',
+    'sphinx.ext.doctest',
+    'sphinx.ext.duration',
+    'sphinx.ext.extlinks',
+    'sphinx.ext.githubpages',
+    'sphinx.ext.graphviz',
+    'sphinx.ext.ifconfig',
+    'sphinx.ext.imgconverter',
+    'sphinx.ext.inheritance_diagram',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.linkcode',
+    'sphinx.ext.napoleon',
+    'sphinx.ext.todo',
+    'sphinx.ext.viewcode',
+    # Sphinx-included math extensions
+    'sphinx.ext.imgmath',
+    'sphinx.ext.mathjax',
+
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -126,3 +149,8 @@
     ## Support markdown
     # 'support_markdown': True,
 }
+
+## This function is required by the linkcode extension.
+## See https://www.sphinx-doc.org/en/master/usage/extensions/linkcode.html
+def linkcode_resolve(domain, info):
+    return None