Package: pyrollup
pyrollupSimple mechanism to rollup API symbols to a Python module from its submodulesBackgroundConventionally, a Python package's public API symbols (or at least a subset) are exported from the top-level package. These symbols are typically imported from various submodules in the module hierarchy. This leads to a rather tedious listing of symbols exported (via a module's__init__.py) at each level.For example:from.submodule_aimportClassA,ClassA2from.submodule_bimportClassBfrom.submodule_cimportClassC__all__=[ClassA,ClassA2ClassB,ClassC,]The definition of__all__can be omitted, although some schools of thought encourage its use.Problem statementWith the above approach, the burden of which symbols are considered public for a given submodule falls on the containing module importing them. In addition,__all__must be maintained alongside symbol imports, potentially resulting in duplication.This can become problematic for complex projects with many nested symbols imported at the top-level package. For example:from.submodule_aimportClassA,ClassA2from.submodule_a.submodule_a1importClassA1_1,ClassA1_2from.submodule_a.submodule_a2importClassA2_1,ClassA2_2from.submodule_a.submodule_a3importClassA3_1,ClassA3_2from.submodule_bimportClassBfrom.submodule_cimportClassC__all__=[ClassA,ClassA2,ClassA1_1,ClassA1_2,ClassA2_1,ClassA2_2,ClassA3_1,ClassA3_2,ClassB,ClassC,]In addition, submodules must similarly maintain their own imports and/or definition of__all__if they export symbols from their submodules.Proposed solutionIdeally, a module should "rollup" public symbols from its submodules, letting each submodule decide which symbols those are. Submodules should therefore optionally provide an allow-list and block-list to describe which public symbols should be propagated to the parent module.In other words, a given module should have ownership of:Its own public symbols (Python convention)Defined by__all__Which of those public symbols should be propagated to parent modules (functionality provided bypyrollup)Defined by__rollup__: allow-list, defaulting to__all____nrollup__: block-list, defaulting to[]Then, the example above can be modified as:frompyrollupimportrollup# import submodulesfrom.importsubmodule_a,submodule_b,submodule_c# import public symbols from each submodulefrom.submodule_aimport*from.submodule_bimport*from.submodule_cimport*# export public symbols from each submodule, filtered by allow-list/block-list__all__=rollup(submodule_a,submodule_b,submodule_c)This allows a project with a complex module hierarchy to flexibly propagate public symbols from wherever they are defined to the top-level package.DownsidesStatic analysis tools are unable to evaluate the value of__all__since with this approach it is computed dynamically upon import. Therefore, code documentation generator tools using static analysis (e.g.autoapiorautodoc2) will fail to detect a module's public symbols.Tools using the traditional approach of importing the package for which documentation is being generated (e.g. Sphinx's built-inautodoc) should have no problem with such dynamic imports.It is possible to use a "hybrid" approach; the author has had some success with extendingautodoc2to work with dynamically-evaluated__all__.TODO: document and generalize this solution