Package: paramparse
IntroductionSectionsHierarchical nestingReplicationCommandsApplicationsPlaceholdersIntroductionParamparse is a lightweight argparse wrapper to allow hierarchically nested class-based parameters suitable for automatic code analysis / intellisense  in advanced IDEs likePycharm.It also provides a unified parameter specification protocol that can be used to provide parameter values through both text files and command line.Please refer to the included parameter skeleton of a large multi object tracking project for which the functionality included in this package was originally developed.
It provides an excellent example of a highly modular project with deeply nested and shared modules.For example, this is one of the deeper instances of module nesting in this example:Params->Trainer->Target->Templates->Siamese->SiamFC->DesignParamsParameter for this configuration can be provided as:trainer.target.templates.siamese.siamfc.design.exemplar_sz=150Specifying multiple parameters for a deeply nested module can quickly become cumbersome especially from command line.
The package thus provides a way to group parameters from the same module using the@identifier.
An example is provided inexample/cfg/params.cfg.
Note that the indentation used in this file is only for ease of human parsing and is not needed as this system of grouping also works from command line.
Example commands are inexample/commands.md.The@identifier specifies a prefixpfto be added to all subsequent arguments so thatarg_nameis then treated aspf.arg_name.Assumingpf=arg1.arg2, following flavors of@identifier usage are supported:usageeffectpf@arg3resetpftoarg3arg3@resetpfto empty@@arg3addarg3topfarg1.arg2.arg3@@@arg3remove rightmost component ofpfand addarg3arg1.arg3@@@remove rightmost component ofpfarg1Usage of the package is very simple and involves callingparamparse.processas demonstrated inexample/main.py.__More detailed usage examples are available inDeep MDPfor which this parser was originally designed.https://github.com/abhineet123/deep_mdp__It also provides three converter functionsfrom_parser,from_dictandfrom_functionthat can create a parameter class compatible with this package from existing parameters inargparse.ArgumentParseranddictformats or using a function's keyword arguments respectively.
The generated class code is either writen to a python source file whose name can be specified as the second argument (defaults toParams.py) or copied to clipboard ifto_clipboard=1is provided (requirespyperclip).Theprocessfunction does type inference from the default value of each param but also supports extracting the type fromrestructuredText/pycharm type docstring(as generated by the converter functions) if it is provided.Note:paramparseuses the reserved parametercfgto specify paths to text files containing parameter values.
If an existing argparse or dict object to be converted intoparamparseclass already has acfgfield used for some other purpose, it will conflict with the parser so please rename this field before or after converting but before runningparamparse.process.Usage of converter functions is demonstrated inexample/utils_demo.pyRunpython3 main.py --hfrom the example folder to see the hierarchical help documentation generated by argparse.Apart from the hierarchical nesting and parameter grouping, an important utility ofparamparseis in the class based representation that allows automated code analysis, navigation and refactoring in IDEs like Pycharm that is not possible when using vanillaargparse.ArgumentParserordict.SectionsSections provide a means to read a cfg file selectively rather than in its entirety.Each section is a named group of consecutive lines within a cfg file that is preceded and succeeded by a line starting with two or more#:preceding line specifies the name of the sectionsucceeding line can either specify the name of the next section or simply mark the end of the current section if it is namelesslines without an explicit section are considered to be in an implicitcommonsection that is always read while parsing a cfg (in addition to the specified sections)Hierarchical nestingSections can be nested to any number of hierarchical levelslevel at which a section lies is determined by the number of#preceding its name##denotes level 1,###denotes level 2 and so onlines starting with a single#are regarded as comments and ignoredeach section at level 2 or greater (i.e. with 3 or more#preceding its name) is considered to be nested within the nearest section before it that is one level above ita section can have multiple childrena section can only have a single parenta section can be included in a command only if all its ancestors have also been includedsections nested within different hierarchies in the same cfg file can have identical namesReplicationMultiple sections can be started in a single line by specifying their names separated by commasall lines in such a multi-name section (inluding any section hierarchies therein) are replicated for each one of the namesthis can reduce redundancy when specifying sections that differ only in ways that can be derived from their names usingplaceholderslong lists of numeric names can be specified succinctly usingrange()andirange()placeholders that can be used with 1, 2 or 3 arguments similar to the pythonrangefunction, withirangebeing inclusive with respect to its upper boundmultiple disjointed ranges can be joined  with each other as well as with manually specified lists using+operator, e,g. range(3, 6)+irange(10, 13)+21,22 = 3,4,5,10,11,12,13,21,22CommandsSections are specified in a command by following the name of the cfg file with a colon (:) and a list of section names also separated with colonsname of a parent section should precede that of its childrenif the cfg file has multiple identically-named sections in different hierarchies and only one of them is to be read (while reading all ancestors of more than one of them), its name should be separated from its parent by a hyphen (-) instead of a:if a cfg file needs to be read again with a different section hierarchy, name of the root section of this hierarchy can be prefixed by++or+++++causes the common sections to be read again in addition to the new section hierarchy+++causes the common sections to be be skipped so that only the new section hierarchy is readApplicationsSections can have many uses:each section (or hierarchy thereof) can be used to specify the parameter values needed to run the script in a given configuration (or variants thereof)sections can be given short and intuitive names to quickly figure out exactly which configuration any given command corresponds tosections can make it very easy to switch between configurations with minimal changes to the commandshierarchical nesting of sections makes the relations between the different configurations explicit and easier to keep track ofplaceholders allow the names and values of parameters within a given section to be derived from the names of the section and its parents.PlaceholdersPlaceholders are reserved keywords that or variables will be replaced by their respective texts while parsing a cfg file.
Paramparse supports the following placeholders:placeholderreplacement%N%/__name__name of the section%N<i>%/__name<i>__ithcomponent of the section name if that name is treated as being composed of underscore-separated components (iis 0-based and the enclosing<>is not part of the placeholder, e.g. with section name =ab_cd_12_34%N0%,%N1%,%N2%and%N3%respectively denoteab,cd,12and34)%N<i>*%/__name<i>*__all components of the section name starting with theithcomponent joined with underscores (i>= 1, e.g. with section name =ab_cd_12_34,%N1*%and%N2*%respectively  denotecd_12_34and12_34)%P%/__parent__name of the parent of the section%GP%/__g_parent__name of the grandparent of the section%GGP%/__gg_parent__name of the great grandparent of the section%R%/__root__name of the root section of the section's hierarchy%F%/__full__names of the section and its parent separated by an underscore%PF%/__parent_full__names of the section's parent and its grand parent separated by an underscore%GF%/__g_full__names of the section, its parent and its grandparent separated by underscores%GGF%/__gg_full__names of the section, its parent, its grandparent and its great grandparent separated by underscores%RI%/__ratio__name of the section interpreted as a number and divided by 100%PRI%/__parent_ratio__same as__ratio__but with parent's name%GPRI%/__g_parent_ratio__same as__ratio__but with grandparent's name%GGPRI%/__gg_parent_ratio__same as__ratio__but with great grandparent's name%L%/__list__name of the section converted into a list by splitting it with underscores (i.e. each component of its name separated by an underscore becomes an element of the list)%PL%/__parent_list__same as__list__but with parent's name%GPL%/__g_parent_list__same as__list__but with grandparent's name%GGPL%/__gg_parent_list__same as__list__but with great grandparent's name%LRI%/__list_ratio__same as__list__with each list member interpreted as a number and divided by 100%PLRI%/__parent_list_ratio__same as__list_ratio__but with parent's name%GPLRI%/__g_parent_list_ratio__same as__list_ratio__but with grandparent's name%GGPLRI%/__gg_parent_list_ratio__same as__list_ratio__but with great grandparent's name%RA%/__range__name of the section converted into a list by extracting 1, 2 or 3 underscore separated numbers and using them as arguments to the pythonrangefunction%PRA%/__parent_range__same as__range__but with parent's name%GPRA%/__g_parent_range__same as__range__but with grandparent's name%GGPRA%/__gg_parent_range__same as__range__but with great grandparent's namerange()generate a list with 1, 2 or 3 comma separated arguments within the parenthesis using similar syntax as the pythonrangefunctionirange()same asrange()but with inclusive upper bound