Package: tomlparams
TOMLParamsTOML-based parameter files made betterKey featuresSimple externalization of parameters in one or more TOML files.Supports loading, saving, default values, parameter name checking
and optional parameter type checking.Parameters available through attribute lookup or dictionary lookup.Supports hierarchical file inclusion with overriding.Support for using environment variables to select parameter set (as well as API).Default values can be specified in code (as Python dictionaries) or in a TOML file.Full support for parameter hierarchy (using TOML Tables),
still with attribute and dictionary lookup.Support for writing consolidated parameters as TOML after hierarchical inclusion
and resolution.Can be subclassed and other attributes can be used without affecting TOML file writing.InstallationFrom PyPI:python -m pip install -U tomlparamsSource installation from Github:python -m pip install -U git+ssh://git@github.com/gofcoe/tomlparams.gitSample UsageThe Simplest Case (Defaults Only)A TOMLParams object is initialized with a set of default values for all parameters.
The defaults can be supplied as a Python dictionary from a TOML file, and all
values should be TOML-serializable (so noNonevalues).fromdatetimeimportdate,datetimefromtomlparamsimportTOMLParamsdefaults={'start_date':datetime.date(2024,1,1),'run_days':366,'tolerance':0.0001,'log':True,'locale':'en_GB','critical_event_time':datetime.datetime(2024,7,31,3,22,22),'logging':{'format':'.csv','events':['financial','telecoms']}}params=TOMLParams(defaults=defaults,name='defaults')The special value for'defaults'for name specifies that the default values
should be used and nothing should be read from a custom TOML parameters file.Parameters are stored as attributes and can be accessed using.style attribute
lookup or dictionary-style lookup with[]:>>>params.run_days366>>>params['start_date']datetime.date(2024,1,1)>>>params.logging.format'.csv'>>>params['logging']['events']['financial','telecoms']Using a TOML file to override some parameter valuesIf thenameis set to anything other than'defaults', that will be
used as the stem name of TOML file in which to look for override parameters.
This defaults tobase(forbase.toml). The directory in which the system
looks for this TOML file can be set withstandard_params_dir, and if not specified
will default to~/tomlparams(i.e.tomlparamsin the user's home directory).So ifbase.tomlexists in the current working directory, and containsstart_date=2024-03-03[logging]format='.json'Then we will have the following (using the samedefaultsdict as before)>>>params=TOMLParams(defaults=defaults,name='defaults')>>>params.run_days366>>>params['start_date']datetime.date(2024,3,3)>>>params.logging.format'.json'>>>params['logging']['events']['financial','telecoms']>>>paramsTOMLParams(start_date:datetime.date(2024,3,3),run_days:366,tolerance:0.0001,logging:ParamsGroup(format:'.json',events:['financial','telecoms']),locale:'en_GB',critical_event_time:datetime.datetime(2024,7,31,3,22,22))Notice how the two values inbase.tomlhave overridden the defaults
(start_dateandlogging.format), but that other parameters,
includinglogging.events, retained their values.Setting defaults from a TOML FileWe can set the defaults from a TOML file instead of a Python dictionary by
simply giving the path to the TOML file as the value fordefaultswhen initializingTOMLParams. If it's an absolute path, it will be read, but if it's a relative path,
the system will look in thestandard_params_dir.This TOML file:start_date=2024-01-01run_days=366tolerance=0.0001locale="en_GB"critical_event_time=2024-07-31 03:22:22[logging]format=".csv"events=["financial","telecoms",]is equivalent to the dict used above.and if stored in./defaults.tomlcan be used with>>>params=TOMLParams(defaults='defaults',standard_params_dir='.')Setting a custom TOML file name and Parameter (Key) CheckingOnly parameters indefaultsare allowed to exist in other TOML parameter files.
an exception is raised if any unexpected values are found in the TOML file.For example, ifnewparams.tomlis:new_param='this will go badly'and we repeated the use the same defaults, passingnewparamsas thenamewe get the following error:>>> params = TOMLParams(
     defaults='defaults',
     name='newparam',
     standard_params_dir='.'
)
*** ERROR: The following issues were found:
 Bad key at root level - key: new_paramHierarchical InclusionA special key,includemay be used as the first line of a TOML file,
and may be set to either single string name
(the stem name of a TOML file to include),
or a list of such names.include=['one','two']Inclusions are processed left-to-right, before the rest of the values in the
file with theincludestatement, but each include is only processed once,
the first time it is encountered, with newer values always overriding old ones.So ifbase.toml(or other named TOML file used)
starts with thisincludeline, and neither ofoneandtwohas any further inclusions, the order of parameter setting will be:values from defaultsvalues fromone.tomlvalues fromtwo.tomlany other valuesIfoneandtwoboth start withinclude='three'then the order of processing will be:values from defaultsvalues fromthree.toml(from its inclusion byone.toml)values fromone.tomlvalues fromtwo.toml(three.tomlwill isnotincluded a second time)any other valuesCircular inclusion does not cause a problem, because each file is only
ever included once, but is potentially confusing for the reader, so is
not recommended.Unlessverboseis set toFalsewhen initializingTOMLParams, any
TOML files processed are reported, in order of inclusion, listing full
paths.So running this code (available in the source repo
asexamples/readme/readme4.py):fromtomlparamsimportTOMLParamsparams=TOMLParams(defaults='defaults2',name='hier',standard_params_dir='.')print(repr(params))we see the following output:$ python readme4.py
Parameters set from: /home/sdf/tomlparams/examples/readme/three.toml
Parameters set from: /home/sdf/tomlparams/examples/readme/one.toml
Parameters set from: /home/sdf/tomlparams/examples/readme/two.toml
Parameters set from: /home/sdf/tomlparams/examples/readme/hier.toml
TOMLParams(
    a='hier',
    b='two',
    c='one',
    d='three',
    e='default',
    group=ParamsGroup(
        a='group hier',
        b='group two',
        c='group one',
        d='group three',
        e='group default',
        subgroup=ParamsGroup(
            a='subgroup hier',
            b='subgroup two',
            c='subgroup one',
            d='subgroup three',
            e='subgroup default'
        )
    )
)where (in order of parameter setting):defaults2.toml(used to specify defaults) is:# defaults2.tomla='default'b='default'c='default'd='default'e='default'[group]a='group default'b='group default'c='group default'd='group default'e='group default'[group.subgroup]a='subgroup default'b='subgroup default'c='subgroup default'd='subgroup default'e='subgroup default'These settings are overridden first bythree.toml:# three.tomla='three'b='three'c='three'd='three'[group]a='group three'b='group three'c='group three'd='group three'[group.subgroup]a='subgroup three'b='subgroup three'c='subgroup three'd='subgroup three'These are next overridden byone.toml:# one.tomlinclude='three'a='one'b='one'c='one'[group]a='group one'b='group one'c='group one'[group.subgroup]a='subgroup one'b='subgroup one'c='subgroup one'In turn, these are overridden bytwo.toml:# two.tomlinclude='three'a='two'b='two'[group]a='group two'b='group two'[group.subgroup]a='subgroup two'b='subgroup two'Finally, the parameters set inhier.tomlover-ride all others:# hier.tomlinclude=['one','two']a='hier'[group]a='group hier'[group.subgroup]a='subgroup hier'Writing out the consolidated TOML fileHierarchical file inclusion is powerful, and allows different sets of
parameters to be combined easily, but it can be hard for the reader to
know what the final set of parameters used is without thinking through
the inclusion hierarchy.  TOMLParams can write out a consolidated
parameter values finally used as a single TOML file containing them
all, without inclusions. The methodwrite_consolidated_tomlon
aTOMLParamsobject will do this:param.write_consolidated_toml('consolidated_params.toml')Determination of Name of TOML Parameters FileWhen aTOMLParamsobject is instantiated, the second
argument---name---determines which set of parameters is used.
Many options for setting this are supported. In determining which
parameters file to use, there are two relevant directories:thestandard parameters directorycan be set by passing in
a path asstandard_params_dir. If this is not specified,
explicitly,~/{params_name}will be used if theparams_nameargument has been set on instantiating TOMLParams,
or~/tomlparamsotherwise.theuser parameters directorycan be set by passing in a
path asuser_params_dir. If this is not specified,
explicitly,~/user{params_name}will be used if theparams_nameargument has been set on instantiating TOMLParams,
or~/usertomlparamsotherwise.Ifnameis set tofoo, TOMLParams will search forfoo.tomlin both the standard parameters directory and the user parameters
directory. If it is found in either it will be used. If it is not
found at all, an exception will be raised. If it is found in both
places, an exception will also be raised.(This behaviour is designed to allow code to supply standard
parameters, possibly in a read-only directory and for users
also to have their own parameters in another directory, while
not allowing either to take priority over the other, so that
which parameters are used is unambiguous.)Ifnameis set to either of the special valuesdefaultordefaults, the default values will be used for all parameters,
whether those were supplied as a Python dictionary or in a
TOML file (specified by thedefaultsargument).Ifnameis not supplied, or is passed asNone, TOMLParams
will look the value in an environment variable. That variable
defaults toTOMLPARAMS, but can be set by setting the argumentenv_varwhen instantiating aTOMLParamsobject.
If this environment variable is set tofoo,foo.tomlwill
be searched for in both parameters directories exactly as if it
had been provided as the value ofname.THe environment variable is particularly convenient when several
programs need to share a set of parameters. Most commonly, the
environment variable will be specified on the command line before
the Python command, e.g.TOMLPARAMS=bar python simulate.pyif the Python program usingTOMLParamsissimulate.py.
Alternatively, a value may be  export a shell start-up file,
or in a script, usingexport TOMLPARAMS=bar.Ifnameis not supplied, and the relevant environment variable
is not set, the value of the argumentbase_params_dirwill be
used; this defaults tobase.Idiomatic command-line useIf you want to usebase.tomlas the standard parameters, but allow
people to list the parameters file (stem) on the command line for a programsimulate.py, with~/simparamsas the standard parameters directory,
and defaults being read fromdefaults.tomlin that directory,
the usual procedure would be to use code like this:importsysfromtomlparamsimportTOMLParamsclassSimulate:def__init__(params,...)if__name__=='__main__:name=sys.argv[1]iflen(sys.argv)>1else'base'params=TOMLParams('defaults.toml',name,params_name='simparams')sim=Simulate(params,...)Environment VariablesAs noted above, there are two environment variables that can affect
the behaviour of TOMLParams.If thenameargument is not passed toTOMLParams(or is passed asNone)
the environment variableTOMLPARAMSwill be consulted (by default).
The name of the environment variable to consult can be changed by
passing a different value forenv_var.Any value for the variablecheck_typescan be overridden by setting
the environment variableTOMLPARAMSCHECKINGtowarn,errororoff.
A variable other thanTOMLPARAMSCHECKINGcan be specified using
the argumenttype_check_env_varwhen initializingTOMLParams.Type CheckingType checking is performed against the types of any values present in
the passed-indefaults. Three levels of action are configurable:WARN(default) — a warning is sent tostderrbut processing continues)ERROR— processing stops. All type-checking and key-checking errors are collected
and reported before exitOFF— no type checking. (Useful when using polymorphic parameters.)Type checking of any collections (e.g.list,setandtuple)
present indefaultsis also performed and the selected action taken
if a type appears in a TOML array that is not present in the
corresponding collection in the defaults.Action on a type checking mismatch can be configured in two ways:Via the environment variable specified by thetype_check_env_varsetting (defaults toTOMLPARAMSCHECKING)
with allowed levelswarn,error, andoff.Via thecheck_typessetting.The environment variable takes precedence over the setting where set.Command line tool tomlparamsAs part of installing the TOMLParams library, a command line tool
called tomlparams is made available. This provides a few functions,
the most important of which is the ability to copy usage examples
to your current working directory.Running the command by itself prints usage information:$tomlparams
TOMLParams

USAGE:tomlparamshelp---showthismessagetomlparamsversion---reportversionnumbertomlparamsexamples---copytheexamplesto./tomlparams_examples

Documentation:https://tomlparams.readthedocs.io/
Sourcecode:https://github.com/smartdatafoundry.com/tomlparams
Website:https://tomlparams.com

Installation:python-mpipinstall-UtomlparamsThe examples can be copied as follows:$ pwd
/home/sdf/tomlparams

$ tomlparams examples
Examples copied to /home/sdf/tomlparams/tomlparams_examples.Thereadmesubdirectory of the resultingtomlparams_examplescontains all the examples from thisREADME.mdfile.Documentation and API HelpThe documentation fortomlparamsis available attomlparams.readthedocs.io.
More detailed help on all the arguments available when initializingTOMLParamsis available from the documentation,
and also from the help system within Python using:>>>importtomlparams>>>help(tomlparams.TOMLParams)