Package: magiconfig
magiconfigConfigure Python with Python.Table of ContentsOverviewPhilosophyFeaturesArgumentParserConstructorparse_args(), parse_known_args()parse_config(config_name, config_obj, config_strict, namespace=None)set_config_options(**kwargs)copy_config_options(config_options)remove_config_options()write_config(namespace, filename, obj=None, attr_imports=None, class_imports=None, attr_reprs=None, class_reprs=None, strict=False)add_config_argument(arg, **kwargs)remove_config_argument(arg)add_config_only(*args, **kwargs)remove_config_only(arg)remove_argument(arg, keep=False)MagiConfigOptionsConstructorMagiConfigwrite(filename, config_obj, attr_imports=None, class_imports=None, attr_reprs=None, class_reprs=None, strict=False)join(other_config, prefer_other=False)MagiConfigErrorOtherSubparser aliasesConvenienceExamples1) Basic setup2) Subparsers3) Config-driven4) Scaling upInspirations(Created bygh-md-toc)Overviewmagiconfig is an extension of argparse that stops theconfiguration complexity clockby enabling users to configure Python with Python.
It provides all the power of Python to manipulate and compose configuration parameters,
bypassing the limitations of text-based configuration languages.PhilosophyThis module treats argparse as an engine that ultimately provides a namespace of attributes ("dests") to be consumed by user applications.
With magiconfig, these attributes can be provided by an importedMagiConfigPython object,
in addition to the usual command-line arguments.Featuresmagiconfig is compatible with both Python 2 and Python 3.
It provides a customArgumentParserclass, which is a drop-in replacement forargparse.ArgumentParser.
It also providesMagiConfig,MagiConfigOptions, andMagiConfigErrorclasses.
The precedence of parameter values is: command line > config file > defaults.ArgumentParserThe API of this class is extended with several additional functions to manage config settings, as well as to provide other useful operations.ConstructorThe constructor supports several additional options:config_options: takes an instance ofMagiConfigOptions; default =None(falls back to standard argparse behavior)config_only_help: include config-only args in the help message (seeadd_config_argument()); default =Trueparse_args(), parse_known_args()These function interfaces are unchanged from argparse, but they return aMagiConfigobject.
If an inputnamespaceargument is provided but is not of typeMagiConfig, a conversion will be attempted.parse_config(config_name, config_obj, config_strict, namespace=None)This is mainly an internal function used inparse_known_args(), but like that function, it could also be used standalone.config_name: name of config file to importconfig_obj: name of config object inside config fileconfig_strict: whether to reject imported config object if it has unknown attributesnamespace:Namespaceobject to append to, if anyRaisesMagiConfigErrorif any required config-only arguments are missing or ifconfig_strictisTrueand the imported config has unknown attributes.set_config_options(**kwargs)This function allows changing the config options after the parser is initialized.
It accepts all parameters that can be used to construct an instance ofMagiConfigOptions.
RaisesMagiConfigErrorif any other parameters are provided.copy_config_options(config_options)This function allows copying anotherMagiConfigOptionsobject to be used in this parser.
RaisesMagiConfigErrorif an object with any unknown parameters is provided.remove_config_options()This function allows removing all config options from the parser.write_config(namespace, filename, obj=None, attr_imports=None, class_imports=None, attr_reprs=None, class_reprs=None, strict=False)namespace:MagiConfigobject to be writtenfilename: name of file to writeobj: name of theMagiConfigobject in the file (default: class memberconfig_options.objor"config"if noconfig_optionsspecified)attr_imports: dictionary with key = attribute name, value = function returning a string ofimportstatementsclass_imports: dictionary with key = class type, value = function returning a string ofimportstatementsattr_reprs: dictionary with key = attribute name, value = function returning arepr-style stringclass_reprs: dictionary with key = class type, value = function returning arepr-style stringstrict: for each attribute, check if callingeval()on therepr-style string returns the original valueThis function can be used to preserve the state of the configuration after any command-line modifications (seeExample 1).
By default, the class module and name of each entry in the configuration are used to determine if import statements are needed,
andrepr()is used to provide a representation of the entry from which it can be reconstructed.
(If an entry is aCollectionorMappingtype, its entries are also inspected for imports. For otherContainertypes, their entries are expected if they can be obtained viavars().)
If these defaults are not correct for a given entry or a given class type, they can be overridden by providing custom functions using the optionsattr_imports,class_imports,attr_reprs,class_reprsas described above.
The order of precedence is:attr_* > class_* > default.
Strict checking of the validity ofrepr-style strings is currently disabled by default, but may be enabled by default in the next major release (3.0.0).add_config_argument(arg, **kwargs)This interface allows adding a dest (arg) that is only provided by the config, not by a command-line argument.
The supportedkwargsare:default,type,choices,required,help(an appropriate subset ofargparse.ArgumentParser.add_argument()).remove_config_argument(arg)arg: name of config-only arg to removeRaisesKeyErrorif arg is not found in the list of config-only args.add_config_only(*args, **kwargs)This function is deprecated and will be removed in magiconfig 3.0.0; please switch toadd_config_argument().This interface allows adding dests that are only provided by the config, not by command-line arguments.args: no default value, not required**kwargs: default value OR required (value=None)RaisesMagiConfigErrorif any dests have already been used by arguments (actions) added to the parser.
Similarly,add_argument()now raisesArgumentErrorif it specifies a dest that has already been added as config-only by this function.remove_config_only(arg)This function is deprecated and will be removed in magiconfig 3.0.0; please switch toremove_config_argument().arg: name of config-only arg to removeRaisesKeyErrorif arg is not found in the list of config-only args.remove_argument(arg, keep=False)This function allows removing a single argument (a missing feature in argparse).arg: flag (for optional arguments) or dest (for positional arguments)keep: for optional arguments,Truewill remove just the single specified flagarg(the entire action is removed only if it has no remaining flags);False(default) always removes the action associated with the flagfor positional arguments, all positional actions with the specified dest are removed (but not optional arguments with that dest)Exits witherror()if an unknown argument is provided.MagiConfigOptionsThis simple class stores options related to the use of configs in theArgumentParser.Constructorargs: the command-line arguments to indicate the config file (default:["-C", "--config"])help: custom help message for config args (optional)required: if the config arg is required when parsing (default:False)default: default value for the config file name (default:None)dest: destination for config arg (default:"config")obj: name of theMagiConfigobject to be imported from the config file (default:"config")obj_args: command-line arguments to indicate the name of the object to be imported (optional)obj_help: custom help message for obj args (optional)obj_dest: destination for obj arg (default:"obj")strict: reject imported config object if it has unknown attributes (default:False)strict_args: optional command-line arguments to toggle strictnessifstrictabove is set toFalse, providing an arg will toggle it toTrue; if set toTrue, will toggle it toFalsestrict_help: custom help message for strict args (optional)strict_dest: destination for strict arg (default:"strict")The values forargs,obj_args, andstrict_argscan be positional arguments (rather than the optional arguments shown here).MagiConfigThis class extendsargparse.Namespaceto add a few useful methods.
It is used both as the input object in config files and as the output object ofArgumentParser.write(filename, config_obj, attr_imports=None, class_imports=None, attr_reprs=None, class_reprs=None, strict=False)filename: name of file to writeconfig_obj: name ofMagiConfigobject in fileother options: see documentation forArgumentParser.write_config()join(other_config, prefer_other=False)other_config: otherMagiConfigobject to mergeprefer_other: prefer values from other config, if dest is present in both configs (default: prefer this config)getattr(),setattr()These class methods are extended to handle nested config objects automatically.
Any nonexistent intermediate objects are initialized asMagiConfiginstances. Example:frommagiconfigimportMagiConfigx=MagiConfig()setattr(x,"y.z","test")print(x)returns:MagiConfig(y=MagiConfig(z='test'))This enables obtaining dests from nested configs by using dots in the dest names.MagiConfigErrorThis class derives fromExceptionand denotes magiconfig-specific errors.OtherSubparser aliases_SubParsersAction.add_parseris modified to backport the use of subparser aliases to Python 2.ConvenienceAll public classes and constants from argparse are added to the magiconfig namespace for easier drop-in usage.A classArgumentDefaultsRawHelpFormatteris defined to present help messages with default values and without line wrapping (fromConfigArgParse).Examples1) Basic setupThe simple script inexamples/example1.pydemonstrates the different ways to set values, as well as some of the features of magiconfig.The help printout for the arguments defined in the script:usage: example1.py [-h] [-C CONFIG] [-f FOO] -b BAR [-i]

optional arguments:
  -h, --help            show this help message and exit
  -C CONFIG, --config CONFIG
                        name of config file to import (w/ object: config)
  -f FOO, --foo FOO     foo arg
  -b BAR, --bar BAR     bar arg
  -i, --ipsum           ipsum argWhen the script is run as follows:python3 examples/example1.py -C examples/config1.py --foo 'foo'It prints the resulting namespace:MagiConfig(bar=3.0, foo='foo', ipsum=False)Here, thebarargument is set by the config fileexamples/config1.py,
thefooargument is set on the command line, and theipsumargument retains its default value.The script also writes the final namespace into a config fileexamples/config1_out.py:frommagiconfigimportMagiConfigconfig=MagiConfig()config.bar=3.0config.foo='foo'config.ipsum=FalseWith this config file, the script can be rerun to produce the same output without
the need to specify any other command-line arguments:python examples/example1.py -C examples/config1_out.py.2) SubparsersThe script inexamples/example2.pydemonstrates how a common config fileexamples/config2.pycan be used with multiple subparsers.The parser has two modes defined,one(with an argumentfoo)
andtwo(with an argumentbar).
Each subparser mode specifies a different config object;
in this case, each of these config objects is a member of a top-level config object.The script can be run in each mode with the same input config file:> python examples/example2.py one -C examples/config2.py
MagiConfig(foo='foo')
> python examples/example2.py two -C examples/config2.py
MagiConfig(bar=2.0)3) Config-drivenIn a config-driven script, it may be desirable to encapsulate many parameters only in the config file,
while supporting only parameters related to running the script as command-line arguments.
The script inexamples/example3.pyis an example.It shows how an organized schema with different categories and parameters can be defined and transmitted to the parser.
This allows the parser to use strict mode to validate input configurations, rejecting any config with unknown parameters.
The config fileexamples/config3.pycan be used with the script:> python examples/example3.py -C examples/config3.py -v
MagiConfig(dataset=MagiConfig(background='background', path='/data', signal='signal'), hyper=MagiConfig(learning_rate=0.1, loss='log'), training=MagiConfig(size=0.5, weights=[1, 1]), verbose=True)This example also shows how config-only arguments can be given default values or marked as required. These attributes are reflected in the help message:> python examples/example3.py --help
usage: example3.py [-h] [-C CONFIG] [-v]

optional arguments:
  -h, --help            show this help message and exit
  -C CONFIG, --config CONFIG
                        name of config file to import (w/ object: config) (default: None)
  -v, --verbose         enable verbose output (default: False)

config-only arguments:
  dataset.background    (required)
  dataset.path            (default: /data)
  dataset.signal        (required)
  hyper.learning_rate
  hyper.loss
  training.size
  training.weights4) Scaling upWhen scaling up an application to handle a large number of possible inputs,
a typical pattern is that some of the parameters are common,
while other parameters may be unique to each input.
Rather than requiring a separate config file for each possible input,
all of the config objects can be generated within a single Python file.
The script inexamples/example4.pyallows the config object name to be specified on the command line;
other config objects in the config file are just ignored.The help message for this script is:usage: example4.py [-h] [-C CONFIG] [-O OBJ] [-f FOO] -b BAR -i INPUT

optional arguments:
  -h, --help            show this help message and exit
  -C CONFIG, --config CONFIG
                        name of config file to import (w/ object from -O,--obj) (default: None)
  -O OBJ, --obj OBJ     name of object to import from config file (default: config)
  -f FOO, --foo FOO     foo arg (default: lorem)
  -b BAR, --bar BAR     bar arg (default: None)
  -i INPUT, --input INPUT
                        input arg (default: None)The script can be run with different inputs all contained inexamples/config4.py:> python3 examples/example4.py -C examples/config4.py -O config.a
MagiConfig(bar=3.0, foo='foo', input='a')
> python3 examples/example4.py -C examples/config4.py -O config.b
MagiConfig(bar=3.0, foo='foo', input='b')InspirationsThis project owes inspiration (and in some cases code) to:ConfigArgParseconfiguratiWMCore