Package: python-configuration
python-configurationA library to load configuration parameters hierarchically from multiple sources and formatsThis library is intended as a helper mechanism to load configuration files hierarchically.Supported FormatsThepython-configurationlibrary supports the following configuration formats and sources:Python filesDictionariesEnvironment variablesFilesystem pathsJSON filesINI filesdotenv type filesOptional support for:YAML files: requiresyamlTOML files: requirestomlifor Python < 3.11Azure Key Vault credentials: requiresazure-keyvaultAWS Secrets Manager credentials: requiresboto3GCP Secret Manager credentials: requiresgoogle-cloud-secret-managerHashicorp Vault credentials: requireshvacInstallingTo install the library:pipinstallpython-configurationTo include the optional TOML and/or YAML loaders, install the optional dependenciestomlandyaml. For example,pipinstallpython-configuration[toml,yaml]Without the optional dependencies, the TOML (Python < 3.11) and YAML loaders will not be available,
and attempting to use them will raise an exception.Getting startedpython-configurationconverts the various config types into dictionaries with dotted-based keys. For example, given this JSON configuration{"a":{"b":"value"}}We can use theconfig_from_jsonmethod to parse it:fromconfigimportconfig_from_jsoncfg=config_from_json("my_config_file.json",read_from_file=True)(Similar methods exist for all the other supported configuration formats (eg.config_from_toml, etc.).)We are then able to refer to the parameters in the config above using any of:cfg['a.b']cfg['a']['b']cfg['a'].bcfg.a.band extract specific data types such as dictionaries:cfg['a'].as_dict=={'b':'value'}This is particularly useful in order to isolate group parameters.
For example, with the JSON configuration{"database.host":"something","database.port":12345,"database.driver":"name","app.debug":true,"app.environment":"development","app.secrets":"super secret","logging":{"service":"service","token":"token","tags":"tags"}}one can retrieve the dictionaries ascfg.database.as_dict()cfg.app.as_dict()cfg.logging.as_dict()or simply asdict(cfg.database)dict(cfg.app)dict(cfg.logging)ConfigurationThere are two general types of objects in this library. The first one is theConfiguration, which represents a single config source.  The second is aConfigurationSetthat allows for multipleConfigurationobjects to be specified.Single ConfigPython FilesTo load a configuration from a Python module, theconfig_from_pythoncan be used.
The first parameter must be a Python module and can be specified as an absolute path to the Python file or as an importable module.Optional parameters are theprefixandseparator.  The following callconfig_from_python('foo.bar',prefix='CONFIG',separator='__')will read every variable in thefoo.barmodule that starts withCONFIG__and replace every occurrence of__with a.. For example,# foo.barCONFIG__AA__BB_C=1CONFIG__AA__BB__D=2CONF__AA__BB__D=3would result in the configuration{'aa.bb_c':1,'aa.bb.d':2,}Note that the single underscore inBB_Cis not replaced and the last line is not prefixed byCONFIG.DictionariesDictionaries are loaded withconfig_from_dictand are converted internally to a flatteneddict.{'a':{'b':'value'}}becomes{'a.b':'value'}Environment VariablesEnvironment variables starting withprefixcan be read withconfig_from_env:config_from_env(prefix,separator='_')Filesystem PathsFolders with files named asxxx.yyy.zzzcan be loaded with theconfig_from_pathfunction.  This format is useful to load mounted KubernetesConfigMapsorSecrets.JSON, INI, .env, YAML, TOMLJSON, INI, YAML, TOML files are loaded respectively withconfig_from_json,config_from_ini,config_from_dotenv,config_from_yaml, andconfig_from_toml.
The parameterread_from_filecontrols whether a string should be interpreted as a filename.CaveatsIn order forConfigurationobjects to act asdictand allow the syntaxdict(cfg), thekeys()method is implemented as the typicaldictkeys. Ifkeysis an element in the configurationcfgthen thedict(cfg)call will fail. In that case, it's necessary to use thecfg.as_dict()method to retrieve thedictrepresentation for theConfigurationobject.The same applies to the methodsvalues()anditems().Configuration SetsConfiguration sets are used to hierarchically load configurations and merge settings. Sets can be loaded by constructing aConfigurationSetobject directly or using the simplifiedconfigfunction.To construct aConfigurationSet, pass in as many of the simpleConfigurationobjects as needed:cfg=ConfigurationSet(config_from_env(prefix=PREFIX),config_from_json(path,read_from_file=True),config_from_dict(DICT),)The example above will read first from Environment variables prefixed withPREFIX, and fallback first to the JSON file atpath, and finally use the dictionaryDICT.Theconfigfunction simplifies loading sets by assuming some defaults.
The example above can also be obtained bycfg=config(('env',PREFIX),('json',path,True),('dict',DICT),)or, even simpler ifpathpoints to a file with a.jsonsuffix:cfg=config('env',path,DICT,prefix=PREFIX)Theconfigfunction automatically detects the following:extension.pyfor python modulesdot-separated python identifiers as a python module (e.g.foo.bar)extension.jsonfor JSON filesextension.yamlfor YAML filesextension.tomlfor TOML filesextension.inifor INI filesextension.envfor dotenv type filesfilesystem folders as Filesystem Pathsthe stringsenvorenvironmentfor Environment VariablesMerging ValuesConfigurationSetinstances are constructed by inspecting each configuration source, taking into account nested dictionaries, and merging at the most granular level.
For example, the instance obtained fromcfg = config(d1, d2)for the dictionaries belowd1={'sub':{'a':1,'b':4}}d2={'sub':{'b':2,'c':3}}is such thatcfg['sub']equals{'a':1,'b':4,'c':3}Note that the nested dictionaries of'sub'in each ofd1andd2do not overwrite each other, but are merged into a single dictionary with keys from bothd1andd2, giving priority to the values ofd1over those fromd2.CaveatsAs long as the data types are consistent across all the configurations that are part of aConfigurationSet, the behavior should be straightforward.  When different configuration objects are specified with competing data types, the first configuration to define the elements sets its datatype. For example, if in the example aboveelementis interpreted as adictfrom environment variables, but the JSON file specifies it as anything else besides a mapping, then the JSON value will be dropped automatically.Other FeaturesString InterpolationWhen setting theinterpolateparameter in anyConfigurationinstance, the library will perform a string interpolation step using thestr.formatsyntax.  In particular, this allows to format configuration values automatically:cfg=config_from_dict({"percentage":"{val:.3%}","with_sign":"{val:+f}","val":1.23456,},interpolate=True)assertcfg.val==1.23456assertcfg.with_sign=="+1.234560"assertcfg.percentage=="123.456%"ValidationValidation relies on thejsonchemalibrary, which is automatically installed using the extravalidation. To use it, call thevalidatemethod on anyConfigurationinstance in a manner similar to what is described on thejsonschemalibrary:schema={"type":"object","properties":{"price":{"type":"number"},"name":{"type":"string"},},}cfg=config_from_dict({"name":"Eggs","price":34.99})assertcfg.validate(schema)cfg=config_from_dict({"name":"Eggs","price":"Invalid"})assertnotcfg.validate(schema)# pass the `raise_on_error` parameter to get the traceback of validation failurescfg.validate(schema,raise_on_error=True)# ValidationError: 'Invalid' is not of type 'number'To use theformatfeature of thejsonschemalibrary, the extra dependencies must be installed separately as explained in the documentation ofjsonschema.fromjsonschemaimportDraft202012Validatorschema={"type":"object","properties":{"ip":{"format":"ipv4"},},}cfg=config_from_dict({"ip":"10.0.0.1"})assertcfg.validate(schema,format_checker=Draft202012Validator.FORMAT_CHECKER)cfg=config_from_dict({"ip":"10"})assertnotcfg.validate(schema,format_checker=Draft202012Validator.FORMAT_CHECKER)# with the `raise_on_error` parameter:c.validate(schema,raise_on_error=True,format_checker=Draft202012Validator.FORMAT_CHECKER)# ValidationError: '10' is not a 'ipv4'ExtrasTheconfig.contribpackage contains extra implementations of theConfigurationclass used for special cases. Currently the following are implemented:AzureKeyVaultConfigurationinconfig.contrib.azure, which takes Azure Key Vault
credentials into aConfiguration-compatible instance. To install the needed dependencies
executepipinstallpython-configuration[azure]AWSSecretsManagerConfigurationinconfig.contrib.aws, which takes AWS Secrets Manager
credentials into aConfiguration-compatible instance. To install the needed dependencies
executepipinstallpython-configuration[aws]GCPSecretManagerConfigurationinconfig.contrib.gcp, which takes GCP Secret Manager
credentials into aConfiguration-compatible instance. To install the needed dependencies
executepipinstallpython-configuration[gcp]HashicorpVaultConfigurationinconfig.contrib.vault, which takes Hashicorp Vault
credentials into aConfiguration-compatible instance. To install the needed dependencies
executepipinstallpython-configuration[vault]FeaturesLoad multiple configuration typesHierarchical configurationAbility to override with environment variablesMerge parameters from different configuration typesContributingIf you'd like to contribute, please fork the repository and use a feature branch. Pull requests are welcome.SeeCONTRIBUTING.mdfor the details.LinksRepository:https://github.com/tr11/python-configurationIssue tracker:https://github.com/tr11/python-configuration/issuesDocumentation:https://python-configuration.readthedocs.ioLicensingThe code in this project is licensed under MIT license.