Package: essentials-configuration
Python configuration utilitiesImplementation of key-value pair based configuration for Python applications.Features:support for most common sources of application settingssupport for overriding settings in sequencesupport for nested structures and lists, using attribute notationstrategy to use environment specific settingsfeatures to handle secrets and values stored in the user folder, for local
developmentfeatures to support validation of configuration items, for example usingpydantic, or user defined classesThis library is freely inspired by .NET CoreMicrosoft.Extensions.Configuration(ref.MSDN documentation,Microsoft Extensions Configuration Deep Dive).The main class is influenced by Luciano Ramalho`s example of
JSON structure explorer using attribute notation, in his bookFluent Python.Overviewessentials-configurationprovides a way to handle configuration roots
composed of several layers, such as configuration files and environment
variables. Layers are applied in order and can override each others' values,
enabling different scenarios like configuration by environment and system
instance.Supported sources:tomlfilesyamlfilesjsonfilesinifilesenvironment variablessecrets stored in the user folder, for development purposedictionarieskeys and valuesAzure Key Vault, usingessentials-configuration-keyvaultcustom sources, implementing theConfigurationSourceinterfaceInstallationpipinstallessentials-configurationTo install with support forYAMLconfiguration files:pip install essentials-configuration[yaml]To install with support forYAMLconfiguration files and theCLIto handle
user secrets:pip install essentials-configuration[full]ExtensionsAzure Key Vault secrets configuration source:essentials-configuration-keyvaultExamplesPlease read the list of examples in theexamples folder. Below
are reported some of the examples that are tested in this repository.TOML filefromconfig.commonimportConfigurationBuilderfromconfig.envimportEnvVarsfromconfig.tomlimportTOMLFilebuilder=ConfigurationBuilder(TOMLFile("settings.toml"),EnvVars(prefix="APP_"))config=builder.build()For example, if the TOML file contains the following contents:title="TOML Example"[owner]name="Tom Preston-Werner"And the environment has a variable such asAPP_OWNER__NAME=AAA, the owner
name from the TOML file gets overridden by the env variable:>>>config<Configuration{'title':'...','owner':'...'}>>>>config.title'TOML Example'>>>config.owner.name'AAA'JSON file and environment variablesIn the following example, configuration values will include the structure
inside the filesettings.jsonand environment variables whose name starts
with "APP_". Settings are applied in order, so environment variables with
matching name override values from thejsonfile.fromconfig.commonimportConfigurationBuilderfromconfig.jsonimportJSONFilefromconfig.envimportEnvVarsbuilder=ConfigurationBuilder(JSONFile("settings.json"),EnvVars(prefix="APP_"))config=builder.build()For example, if the JSON file contains the following contents:{"logging":{"level":"INFO"},"example":"Hello World","foo":"foo"}And the environment has a variable namedAPP_foo=AAA:>>>config<Configuration{'logging':'...','example':'...','foo':'...'}>>>>config.foo'AAA'>>>config.logging.level'INFO'YAML file and environment variablesIn this example, configuration will include anything inside a filesettings.yamland environment variables. Settings are applied in order, so
environment variables with matching name override values from theyamlfile
(using theyamlsource requires alsoPyYAMLpackage).fromconfig.commonimportConfigurationBuilderfromconfig.envimportEnvVarsfromconfig.yamlimportYAMLFilebuilder=ConfigurationBuilder()builder.add_source(YAMLFile("settings.yaml"))builder.add_source(EnvVars())config=builder.build()YAML file, optional file by environmentIn this example, if an environment variable with nameAPP_ENVIRONMENTand
valuedevexists, and a configuration file with namesettings.dev.yamlis
present, it is read to override values configured insettings.yamlfile.importosfromconfig.commonimportConfigurationBuilderfromconfig.envimportEnvVarsfromconfig.yamlimportYAMLFileenvironment_name=os.environ["APP_ENVIRONMENT"]builder=ConfigurationBuilder(YAMLFile("settings.yaml"),YAMLFile(f"settings.{environment_name}.yaml",optional=True))config=builder.build()Filtering environment variables by prefixfromconfig.commonimportConfigurationBuilderfromconfig.envimportEnvVarsbuilder=ConfigurationBuilder()builder.add_source(EnvVars(prefix="APP_"))config=builder.build()INI filesINI files are parsed using the built-inconfigparsermodule, therefore
support[DEFAULT]section; all values are kept as strings.fromconfig.commonimportConfigurationBuilderfromconfig.iniimportINIFilebuilder=ConfigurationBuilder()builder.add_source(INIFile("settings.ini"))config=builder.build()Dictionariesfromconfig.commonimportConfigurationBuilderbuilder=ConfigurationBuilder()builder.add_map({"host":"localhost","port":8080})builder.add_map({"hello":"world","example":[{"id":1},{"id":2}]})config=builder.build()assertconfig.host=="localhost"assertconfig.port==8080assertconfig.hello=="world"assertconfig.example[0].id==1assertconfig.example[1].id==2Keys and valuesfromconfig.commonimportConfigurationBuilderbuilder=ConfigurationBuilder()builder.add_map({"host":"localhost","port":8080})builder.add_value("port",44555)config=builder.build()assertconfig.host=="localhost"assertconfig.port==44555User secretsThe library provides a strategy to handle secrets during local development,
storing them into the user folder.The following example shows how secrets can be configured for a project:configsettingsinit
configsettingsset"Foo""Some secret value"Secrets are organized by project, and the project information is obtained frompyproject.tomlfiles (from theproject.nameproperty). Ifpyproject.tomlfile does not exist, one is generated automatically with a random name.Then, from a Python app, it's possible to load the secrets from the user folder:fromconfig.commonimportConfigurationBuilderfromconfig.jsonimportJSONFilefromconfig.secretsimportUserSecretsbuilder=ConfigurationBuilder(JSONFile("settings.json"),UserSecrets())config=builder.build()print(config)# config contains both values from `settings.json`, and secrets read from the user# folderSecrets are optional and should be used only for local development, they are
stored in unencrypted form in the user's folder.Production apps should use dedicated services to handle secrets, likeAzure Key Vault,AWS Secrets Manager, or similar services.
For Azure Key Vault, an implementation is provided inessentials-configuration-keyvault.Handling user settingsUser settings (stored in the user's folder) can be handled using the providedconfigCLI.These settings can be useful to store secrets and other values during local development, or in general when working with desktop applications.Overriding nested valuesIt is possible to override nested values by environment variables or
dictionary keys using the following notation for sub properties:keys separated by colon ":", such asa:d:ekeys separated by "__", such asa__d__efromconfig.commonimportConfigurationBuilder,MapSourcebuilder=ConfigurationBuilder(MapSource({"a":{"b":1,"c":2,"d":{"e":3,"f":4,},}}))config=builder.build()assertconfig.a.b==1assertconfig.a.d.e==3assertconfig.a.d.f==4builder.add_value("a:d:e",5)config=builder.build()assertconfig.a.d.e==5assertconfig.a.d.f==4Overriding nested values using env variablesimportosfromconfig.commonimportConfigurationBuilder,MapSourcefromconfig.envimportEnvVarsbuilder=ConfigurationBuilder(MapSource({"a":{"b":1,"c":2,"d":{"e":3,"f":4,},}}))config=builder.build()assertconfig.a.b==1assertconfig.a.d.e==3assertconfig.a.d.f==4# NB: if an env variable such as:# a:d:e=5# or...# a__d__e=5## is defined, it overrides the value  from the dictionaryos.environ["a__d__e"]="5"builder.sources.append(EnvVars())config=builder.build()assertconfig.a.d.e=="5"Overriding values in list items using env variablesimportosfromconfig.commonimportConfigurationBuilder,MapSourcefromconfig.envimportEnvVarsbuilder=ConfigurationBuilder(MapSource({"b2c":[{"tenant":"1"},{"tenant":"2"},{"tenant":"3"},]}),EnvVars(),)os.environ["b2c__0__tenant"]="5"config=builder.build()assertconfig.b2c[0].tenant=="5"assertconfig.b2c[1].tenant=="2"assertconfig.b2c[2].tenant=="3"Typed configTo bind configuration sections with types checking, for example to usepydanticto
validate application settings, use theconfig.bindmethod like in
the following example:# example-01.yamlfoo:value:"foo"x:100# examplefrompydanticimportBaseModelfromconfig.commonimportConfigurationBuilderfromconfig.yamlimportYAMLFileclassFooSettings(BaseModel):value:strx:intbuilder=ConfigurationBuilder(YAMLFile("example-01.yaml"))config=builder.build()# the bind method accepts a variable number of fragments to# obtain the configuration section that should be used to instantiate the given typefoo_settings=config.bind(FooSettings,"foo")assertisinstance(foo_settings,FooSettings)assertfoo_settings.value=="foo"assertfoo_settings.x==100Goal and non-goalsThe goal of this package is to provide a way to handle configuration roots,
fetching and composing settings from different sources, usually happening
once at application's start.The library implements only a synchronous API and fetching of application
settings atomically (it doesn't support generators), like application settings
fetched from INI, JSON, or YAML files that are read once in memory entirely.
An asynchronous API is currently out of the scope of this library, since its
primary use case is to fetch configuration values once at application's start.