Package: onacol
Onacol is a low-opinionated configuration management library with following
features:YAML (=structured and hierarchical) configuration file supportEnvironment variables support (explicit and implicit)CLI arguments supportConfiguration merging/overwriting/layeringParameter validation (viaCerberus)Configuration schema, documentation and default values are defined in
single YAML -> No code schema.Minimal dependenciesComparison with other Python configuration libraries/frameworksAs the library name suggests, author is painfully aware this is not a unique
solution to the problem of application configuration. However, in the plethora
of existing solutions, none was completely fulfilling the features/requirements
mentioned above. So, with great reluctance,I had to make my own.Following table lists known/popular configuration frameworks and their
features relative to Onacol, but not comparing other features that some of those
libraries have and Onacol doesn’t, so check them out - you may find it suits
your need better.Popular configuration framework comparisonFrameworkYAMLENV varsCLI argsMergingValidationNo code schemaHydra✔️✔️❓✔️✔️✖️Pydantic❓❓✔️✔️✔️✖️Dynaconf✔️❓✔️✔️✔️✖️python-dotenv✖️✔️✖️✖️✖️✖️Gin Config❓❓❓❓✔️✖️Python Decouple✖️✖️✔️✔️✖️✖️OmegaConf✔️✔️✔️✔️✔️✖️Confuse✔️✔️❓✔️✔️✖️Everett✔️✔️✔️❓✔️✖️parse_it✔️✔️✔️✔️❓✖️Grift✖️✖️✖️❓✔️✖️profig✖️✔️✖️❓✔️✖️tweak✔️✖️✖️✔️✖️✖️Bison✔️❓✔️✔️✔️✖️Config-Man✖️✔️✔️❓✔️✖️figga✔️✖️✔️❓✖️✖️Onacol✔️✔️✔️✔️✔️✔️InstallationAs usually with pip:$ pip install onacolUsageDefault configuration file & schemaThe whole point of this library is the definition of both default configuration
and configuration schema in one YAML file (i.e. single source of configuration
truth).Let’s start with a simpledefault_config.yamlfile that is part of an example
application’s package. This example file contains default values for the
configuration.general:# Logging level for this application.log_level:INFOui:# Address and port of the UI webserveraddr:0.0.0.0port:8888sensors:sensor_reset_interval:30.0# Sensor reset interval in secondsconnected_units:-id:0# Sensor ID <0, 16>name:"Basicsensor"min_trigger_limit:30# Minimal triggering limit [cm]max_trigger_limit:120# Maximal triggering limit [cm]-id:1name:"Additionalsensor"min_trigger_limit:40max_trigger_limit:100This file can be used as it is. However, we can add a schema definition to the
structure, that will allow parameter validation and automatic type conversion.This is done by adding metadata to the YAML structure. Following metadata are
recognized by Onacol:oc_schema:Cerberusvalidator/schema definitions.oc_default: Default value (if metadata are attached to the YAML element, it
can no longer bear the value directly).oc_schema_id: Definition of a schema reference (seeRepeating schema elements)Schema metadata are NOT MANDATORY. We can only provide them to parameters for
which we think validation (or type conversion) may be useful.general:# Logging level for this application.log_level:INFOui:# Address and port of the UI webserveraddr:oc_default:0.0.0.0oc_schema:type:stringregex:"^(?:[0-9]{1,3}\\.){3}[0-9]{1,3}$"port:oc_default:8888oc_schema:type:integersensors:sensor_reset_interval:# Sensor reset interval in secondsoc_default:30.0oc_schema:type:floatmin:0.0max:100.0connected_units:-id:# Sensor ID <0, 16>oc_default:0oc_schema:type:integermin:0max:16name:"Basicsensor"min_trigger_limit:# Minimal triggering limit [cm]oc_default:30oc_schema:type:integermin:0max:200max_trigger_limit:# Maximal triggering limit [cm]oc_default:120oc_schema:type:integermin:0max:200-id:1name:"Additionalsensor"min_trigger_limit:40max_trigger_limit:100Note that for list definitions, schema is added only to the first element of the
list. Other elements will be validated based on the first element’s schema.Loading and validating configuration in an applicationOnacol is used by the application via theConfigManagerinstance.ConfigManagercan load configurations from multiple sources (files,
command line optional arguments, environment variables), but does not do it
automatically - the sources and order is up to the app implementation.A complete minimalistic example of an application (usingClickas a CLI
framework):"""Console script for onacol_test."""importsysimportclickimportpkg_resourcesfromonacolimportConfigManager# Localizing the defaults/schema configuration YAML in the packageDEFAULT_CONFIG_FILE=pkg_resources.resource_filename("onacol_test","default_config.yaml")# This must be here in order to retrieve args and options# that are not Click related (see https://stackoverflow.com/a/32946412)@click.command(context_settings=dict(ignore_unknown_options=True,allow_extra_args=True))@click.pass_context# The rest is the usual Click stuff@click.option("--config",type=click.Path(exists=True),default=None,help="Path to the configuration file.")@click.option("--get-config-template",type=click.File("w"),default=None,help="Write default configuration template to the file.")defmain(ctx,config,get_config_template):# Wrap optional config file into a listuser_config_file=[config]ifconfigelse[]# Instantiate config_managerconfig_manager=ConfigManager(DEFAULT_CONFIG_FILE,env_var_prefix="OCTEST",optional_files=user_config_file)# Generate configuration for the --get-config-template option# Then finish the applicationifget_config_template:config_manager.generate_config_example(get_config_template)return# Load (implicit) environment variablesconfig_manager.config_from_env_vars()# Parse all extra command line optionsconfig_manager.config_from_cli_args(ctx.args)# Validate the configconfig_manager.validate()# Finally, let's review interesting bits of the configprint("---------<Application configuration>-------------")print(f"Log level:{config_manager.config['general']['log_level']}")print(f"UI:{config_manager.config['ui']['addr']}"f"(port:{config_manager.config['ui']['port']})")print(f"Sensor reset interval: "f"{config_manager.config['sensors']['sensor_reset_interval']}")print(f"Sensors:")forsensorinconfig_manager.config["sensors"]["connected_units"]:print(f"\t{sensor['name']}[{sensor['id']}]\tTrigger limits: "f"({sensor['min_trigger_limit']},{sensor['max_trigger_limit']})")if__name__=="__main__":sys.exit(main())# pragma: no coverIn this example, the application is bundling thedefault_config.yamlfrom
the examples above as the default configuration/schema file.
Then it accepts additional configuration file via command
option, and on the top it uses the environment variables and additional
configuration via command line options. Configuration from all sources
are layered/overwritten on the top of the current configuration.As you can see in the code, the sources of configuration as well as their
prioritization depend on the order of whichConfigManagermethods are
called, there is no default and even the validation must be called explicitly.Configuration using additional fileIn the example app, additional config file are loaded with the--configoptional command line argument, that is used in theConfigManager’soptional_filesinit option. There is also theConfigManager.config_from_filemethod to do this anytime after init.Let’s use the following config file (my_config.yaml):general:log_level:DEBUGui:port:127.0.0.1And load it with the app:$ python main.py --config my_config.yaml
---------<Application configuration>-------------
Log level: DEBUG
UI: 127.0.0.1 (port: 8888)
Sensor reset interval: 30.0
Sensors:
         Basic sensor [0]        Trigger limits: (30, 120)
         Additional sensor [1]   Trigger limits: (40, 100)As you can see, the relevant default config parameters have been overwritten,
the others stay default. This layering works over configuration dicts of
unlimited depth, but does not work with lists (by design).Configuration using environment variablesThere are two ways how to use environment variables with Onacol:Implicit way- Onacol detects environment variables with defined prefix
and use them to overwrite current configuration.Explicit way- environment variables are referenced in the configuration
files and Onacol resolves the references upon loading the file.Using environment variables implicitlyIn the example app source, we defined theenv_var_prefixwith valueOCTEST. Using theConfigManager.config_from_env_varsmethod  will then
make Onacol parse existing environment variables for names
starting with the chosen prefix, and then use the rest of the name as path for
the configuration structure (using uppercase and__as the level separator).Let’s continue with the previous example:$ export OCTEST_SENSORS__SENSOR_RESET_INTERVAL=20.1
$ python main.py --config my_config.yaml
Log level: DEBUG
UI: 127.0.0.1 (port: 8888)
Sensor reset interval: 20.1
Sensors:
         Basic sensor[0] Trigger limits: (30, 120)
         Additional sensor[1] Trigger limits: (40, 100)Again, environment variable overwrites the original value. Environment variable
values are always strings. However, as we defined schema and type for the
configuration parametersensor_reset_interval, it was automatically
converted to integer. Although schema is not mandatory, it’s always useful for
parameters that can be configured via environment variables.When schema is not defined, Onacol tries to apply JSON conversion rules to
the value of the environment variable. That helps in most cases, but can
cause problems if you pass value such as “1.2”. In that case, it will be
automatically converted to float. If you want to receive it as string, you
must define schema for that particular config.It is also possible to overwrite entire lists with environment variables.
To do that, use again JSON as format:$ export OCTEST_SENSORS__CONNECTED_UNITS='[{"id": 2, "name": "JSON sensor", "min_trigger_limit": 10, "max_trigger_limit": 90}]'
$ python main.py --config my_config.yaml
---------<Application configuration>-------------
Log level: DEBUG
UI: 127.0.0.1 (port: 8888)
Sensor reset interval: 30.0
Sensors:
         JSON sensor [2]         Trigger limits: (10, 90)As explained above, lists are always overwritten completely, no layering.
It is not possible to use JSON to overwrite dicts in the configuration
structure.Using environment variables explicitlyEnvironment variables can be also explicitly referred in the configuration YAML
file with syntax${oc_env:ENV_VAR}:general:log_level:DEBUGui:addr:${oc_env:MY_ADDR}This reference is being resolved before the YAML is parsed (it’s a primitive
regex substitution). Therefore the YAML type conversion is used for non-string
values. Explicit environment variable references can be only used in file-type
configuration sources. Example:$ export MY_ADDR=192.168.1.10
$ python main.py --config my_config.yaml
---------<Application configuration>-------------
Log level: DEBUG
UI: 192.168.1.10 (port: 8888)
Sensor reset interval: 30.0
Sensors:
         Basic sensor [0]        Trigger limits: (30, 120)
         Additional sensor [1]   Trigger limits: (40, 100)In explicitly used environment variables, where schema is not defined, then
of course YAML default conversion rules are used.Configuration using command-line optionsCommand-line optional arguments can be also parsed by Onacol to retrieve
configuration parameters. The logic is very similar to the implicit usage of
environment variables, but no prefix is used and the level separator is--:$ python main.py --config my_config.yaml --ui--port 8080  --sensors--sensor-reset-interval 15.8
---------<Application configuration>-------------
Log level: DEBUG
UI: 127.0.0.1 (port: 8080)
Sensor reset interval: 15.8
Sensors:
         Basic sensor [0]        Trigger limits: (30, 120)
         Additional sensor [1]   Trigger limits: (40, 100)As with implicit environment variable, config parameters with defined schema get
automatically converted to their types. It’s also allowed to use JSON lists.Generation of an example/template config fileDefault configuration/schema can be used to generate an example (template)
config file withConfigManager.generate_config_examplemethod. This file
has the schema information stripped, but retains the comments  used in the
defaults YAML file.The example app has the–get-config-templateoption to demonstrate it:$ python main.py --get-config-template config_template.yamlwill generate followingconfig_template.yamlfile:general:# Logging level for this application.log_level:INFOui:# Address and port of the UI webserveraddr:0.0.0.0port:8888sensors:sensor_reset_interval:30.0# Sensor reset interval in secondsconnected_units:-id:0# Sensor ID <0, 16>name:Basic sensormin_trigger_limit:30# Minimal triggering limit [cm]max_trigger_limit:120# Maximal triggering limit [cm]-id:1name:Additional sensormin_trigger_limit:40max_trigger_limit:100The comments are retained by the magic ofRuamel YAML, and there are some
limits. For proper retaining of comments, try to put the comments at the end
of line and avoid above-line comments where the preceding element is a schema
element.Exporting current configuration to a config fileThe current state of the configuration can be dumped to a file using
theConfigManager.export_current_configmethod.Repeating schema elementsIn case the configuration schema has repeating elements, it’s possible to define
schema for just one element, declare a reference for it withoc_schema_idand then refer other elements to that schema definition directly withoc_schema:network_interfaces:ethernet_interface:name:# Element nameoc_default:"eth0"oc_schema:type:stringid:oc_default:0oc_schema:type:integerip_addr:oc_default:192.168.1.2oc_schema:type:stringregex:"^(?:[0-9]{1,3}\\.){3}[0-9]{1,3}$"# Here we declare re-usable schemaoc_schema_id:network_interface_itemwifi_interface:name:wifiid:1ip_addr:192.168.2.3oc_schema:network_interface_item# Here we reference the previously declared schema:Configuration layeringWhen default or current configuration gets overwritten with new config values,
the previous values are kept internally and can be accessed. This is done using
the cascading features ofCascaDict(the configuration structure is kept inConfigManager.configasCascaDictinstance).If you are not interested in this, just use it as if it was a regulardict.Other notesFor any sort of configuration with variable amount of elements, use lists,
not dicts. Onacol is written on assumption that the configuration tree
consists of more-or-less fixed dicts and variable length lists.To create a default config/schema that shall enforce the end user to overwrite
some parameters, usenullas the default value and use schema withnullable: false- seeCerberos docs.
Validation will then report error when this value is not overwritten.LicenseFree software: MIT licenseDocumentationFull docs athttps://onacol.readthedocs.io.