Package: confect
IntroductionWhy you need a configuration library?For storing secretsYou have a project that needs to access database or other services with password or some secret keys.
Storing secrets and passwords in your code is not smart.
You need a configuration file and a library for loading and using it in the runtime.For different runtime environmentsFor example, database IP addresses and passwords in development environment normally differs from production environment.
You need multiple configuration files for storing those information for different environment, and load one of them in the run time.For better parameter managementYou’re running some experiments, e.g. working on Machine Learning projects.
There’re a bunch of parameters needs to be changed in the run time.
And you want to manage them in a smarter and more elegant way.How confect differs from others?Python configuration filesThis makes it possible tohave complex type objects as configuration values, like Decimal, timedelta
or any class instancedynamically handle complicated logic, you can use conditional statements
likeifin it.read other TOML/YMAL/JSON/ini files or even environment variables in the
configuration file.Loads configuration file through module importingConfect loads configuration file through a given file path, or through module importing.
It’s easy to control the source of configuration file throughPYTHONPATH.Loads configuration file multiple timesSometimes we need multiple configuration files — one for project,
one for team and one for personal use.
And we want that the personal configuration file has the highest priority.
If there’s a configuration setting existing in that file, it would override values
from other files.Loads configuration properties from environment variableThis feature is convenient if you want to change a single or some properties values,
and don’t want to modify the configuration file.Attachs command line options to some click_ commandYou can change any configuration value through command line options, if your command is created byclick.Better maintainabilityConfect forces users to define configuration properties and set a default value before using them.
And theconfobject is immutable for reducing the possibility of making errors.Installconfectis a Python package hosted on PyPI and works only with Python 3.6 up.Just like other Python packages, install it bypipinto avirtualenv, or usepoetryto manage project dependencies and virtualenv.$pipinstallconfectBasicsConf ObjectCallingconf = confect.Conf()creates a new configuration manager object.Supposeprojxis your top-level package name. Put the following lines intoprojx/core.pyorprojx/confspec.pyimportconfectconf=confect.Conf()It is possible to create multipleConfobjects, but normally it’s not what
you want. In most cases, initialize and manage only oneConfobject in your
application, then import and use it anywhere.Configuration Properties DeclarationConfiguration properties should be declared before using it.This feature
makes your code more readable and maintainable.Two ways to declare properties.context manager:withconf.declare_group(group_name)asgroup_name:group_name.prop1='default value'group_name.prop2=42function callconf.declare_group(group_name,prop1='default value',prop2=42)Group names and property names should be valid Python variable names, which
consist of letters (A-Z, a-z), digits (0-9), and the underscore character (_).
Normally, the group name is your class name, module name or subpackage name.Default Value and ParserDefault values of all properties should be defined along with the declaration.
Useconfect.prop(default, desc=None, prop_type=None)to specify details other than the
default value.descis for commentary and the help message in CLI option.
Argument ofprop_typeis an instance of confect.PropertyType which is
responsable for CLI argument and environment variable parsing.prop_typeof
popular Python types would be infered from default value automatically.Default values don’t have to be a workable value (e.g. fake secret keys or
passwords). The true workable value can be defined in the configuration file.
However, even if it’s not a workable value, the mock default values still make
the declaration and the code more readable and maintainable. For instance:withconf.declare_group('aws')asaws:aws.access_key_id='true-access-key'aws.secret_access_key='fake-key-plz-set-it-in-local_conf.py'Declaration Exampleimportconfectconf=confect.Conf()# declare properties with context managerwithconf.declare_group('api')asapi:# default value only. confect would infer property type automaticallyapi.cache_prefix='projx_cache'api.cache_expire=confect.prop(default=60*60*24,desc="expire time in seconds")# add description for CLI help message and commentaryapi.url_base_path=confect.prop(default='api/v2/',desc='URL base path of API')withconf.declare_group('db')asdb:db.host='127.0.0.1'db.db_name='projx'db.username='projx_admin'# if default value has to be None, it'd be better to assign property# type manually for parsingdb.password=confect.prop(default=None,prop_type=confect.prop_type.String(),desc='`None` for no password')db.port=confect.prop(default=None,prop_type=confect.prop_type.Integer(),desc='`None` for db engine default port')# declare properties with function callconf.declare_group('ctr_predict_model',model_pickle_s3folder='s3://some-bucket/path/to/folder',model_version=confect.prop(default='v3'))Declaration LocationProperty declarations can be put into the module where theconfobject is
located. Or, you can put them into those modules where you need these
configurations, likeprojx/db.pyorprojx/api.py. Just make sure your
application import all these modules eagerly, not lazily.Configuration AccessAfter the group and properties are declared, they are accessable through
theconfobject directly, likeconf.group_name.prop_name.projx/api.pyfromprojx.coreimportconf@routes(conf.api.url_base_path+'add')@redis_cache(key=conf.api.cache_prefix,expire=conf.api.cache_expire)defadd(a,b)returna+bprojx/db.pyfromprojx.coreimportconfengine=create_engine(f'mysql://{conf.db.username}:{conf.db.password}'f'@{conf.db.host}/{conf.db.db_name}')Access ErrorsMake sure that the configuration properties aredeclared before access. If not,
exceptions would be raised.>>>conf.unknown_group.unknown_propTraceback(mostrecentcalllast):...UnknownConfError:"Unknown configuration group 'unknown_group'">>>conf.api.unknown_propTraceback(mostrecentcalllast):...UnknownConfError:"Unknown 'unknown_prop' property in configuration group 'api'"Configuration properties and groups are immutable.They are meant to be
altered globally by loading configuration files, environment variables or CLI
argument.>>>conf.api.cache_expire=60*60*3Traceback(mostrecentcalllast):...confect.error.FrozenConfPropError:Configurationpropertiesarefrozen.Setting Configuration PropertiesConfiguration properties are immutable in the application runtime. This feature make surethe runtime environment is stable without unexpected behavior.The standard ways to change the configuration properties are:Load from Python fileconf.load_module(module_name)andconf.load_file(file_path). (CheckLoading Configuration File)Load from environment variableconf.load_envvar(prefix). (CheckLoading Environment Variables)Override by CLI optionsconf.click_options(click_command). (CheckCommand Line Options)Confect still provide a hacky way to change them in the runtime, but use them wisely.Alter configuration in the runtime(CheckRuntime Configuration Altering)Loading Configuration FileConfect loads Python configuration files. That makes your configuration file
programmable and unrestricted as we described in the sectionHow confect
differs from others?.Two ways to load configuration file.Through Python module importing:conf.load_module(module_name)Through Python file reading:conf.load_file(file_path)No matter the loading statement is located before or after properties
declaration, property values in configuration file always override default
values in the declarations. It’s possible to load configuration file multiple times,
the latter one would replace values from former loading.Be aware,you should access your configuration properties after load
configuration files.If not, you might get wrong/default value. Therefore, we
usually load configuration file right after the statement of creating theConfobject.importconfectconf=confect.Conf()# load configuration files through importingtry:conf.load_module('local_conf')exceptImportError:passSYSTEM_CONF_PATH=Path('path/to/system_conf.py')ifSYSTEM_CONF_PATH.exists():conf.load_file(SYSTEM_CONF_PATH)UsePYTHONPATHenvironment varibale to control the source of configuration file.$vilocal_conf.py$exportPYTHONPATH=.$pythonyour_application.pyWrite Configuration FileConfiguration files are written in Python, but they are isolated from your application. The configuration declaration(checkConfiguration Properties Declaration) use theconfobject directly, and declare properties with default value withconf.declar_group(...). While configuration files useconfect.cto override declared properties. Configuration files shouldn’t be import directly, they can only be loaded withconf.load_module(module_name))orconf.load_file(file_path).In configuration file, importconfect.cobject and set all properties on it
as ifcis the conf object. Here’s an example of configuration file.local_conf.pyfromconfectimportcimportosDEBUG=TrueifDEBUG:c.cache.expire=1c.cache.key=os.environ['CACHE_KEY']# loading some secret file and set configurationimportjsonwithopen('db_secret.json')asf:db_secret=json.load(f)c.db.username=db_secret['username']c.db.password=db_secret['password']It’s not necessary and is unusual to have all configuration properties be defined in the
configuration file.Put only those configuration properties that you want to override to the configuration file.You can set any property in any configuration group onto thecobject.
However,they are only accessable if you declared it in the source code withConf.declare_group(group_name). SeeConfiguration Properties Declarationfor details.Thecobject only exits when loading a python configuration file, it’s not
possible to import it in your source code.Advanced UsageLoading Environment Variables# overrides configuration with environment variables with the prefixprojxconf.load_envvars(‘projx’)Conf.load_envvars(prefix: str)automatically searches environment variables
in<prefix>__<group>__<prop>format. All of these three identifier are case
sensitive. If you have a configuration propertyconf.cache.expire_timeand
you callConf.load_envvars('projx'). It will set thatexpire_timeproperty to the parsed value fromprojx__cache__expire_timeenvironment
variable.>>> import os
>>> os.environ['projx__cache__expire'] = '3600'>>> conf = confect.Conf()
>>> conf.load_envvars('projx')  # doctest: +SKIPIfcache.expirehas been declared, then>>> conf.cache.expire
3600Command Line Optionsconf.click_optionsdecorator attachs all declared configuration to aclickcommand.projx/__main__.pyimportclickfromprojx.coreimportconf@click.command()@conf.click_optionsdefcli():click.echo(f'cache_expire:{conf.api.cache_expire}')if__name__=='__main__':cli()It automatically creates a comprehensive help message with all properties and default values.$python-mprojx.cli--helpUsage: cli.py [OPTIONS]

Options:
  --api-cache_expire INTEGER  [default: 86400]
  --api-cache_prefix TEXT     [default: projx_cache]
  --api-url_base_path TEXT    [default: api/v2/]
  --db-db_name TEXT           [default: proj_x]
  --db-username TEXT          [default: proj_x_admin]
  --db-password TEXT          [default: your_password]
  --db-host TEXT              [default: 127.0.0.1]
  --help                      Show this message and exit.The option do change the value of configuration property.$python-mprojx.clicache_expire: 86400$python-mprojx.cli--api-cache_expire33cache_expire: 33ParserConfect includes predefined parsers of these primitive types.str:sint:ast.literal_eval(s)float:ast.literal_eval(s)bytes:s.encode(encoding)datetime.datetime:dt.datetime.strptime(s, fmt)datetime.date:dt.datetime.strptime(s,fmt).date()tuple:json.loads(s)dict:json.loads(s)list:json.loads(s)Complex Configuration LoadingThe code in the sectionConf Objectis a simple example that loads only through module importing.
Here’s an much more complex example that demostrates how to dynamically select and load configurations.importsysimportconfectconf=confect.Conf()# load configuration fileiflen(sys.argv)==2:conf.load_file(sys.argv[1])else:try:conf.load_file('path/to/team_conf.py')FileNotFoundError:logger.warning('Unable to find team configuration file')try:conf.load_file('path/to/personal_conf.py')FileNotFoundError:logger.info('Unable to find personal configuration file')# load configuration file through importingtry:conf.load_module('projx_conf')exceptImportError:logger.warning('Unable to load find configuration module%r','proj_x_conf')# overrides configuration with environment variablesconf.load_envvars('projx')Runtime Configuration AlteringConf.mutate_locally()context manager creates an environment that makesConfobject temporarily mutable. All changes would be restored when it
leaves the block. It is usaful on writing test case or testing configuration
properties in Python REPL.>>> conf = Conf()
>>> conf.declare_group(  # declare group through keyword arguments
...      'dummy',
...      prop1=3,
...      prop2='some string')
...
>>> with conf.mutate_locally():
...      conf.dummy.prop1 = 5
...      print(conf.dummy.prop1)
5
...     call_some_function_use_this_property()
>>> print(conf.dummy.prop1)  # all configuration restored
3To-DosA public interface for exporting a conf group into a dictionaryA plugin forargparsethat adds command line options for altering configuration properties.Copy-on-write mechenism inconf.mutate_locally()for better performance and memory usage.API reference page