Package: typed-config
typed-configTyped, extensible, dependency free configuration reader for Python projects for multiple config sources and working well in IDEs for great autocomplete performance.pip install typed-configRequires python 3.6 or above. (Also, needstyping_extensionsfor Python <= 3.7, so not completely dependency free if using old Python).Basic usage# my_app/config.pyfromtypedconfigimportConfig,key,sectionfromtypedconfig.sourceimportEnvironmentConfigSource@section('database')classAppConfig(Config):host=key(cast=str)port=key(cast=int)timeout=key(cast=float)config=AppConfig()config.add_source(EnvironmentConfigSource())config.read()# my_app/main.pyfrommy_app.configimportconfigprint(config.host)In PyCharm, and hopefully other IDEs, it will recognise the datatypes of your configuration and allow you to autocomplete. No more remembering strings to get the right thing out!Upgrading from 0.x.xThere is one breaking change when moving from0.x.xto1.x.x. Thekeyfunction now expects all arguments to be keyword arguments. So simply replace any calls like so:key('section','key',True,str,'default')# 0.x.xkey(section_name='section',key_name='key',required=True,cast=str,default='default')# 1.x.xThe reason for this change is to tie down the return type ofkeyproperly. Previously, whenrequired=Falseor whencast=Nonethe return type would not include the possibility ofNoneorstring. The type checker should now be able to infer the return type based on the values ofrequired,castanddefault.How it worksConfiguration is always supplied in a two level structure, so your source configuration can have multiple sections, and each section contains multiple key/value configuration pairs. For example:[database]host=127.0.0.1port=2000[algorithm]max_value=10min_value=20You then create your configuration hierarchy in code (this can be flat or many levels deep) and supply the matching between strings in your config sources and properties of your configuration classes.You provide one or moreConfigSources, from which the config for your application can be read. For example, you might supply anEnvironmentConfigSource,  and twoIniFileConfigSources. This would make your application first look for a configuration value in environment variables, if not found there it would then look at the first INI file (perhaps a user-specific file), before falling back to the second INI file (perhaps a default configuration shared between all users). If a parameter is still not found and is a required parameter, an error would be thrown.There is emphasis on type information being available for everything so that an IDE will autocomplete when trying to use your config across your application.Multiple data sourcesfromtypedconfigimportConfig,key,section,group_keyfromtypedconfig.sourceimportEnvironmentConfigSource,IniFileConfigSource@section('database')classDatabaseConfig(Config):host=key(cast=str)port=key(cast=int)username=key(cast=str)password=key(cast=str)config=DatabaseConfig()config.add_source(EnvironmentConfigSource(prefix="EXAMPLE"))config.add_source(IniFileConfigSource("config.cfg"))# OR provide sources directly to the constructorconfig=DatabaseConfig(sources=[EnvironmentConfigSource(prefix="EXAMPLE"),IniFileConfigSource("config.cfg")])Since you don't want to hard code your secret credentials, you might supply them through the environment.
So for the above configuration, the environment might look like this:exportEXAMPLE_DATABASE_USERNAME=my_usernameexportEXAMPLE_DATABASE_PASSWORD=my_very_secret_passwordexportEXAMPLE_DATABASE_PORT=2001Those values which couldn't be found in the environment would then be read from the INI file, which might look like this:[database]HOST=db1.mydomain.comPORT=2000Note after this,config.portwill be equal to2001as the value in the environment took priority over the value in the INI file.CachingWhen config values are first used, they are read. This is lazy evaluation by default so that not everything is read if not necessary.After first use, they are cached in memory so that there should be no further I/O if the config value is used again.For fail fast behaviour, and also to stop unexpected latency when a config value is read partway through your application (e.g. your config could be coming across a network), the option is available to read all config values at the start. Just callconfig.read()This will throw an exception if any required config value cannot be found, and will also keep all read config values in memory for next time they are used. If you do not usereadyou will only get the exception when you first try to use the offending config key.Hierarchical configurationUsegroup_keyto represent a "sub-config" of a configuration. Set up "sub-configs" exactly as demonstrated above, and then create a parent config to compose them in one place.fromtypedconfigimportConfig,key,section,group_keyfromtypedconfig.sourceimportEnvironmentConfigSource,IniFileConfigSource@section('database')classDatabaseConfig(Config):host=key(cast=str)port=key(cast=int)@section('algorithm')classAlgorithmConfig(Config):max_value=key(cast=float)min_value=key(cast=float)classParentConfig(Config):database=group_key(DatabaseConfig)algorithm=group_key(AlgorithmConfig)description=key(cast=str,section_name="general")config=ParentConfig()config.add_source(EnvironmentConfigSource(prefix="EXAMPLE"))config.add_source(IniFileConfigSource("config.cfg"))config.read()The first time theconfig.databaseorconfig.algorithmis accessed (which in the case above is whenread()is called), then an instance will be instantiated. Notice that it is the class definition, not an instance of the class, which is passed to thegroup_keyfunction.Custom section/key names, optional parameters, default valuesLet's take a look at this:fromtypedconfigimportConfig,key,section@section('database')classAppConfig(Config):host1=key()host2=key(section_name='database',key_name='HOST2',required=True,cast=str,default=None)Bothhost1andhost2are legitimate configuration key definitions.section_name- this name of the section in the configuration source from which this parameter should be read. This can be provided on a key-by-key basis, but if it is left out then the section name supplied by the@sectiondecorator is used. If all keys supply asection_name, the class decorator is not needed. If bothsection_nameand a decorator are provided, thesection_nameargument takes priority.key_name- the name of this key in the configuration source from which this parameter is read. If not supplied, some magic uses the object property name as the key name.required- default True. If False, and the configuration value can't be found, no error will be thrown and the default value will be used, if provided. If a default not provided,Nonewill be used.cast- probably the most important option for typing.If you want autocomplete typing support you must specify this. It's just a function which takes a string as an input and returns a parsed value. See the casting section for more. If not supplied, the value remains as a string.default- only applicable ifrequiredis false. Whenrequiredis false this value is used if a value cannot be found.TypesfromtypedconfigimportConfig,key,sectionfromtypingimportListdefsplit_str(s:str)->List[str]:return[x.strip()forxins.split(",")]@section('database')classAppConfig(Config):host=key()port=key(cast=int)users=key(cast=split_str)zero_based_index=key(cast=lambdax:int(x)-1)config=AppConfig(sources=[...])In this example we have three ways of casting:Not casting at all. This defaults to returning astr, but your IDE won't know that so if you want type hints usecast=strCasting to an built in type which can take a string input and parse it, for exampleintDefining a custom function. Your function should take one string input and return one output of any type. To get type hint, just make sure your function has type annotations.Using a lambda expression. The type inference may or may not work depending on your expression, so if it doesn't just write it as a function with type annotations.ValidationYou can validate what has been supplied by providing a customcastfunction to akey, which validates the configuration value in addition to parsing it.Extending configuration using shared ConfigProviderMultiple application modules may use different configuration schemes while sharing the same configuration source. Analogously, variousConfigclasses may provide different view of the same configuration data, sharing the sameConfigProvider.# app/config.pyfromtypedconfig.providerimportConfigProviderfromtypedconfig.sourceimportEnvironmentConfigSource,IniFileConfigSourceprovider=ConfigProvider()provider.add_source(EnvironmentConfigSource(prefix="EXAMPLE"))provider.add_source(IniFileConfigSource("config.cfg"))__all__=["provider"]# app/database/config.pyfromtypedconfigimportConfig,key,sectionfromapp.configimportprovider@section('database')classDatabaseConfig(Config):host=key(cast=str)port=key(cast=int)database_config=DatabaseConfig(provider=provider)# app/algorithm/config.pyfromtypedconfigimportConfig,key,sectionfromapp.configimportprovider@section('algorithm')classAlgorithmConfig(Config):max_value=key(cast=float)min_value=key(cast=float)algorithm_config=AlgorithmConfig(provider=provider)Shared configuration provider can be used by plugins, which may need to declare additional configuration sections within the same configuration files as the main application.  Let's assume we have[database]section used by main application and[app_extension]that provides 3rd party plugin configuration:[database]host=127.0.0.1port=2000[app_extension]api_key=secrete.g.app/config.pymay look like that:fromtypedconfigimportConfig,key,section,group_keyfromtypedconfig.sourceimportEnvironmentConfigSource,IniFileConfigSource@section('database')classDatabaseConfig(Config):"""Database configuration"""host=key(cast=str)port=key(cast=int)classApplicationConfig(Config):"""Main configuration object"""database=group_key(DatabaseConfig)app_config=ApplicationConfig(sources=[EnvironmentConfigSource(),IniFileConfigSource("config.cfg")])and plugin can read additional sections by using the same configuration provider as main application config.e.g.plugin/config.py:fromtypedconfigimportConfig,key,section,group_keyfromapp.configimportApplicationConfig,app_config@section('app_extension')classExtensionConfig(Config):"""Extension configuration"""api_key=key(cast=str)# ExtendedAppConfig extends ApplicationConfig# so original sections are also includedclassExtendedAppConfig(ApplicationConfig):"""Extended main configuration object"""app_extension=group_key(ExtensionConfig)# ExtendedAppConfig uses the same provider as the main app_configextended_config=ExtendedAppConfig(provider=app_config.provider)fromplugin.configimportextended_config# Plugin can access both main and extra sectionsprint(extended_config.app_extension.api_key)print(extended_config.database.host)Configuration variables which depend on other configuration variablesSometimes you may wish to set the value of some configuration variables based on others. You may also wish to validate some variables, for example allowed values may be different depending on the value of another config variable. For this you can add apost_read_hook.The default implementation ofpost_read_hookreturns an emptydict. You can override this by implementing your ownpost_read_hookmethod. It should receive onlyselfas an input, and return adict. Thisdictshould be a simple mapping from config keys to values. For hierarchical configurations, you can nest the dictionaries. If you provide apost_read_hookin both a parent and a child class which both make changes to the same keys (don't do this) then the values returned by the child method will overwrite those by the parent.This hook is called whenever you call thereadmethod. If you use lazy loading and skip calling thereadmethod, you cannot use this hook.# my_app/config.pyfromtypedconfigimportConfig,key,group_key,sectionfromtypedconfig.sourceimportEnvironmentConfigSource@section('child')classChildConfig(Config):http_port_plus_one=key(cast=int,required=False)@section('app')classAppConfig(Config):use_https=key(cast=bool)http_port=key(key_name='port',cast=int,required=False)child=group_key(ChildConfig)defpost_read_hook(self)->dict:config_updates=dict()# If the port has not been provided, set it based on the value of use_httpsifself.http_portisNone:config_updates.update(http_port=443ifself.use_httpselse80)else:# Modify child configconfig_updates.update(child=dict(http_port_plus_one=self.http_port+1))# Validate that the port number has a sensible value# It is recommended to do validation inside the cast method for individual keys, however for dependent keys it can be useful hereifself.http_portisnotNone:ifself.use_https:assertself.http_portin[443,444,445]else:assertself.http_portin[80,81,82]returnconfig_updatesconfig=AppConfig()config.add_source(EnvironmentConfigSource())config.read()Configuration SourcesConfiguration sources are how your mainConfigclass knows where to get its data from. These are totally extensible so that you can read in your configuration from wherever you like - from a database, from S3, anywhere that you can write code for.You supply your configuration source to your config after you've instantiated it, butbeforeyou try to read any data from it:config=AppConfig()config.add_source(my_first_source)config.add_source(my_second_source)config.read()Or you can supply the sources directly in the constructor like this:config=AppConfig(sources=[my_first_source,my_second_source])config.read()Modifying or refreshing configuration after it has been loadedIn general it is bad practice to modify configuration at runtime because the configuration for your program should be fixed for the duration of it.  However, there are cases where it may be necessary.To completely replace the set of config sources, you can useconfig=AppConfig(sources=[my_first_source,my_second_source])config.set_sources([my_first_new_source,my_second_new_source])To replace a specific config source, for example because a config file has changed and you need to re-read it from disk, you can usereplace_source:fromtypedconfig.sourceimportIniFileConfigSourceoriginal_source=IniFileConfigSource("config.cfg")config=AppConfig(sources=[source])# Now say you change the contents to config.cfg and need to read it againnew_source=IniFileConfigSource("config.cfg")# re-reads file during constructionconfig.replace_source(original_source,new_source)Important: if you add or modify the config sources the config has been read, or need to refresh the config for some reason, you'll need to clear any cached values in order to force the config to be fetched from theConfigSources again. You can do this byconfig.clear_cache()config.read()# Read all configuration values againSupplied Config SourcesEnvironmentConfigSourceThis just reads configuration from environment variables.fromtypedconfig.sourceimportEnvironmentConfigSourcesource=EnvironmentConfigSource(prefix="XYZ")# OR justsource=EnvironmentConfigSource()It just takes one optional input argument, a prefix. This can be useful to avoid name clashes in environment variables.If prefix is provided, environment variables are expected to look like{PREFIX}_{SECTION}_{KEY}, for exampleexport XYZ_DATABASE_PORT=2000.If no prefix is provided, environment variables should look like{SECTION}_{KEY}, for exampleexport DATABASE_PORT=2000.IniFileConfigSourceThis reads from an INI file using Python's built inconfigparser. Read the docs forconfigparserfor more about the structure of the file.fromtypedconfig.sourceimportIniFileConfigSourcesource=IniFileConfigSource("config.cfg",encoding='utf-8',must_exist=True)The first argument is the filename (absolute or relative to the current working directory).encodingis the text encoding of the file.configparser's default is used if not supplied.must_exist- defaultTrue. If the file can't be found, an error will be thrown by default. Settingmust_existto beFalseallows the file not to be present, in which case this source will just report that it can't find any configuration values and yourConfigclass will move onto looking in the nextConfigSource.IniStringConfigSourceThis reads from a string instead of a filefromtypedconfig.sourceimportIniStringConfigSourcesource=IniStringConfigSource("""[section_name]key_name=key_value""")DictConfigSourceThe most basic source, entirely in memory, and also useful when writing tests. It is case insensitive.fromtypedconfig.sourceimportDictConfigSourcesource=DictConfigSource({'database':dict(HOST='db1',PORT='2000'),'algorithm':dict(MAX_VALUE='20',MIN_VALUE='10')})It expects data typeDict[str, Dict[str, str]], i.e. such thatstring_value = d['section_name']['key_name']. Everything should be provided as string data so that it can be parsed in the same way as if data was coming from a file or elsewhere.This is an alternative way of supplying default values instead of using thedefaultoption when defining yourkeys. Just provide aDictConfigSourceas the lowest priority source, containing your defaults.Writing your ownConfigSourcesAn abstract base classConfigSourceis supplied. You should extend it and implement the methodget_config_valueas demonstrated below, which takes a section name and key name, and returns either astrconfig value, orNoneif the value could not be found. It should not error if the value cannot be found,Configwill throw an error later if it still can't find the value in any of its other available sources. To make it easier for the user try to make your source case insensitive.Here's an outline of how you might implement a source to read your config from a JSON file, for example. Use the__init__method to provide any information your source needs to fetch the data, such as filename, api details, etc. You can do sanity checks in the__init__method and throw an error if something is wrong.importjsonfromtypingimportOptionalfromtypedconfig.sourceimportConfigSourceclassJsonConfigSource(ConfigSource):def__init__(self,filename:str):# Read data - will raise an exception if problem with filewithopen(filename,'r')asf:self.data=json.load(f)# Quick checks on data formatasserttype(self.data)isdictfork,vinself.data.items():asserttype(k)isstrasserttype(v)isdictforv_k,v_vinv.items():asserttype(v_k)isstrasserttype(v_v)isstr# Convert all keys to lowercaseself.data={k.lower():{v_k.lower():v_vforv_k,v_vinv.items()}fork,vinself.data.items()}defget_config_value(self,section_name:str,key_name:str)->Optional[str]:# Extract info from data which we read in during __init__section=self.data.get(section_name.lower(),None)ifsectionisNone:returnNonereturnsection.get(key_name.lower(),None)Additional config sourcesIn order to keeptyped-configdependency free,ConfigSourcesrequiring additional dependencies are in separate packages, which also havetyped-configas a dependency.These are listed here:pip install nameimport nameDescriptiontyped-config-aws-sourcestypedconfig_awssourceConfig sources usingboto3to get config e.g. from S3 or DynamoDBCast function libraryThetypedconfig.castsmodule contains helper functions that implement common casting operations. These would generally be passed to thecastparameter of thekey()functionCasting to anEnumtype withenum_casttheenum_castfunction converts a string input from a source to an member of anEnumtype.For example:fromenumimportEnumfromtypedconfig.castsimportenum_castfromtypedconfigimportConfig,key...classColorEnum(Enum):RED=1GREEN=2BLUE=3...classMyConfig(Config):color=key(cast=enum_cast(ColorEnum))In this example, if theConfigSourcereads the string"RED", the value ofcolorwill be set toColorEnum.REDNote that theenum_castfunction is designed to read thenameof an enum member, not its value (1,2or3in the example above)Casting to atuplewithtuple_castIf the source contains a list of items,tuple_castwill parse them as a python tuple, optionally applying a
base_cast function to each element.
The default behavior is to call strip() on each element and ignore trailing delimiters.For example, given the inputnums=1, 2, 3, 4,Then you could usenums=key(cast=tuple_cast())to read the string input and cast it to("1", "2", "3", "4")key(cast=tuple_cast(base_cast=int))Would cast the input to(1, 2, 3, 4)key(cast=tuple_cast(ignore_trailing_delimiter=False))Would cast the input to("1", "2", "3", "4", "")key(cast=tuple_cast(strip=False))Would cast the input to("1", " 2", " 3", " 4")Finally, if a delimiter other that "," is used - say the input string is"1:2:3:4"- that can be handled likekey(cast=tuple_cast(delimiter=":"))ContributingIdeas for new features and pull requests are welcome. PRs must come with tests included. This was developed using Python 3.7 but Travis tests run with all versions 3.6-3.9 too.Development setupClone the git repositoryCreate a virtual environmentvirtualenv venvActivate the environmentvenv/scripts/activateInstall development dependenciespip install -r requirements.txtCode styleCode style isblackand this is checked by the CI. To autoformat your code asblackbefore committing, just runblack typedconfig testRunning testspytestTo run with coverage:pytest --covMaking a releaseBump version number intypedconfig/__version__.pyAdd changes toCHANGELOG.mdCommit your changes and tag withgit tag -a v0.1.0 -m "Summary of changes"Travis will deploy the release to PyPi for you.Staging releaseIf you want to check how a release will look on PyPi before tagging and making it live, you can do the following:pip install twineif you don't already have itBump version number intypedconfig/__version__.pyClear the dist directoryrm -r distpython setup.py sdist bdist_wheeltwine check dist/*Upload to the test PyPItwine upload --repository-url https://test.pypi.org/legacy/ dist/*Check all looks ok athttps://test.pypi.org/project/typed-configIf all looks good you can git tag and push for deploy to live PyPiHere isa good tutorialon publishing packages to PyPI.