Package: tconf
Why yes, it’s turtles all the way down 🐢🐢🐢🐢…This is an attempt at a cascading configuration library that I’d like to use.
The basic idea is that you get a configuration object and read its properties
for the values of options you’ve potentially defined all over the place, haha.Behind the scenes TurtleConfig will look at command-line parameters,
the environment,
user configuration files,
host/site configuration,
and the app’s own defaults to find the best value.
Typically in that order,
although you’re welcome to modify that list of configuration sources to
whatever you see fit.Instead of the typical hodge-podge of custom code needed for that,
setting up “TC” looks a little something like this.
First, you’ll define a schema via a standard Python object.
Either imported from a module,
or locally via a class/object,
which is easier to view here:importosfromtconfimportTurtleConfigimportschemaasAppSchema# or…classAppSchema:an_option=True# simple optionsclassmain:# or perhaps a hierarchyjpeg_quality=95file_path='/foo'Schema objects support type annotations and validation as well,
but don’t need to be given most of the time.Next, create the TurtleConfig object:cfg=TurtleConfig('AppyMcApp',# The app's namesources=(# A sequence of option sourcesos.environ,'{user_config_dir}/options.ini',# appdirs'{site_config_dir}/options.ini',# appdirs'/path/to/options.xml',# or JSON, YAML, etc.AppSchema,# The schema/defaults object),ensure_paths=True,# creates folders/files when needed)Virtual PathsNote,
theuser_config_dir&site_config_dirvariables above are provided by
theappdirsmodule
so you don’t have to worry about cross platform locations—unless you want to.Tip:  You may want to start logging atDEBUGlevel before loading TC,
as it helps to visualize what is going on inside its shell.
Theoutmodule
is perhaps the easiest way to do that:importsys,outout.configure(level='debug'if'-d'insys.argvelse'info')Try It!First, install the small pure-python package, and off you go:⏵python3-mpipinstalltconfInterfacesOnce loaded,
there are two interfaces to get at options,
via attributes or dictionary-like access:# Attribute interface>>>cfg.main.jpeg_quality95# Dictionary interface>>>cfg['main.jpeg_quality']95Now, why would you use one form over the other?Well, the first (attribute interface) is easier to type, read,
and the design I originally wanted.
However, it has limitations in a number of circumstances that are damn near
impossible to overcome,
which you’ll read about below.So the second (dictionary form) is generally preferred unless the app has
simple needs,
such as a single-level configuration.
An editor “snippet” can mitigate the extra keystrokes.Value TypesSome configuration sources are limited (in a good way)† in that they return
option values only as strings.
For example, the environment, strict-yaml, and.inifiles have string-only
values.
However, our app is likely to need real types,
such as integers, booleans, or lists of strings, etc.
What to do?Under TC,
the types of option values are required to be the same type as their schema
defaults,
as defined in Python.
Remember, defaults are found from the last class/module/object passed as a
source,
theAppSchemaobject as seen in the example above.The default value types may be annotated,
otherwise the given default will be inspected for its type as a fallback.
This means you can skip having to write type annotations much of the time
as they are inferred.Points of interest:Turtle will attempt to convert or “coerce” string values gathered from a
config source into the expected (annotated or inferred) non-string type.Values are then type checked via thetypeguardmodule.Currently, simple and compound Python types are supported:str, int, float, boollist, tuple, dict, setList, Tuple, Sequence, Dict, Set(from le typing module)Union, Any, Optional, etc.(ditto)Annotations are required when complex, compound value types are needed,
as defined with the stdlibtypingmodule.Annotations may also support kwargs for an ArgumentParser, see below.† Conversion of types is better done in the application-layer than in the file
format to avoid unexpected edge-case bugs like“the Norway problem.”Configuration SourcesEach configuration source has an Adapter class to integrate the different
interfaces into one.
As mentioned,
when looking for options,
the sources are searched in order from
top to bottom,
front to back,
until a suitable value is found.
If an option is not found in any source,
anAttributeErrororKeyError(depending on interface)
is raised to ensure bugs are found early.Environment VariablesPerhaps you’d like to override options with environment variables.
This is what it looks like:>>>os.environ['PY_APPYMCAPP.MAIN.JPEG_QUALITY']='94'>>>cfg['main.jpeg_quality']94An environment variable matching one of our configuration values
is uppercase and prefixed withPY_and the application name.
Both parts of the prefix are able to be modified by modifying the app_name
and/or passing anenv_prefix='…'to theTurtleConfigconstructor.Limitations:Due to limits with how the environment adapter works,
it cannot provide hierarchical access to settings via the attribute interface.The reason is that the attributes are evaluated left to right.
At access time,
the object doesn’t yet have enough information to know if it should return the
final value or continue down the attribute chain.
It could decide on one or the other,
leading to a number of broken cases from either decision.
Bare attributesdowork with the environment when options are kept to a
single-level, however.
As mentioned previously,
dictionary-style access (shown above) works consistently.ConfigParser.inifiles have two levels by design and are great for config files.
Therefore they do work hierarchically by default and would typically require
exactly two levels.There is one exception for convenience, however.
If a single-level option is requested,
the section[main](configurable also) is tried as a fallback.
This is so one can use a single-level as well as a dual-level config with
ConfigParser,
simply by putting root options under[main]:⏵cattest.ini[main]jpeg_quality=96# snip>>>cfg.main.jpeg_quality96>>>cfg['main.jpeg_quality']96>>>cfg.jpeg_quality# looks in [main] also96JSONJSON is not a great format for humans to edit,
but still relatively common as configuration:⏵cattest.json{"an_option":true,"main":{"jpeg_quality":96,# ~snip~>>>cfg.main.jpeg_quality96>>>cfg['main.jpeg_quality']96>>>cfg.does_not_exist# …AttributeError:'does_not_exist'notfound.>>>cfg.an_optionTrueCompound data types are better encoded in the JSON itself
rather than trying to smash “PyON” into strings.XMLRequiresxmltodict:⏵pip3installtconf[xml]# or⏵pip3installxmltodict⏵cattest.xml<?xmlversion="1.0"encoding="UTF-8"?><root><an_option>true</an_option><a_null2/><main><jpeg_quality>96</jpeg_quality># ~snip~>>>cfg.main.jpeg_quality96>>>cfg['main.jpeg_quality']96>>>cfg.a_null2# implied None>>>cfg.an_optionTrueThis Adapter is kinda weak so far,
could use a rewrite.Limitations:Throws out the root element for parity with other source types.Finds only the first node (tag) at each level due to a dictionary-like
implementation.XML attributes are not currently reachable.  :-/Strict YAMLA much safer, simpler subset of YAML, which requires thestrictyamlmodule:⏵pip3installtconf[yaml]# or⏵pip3installstrictyaml⏵cattest.yamlan_option:truea_null:nullmain:jpeg_quality:96# snipSee JSON above for similar Python snippet.OthersIt’s trivial to add an adapter for other sources and file formats.
First subclassadapters._Adapterand add an instance to the sources list.
There is anfile_adapter_mapin the adapters module root to register file
extensions to avoid having to pass an instance every time, if desired.Tip: Additionally,
passing adapters into the source list manually can also be used to give an
Adapter different arguments than it would normally get.See the next section for an example,
and“use the source, Luke!”ArgumentParserYou may have been thinking, what about the command-line?
Good news,
there’s an ArgumentParser subclass available if you’d likealloptions
presented auto-magically.
Types and parameters are passed to ArgumentParser through annotations of theAppSchemaobject:# appy.pyfromtconfimportTurtleConfig,TurtleArgumentParser# 👀classAppSchema:# snip…classmain:# how to add a type via annotation,# simple types are already detected however:jpeg_quality:int=95# Also use annotations to pass a dictionary of# kwargs to ArgParser, w/o descriptive help text:jpeg_quality:dict(# 👀type=int,desc='The jpeg quality level',)=95tcfg=TurtleConfig('AppyMcApp',sources=(TurtleArgumentParser(AppSchema),# 👀# environment, config files, etc…AppSchema,),)Next, give it a try:⏵appy.py-husage:appy.py[-h][--a-simple-option][--main-jpeg-qualityI][--main-sync-dates-to-filesystem][--main-work-in-place][--rotate-resampleS][--sort-templateS][--specific-nameS]optionalarguments:-h,--helpshowthishelpmessageandexit--a-simple-option🐢(False,setsTrue)--main-jpeg-qualityI🐢Thejpegqualitylevel(int)--main-sync-dates-to…🐢(True,setsFalse)--main-work-in-place🐢(False,setsTrue)--rotate-resampleS🐢(str)--sort-templateS🐢(str)--specific-nameS🐢(str)Option help text can be set directly in the annotation withhelp='…'.
To use a template for help based on a given description and auto-detected type,
usedesc='…'instead.
The template is configurable as well via TurtleArgumentParser kwargs.Hiding Options:Options shown by an ArgumentParser can be hidden by passing thehelp=argparse.SUPPRESSvalue via the kwargs annotation to the option.Given enough options,
eventually the display of every possible option is too much,
and suppression gets tedious.
When something simpler to be presented to the end user is preferred,
this also works as you’d expect:fromargparseimportArgumentParserparser=ArgumentParser()parser.add_argument('--quality',default=cfg['main.jpeg_quality'],)Then, useargsinstead ofcfgafterward for the options that take
precedence.
Remember—dots in an options string are presented as underscores in the
ArgumentParser Namespace,
and dashes on the command-line:print('quality:',args.main_jpeg_quality)MiscPerformance“Why yes, it’s a racing Turtle.”TheTurtleConfigobject caches results so it doesn’t have to go crawling
through multiple files to find the value every time.
So don’t get fancy with changing the environment on the fly,
or editing config files unless you’ve cleared the cache with:cfg.clear_turtle_cache()ExceptionsThese are thrown when a error occurs.Access errors, say you’ve passed a bad name not found anywhere:AttributeError,attribute interfaceKeyError,dict interfaceOption value errors, when the value returned is bogus:ValueError,wrong value in this contextSyntaxError,string unable to be evaluatedTypeError,wrong type returnedOb-la-di ob-la-da life goes on bra…To Do:Candidates for implementation:TOML.envfilesWindows registryLicenseReleased under the LGPL, version 3+.