Package: yamlfig
Theyamlfigpackage provides developers with a framework for
defining rules that test and verify a config file's structure.  Those
rules are captured in a parser object which can be applied to
YAML-based config files to validate them.In particular, this module enables a developer to:define which fields are required, optional, or will be assigned
default values if omitted;declare types for those fields (e.g.,str,int,date,dict,list, etc.);run arbitrary functions to test the values in each field (e.g.,
regular expressions matches or file-path-existence checks).After a config file is parsed, validated, and accepted, the returned
object can be used to access the field values with some confidence
that they exist, are of the expected type, and have whatever other
properties the rules established.  If a config file is rejected, an
error explaining the violated rule is raised.This package was inspired by the similar capability thatargparsebrought to command-line argument parsing.ContentsGetting StartedInstallationWebsite and RepositoryExample UsageWalk-throughDetailsBasic UsageFields, Paths, and StructureHandling Parsed ObjectsOptional, Default, and No-Follow RulesPath Type CheckingRule Test FunctionsWarnings and CaveatsNext StepsFuture WorkSupport and CollaborationGetting StartedInstallationpip install yamlfigWebsite and RepositoryThe Python package is hosted on PyPI:https://pypi.com/project/yamlfigThe source code, documentation, and issue tracker is hosted on GitHub:https://github.com/ktkalaru/yamlfigExample UsageAs an example for when a developer might useyamlfig, consider
developing a server that binds to an address and port.  When any of a
list of authorized users connects, it displays the server's name and
the contents of a file.  The following YAML file could act as the
config file for such a server:$ cat > quickstart_server.yamlname:Simple Single-File Serverserver:port:81file_path:'quickstart_shared_file.txt'users:-alice-bob-carolThe following script uses theyamlfigpackage to construct a
parser for this example server.  It instantiates a parser object and
adds a set of rules that establish which fields and structures should
be in a server's config file.  Those rules establish what form those
fields must take and what to do if they are missing.  It then invokes
this parser on a config file passed as a command line argument.  Where
an actual server script would then use those values to spin up a
server, this script just demonstrates that the values can be accessed
from the parsed object by printing them:$ cat > quickstart_server.pyfrom__future__importprint_functionimportsysfromyamlfigimportYamlConfigParser,test# Construct a parser for the server config fileconfp=YamlConfigParser()confp.add_rule('name',path_type=str)confp.add_rule('description',optional=True)confp.add_rule('server.addr',path_type=str,default='127.0.0.1',test=test.is_ipv4_address)confp.add_rule('server.port',path_type=int,test=test.is_interval(1,65535,include_upper=True))confp.add_rule('file_path',path_type=str,test=test.is_file_path('exists','isfile'))confp.add_rule('users',path_type=list)confp.add_rule('users.*',test=test.is_regex('^[a-z][a-z0-9]*$'))# Parse the config fileconf=confp.parse_file(sys.argv[1])# Retrieve values from the conf objectprint('conf.name ={0}'.format(repr(conf.name)))print('conf.description ={0}'.format(repr(conf.description)))print('conf.server.addr ={0}'.format(repr(conf.server.addr)))print('conf.server.port ={0}'.format(repr(conf.server.port)))print('conf.file_path ={0}'.format(repr(conf.file_path)))foridxinconf.users:print('conf.users[{0}] ={1}'.format(idx,repr(conf.users[idx])))When we execute this server script on the above config file, it prints
the following values:$ python quickstart_server.py quickstart_server.yamlconf.name = 'Simple Single-File Server'
conf.description = None
conf.server.addr = '127.0.0.1'
conf.server.port = 81
conf.file_path = 'quickstart_shared_file.txt'
conf.users[0] = 'alice'
conf.users[1] = 'bob'
conf.users[2] = 'carol'Note how the fields and values printed came not only from the config
file but also from the rules.  Fields marked as optional or taking a
default are present even though they were not in the config file.
Also note how the values have the type and form required by the rules
(e.g., an integer within a given interval, a string in IPv4
dotted-quad notation, and a list of usernames that all match a regular
expression).Walk-throughIn this example, the config file specified anameto display, aportto listen on within aserverblock, afile_pathto a
file to share, and the list of accounts of authorizedusers.  The
rules added to theconfpparser ensure that those rules exist; they
also define rules for some paths not in the config file and configured
asoptionalor taking adefaultvalue:adescriptionfield, if provided, would be displayed to the user, but
it is optional;anaddrfield within theserverblock that, if provided, would
set the binding IP address, but that takes a default of127.0.0.1if omitted.Additionally, theconfpparser verifies that the values present in
the config file are suitable for our intended use.  Some of the values
are type-checked or otherwise validated:theaddrfield, if provided, will be tested to confirm that it
has the format of an IPv4 address (i.e., a string in dotted-quad
notation);theportfield will have its type checked to ensure it is anint, and its value will be tested to confirm that in the range
1 to 65535;thefile_pathwill be interpreted as a path to a filesystem
object, and that object will be tested to confirm it exists and
is a file (rather than a directory);all the account names in theuserslist will be tested against
a regular expression to confirm they match the format of accounts
on this particular system (i.e., they start with a lowercase
letters followed by zero or more digits or lowercase letters).Once theconfpparser is constructed and configured, itsparse_filemethod is called on the config-file path given on the
command line, and it returns a parsedconfobject.  To demonstrate
that theconfobject contains all the fields and values from the
config file merged with the optional fields and defaults from the
parser rules, it prints those fields and values.The values in theconfobject returned byparse_filehave also
been type-checked and tested.  Had the fields and values in the config
file not conformed to the rules of theconfpparser, aParseErrorexception would have been raised.  Some examples:If thenamefield were omitted:ParseError: quickstart_server.yaml: "name" is missingIf theserverblock contained a field calledthe_ip_addressthat did not match any of the parser's rules:ParseError: quickstart_server.yaml: "server.the_ip_address" unexpected by parserIf theportfield of theserverblock contained the string"eighty-one"rather than the integer81:ParseError: quickstart_server.yaml: "server.port" has type str not type intIf theaddrfield were present in the server block and had
the value452.34.256.193:ParseError: quickstart_server.yaml: "server.addr" failed test: 1st octet of "452.34.256.193" exceeds 255If thefile_pathfield had been the path to an existing
directory namedsome_directoryinstead of the path to an
existing file:ParseError: quickstart_server.yaml: "file_path" failed test: "some_directory" is not a fileIf the 3rd value of theuserslist been the display nameCarol C.instead of the usernamecarol(and noting zero-based
indexing):ParseError: quickstart_server.yaml: "users.2" failed test: "Carol C." does not match /^[a-z][a-z0-9]*$/The presence of such errors in the config file would have stopped
execution and provided a relatively informative explanation of which
rule failed and why.  Because none of these errors were raised, a
developer has some assurance that the structure and values in theconfobject meets their expectations.Just as important as what happened in this example above iswhat
didn't happen.  When theread_filefunction returned theconfobject, it didn't raise aParseErrorexception.  Since it executed
successfully, we know that all the parser assertions hold about which
fields must exist and what formats they take; the remaining code does
not need to perform such checks and error handling itself.DetailsWhatyamlfigprovides beyond a standard YAML parser is validation,
specifically verification that a config file conforms to the various
rules established for it.  In this section, we introduce and describe
these rules, and the various constraints that can be placed on
a config file's structure and values.Basic UsageThe typical steps when usingyamlfigare:instantiate aYamlConfigParserobject, which we usually callconfp,configure it by usingadd_ruleto add rules for each field we
intend to control through a config file,invokeparse_fileon a config file which either raises aParseErroror returns aYamlConfigobject, usually calledconf, anduse thatYamlConfigin subsequent code, confident that its
structure and values have already been validated.The following script illustrates this typical pattern by usingyamlfig.  For the sake of the example, let's say we need a config
file to drive how often a loop is run, which of two functions is
called by the loop, and what parameter is passed to that function:$ cat basic_usage.pyimportsysfromyamlfigimportYamlConfigParser# 1. Instantiate a YamlConfigParser object (confp)confp=YamlConfigParser()# 2. Configure the parser by adding rules for each fieldconfp.add_rule('loop_count')confp.add_rule('do_special_function')confp.add_rule('function_parameter')# 3. Invoke the parser on a config file (provided as an argument)conf=confp.parse_file(sys.argv[1])# 4. Use the YamlConfig object in subsequent codeforloop_indexinrange(conf.loop_count):ifconf.do_special_function:special_function(conf.function_parameter)else:regular_function(conf.function_parameter)One config file would cause the script to produce one behavior:$ cat basic_config_1.yamlloop_count:7do_special_function:yesfunction_parameter:"ameerkat"Running the script onbasic_config_1.yamlwould causespecial_functionto be invoked 7 times, each with the parameter"a meerkat".Another config file would cause the script to produce a different behavior:$ cat basic_config_2.yamlloop_count:3do_special_function:nofunction_parameter:a ponyRunning the script onbasic_config_2.yamlwould causeregular_functionto be invoked 3 times, each with the parameter"a pony".Just as important is understanding the behavior of the script on a bad
config file.  The following config file is missing one of the three
required fields:$ cat basic_config_bad.yamlloop_count:3function_parameter:'aunicorn'Running the script onbasic_config_bad.yamlexits unsuccessfully and
prints an exception:Traceback (most recent call last):
  File "basic_usage.py", line 22, in <module>
    conf = confp.parse_file(sys.argv[1])
  [...]
yamlfig.base.ParseError: basic_config_bad.yaml: "do_special_function" is missingThe error is raised within theconfp.parse_filefunction.  All
verification and validation occurs as part of that function called in
step 3 of the pattern, so if it return successfully, theYamlConfigobject conforms with the parser rules.Fields, Paths, and StructureThe fundamental thing thatyamlfigrules do is establish which
fields should be in a config file and which fields should not.Test that a field existsconfp.add_rule('dirname')The first argument toadd_ruleis therule_path.  Every rule added
to aconfpobject must have one, and—unless additional
modifiers make the field optional or take a default value—it is
an existence requirement for the field.  Given the rule above, any
config file must contain a line such as:dirname:/var/share/SomeApp/SharedDirA config file without adirnamefield would generate a parse error.Test that a path existsPart of YAML's descriptive power comes from its ability to encode
nested structures, like maps and lists, andyamlfigrules can
describe constraints on that structure:confp.add_rule('server.storage.dirname')Rules use the'.'character
to delimit fields within a nested structure.  The rule above expects
there to be aserverblock, within which is contained astorageblock, within which there is adirnamefield.  The following config
would satisfy such a rule:server:storage:dirname:/var/share/SomeApp/SharedDirSuch a rule path implicitly includes existence requirements forserverandserver.storage.  The existence of those paths would not
need to be explicitly required through separate rules, unless we
wanted to modify them (e.g., by making them optional or take defaults
as described in alater
section.Test that a block has a specific substructureIn this example, we need a config file to describe how a server's
local storage cache is configured (i.e., where it is on the
filesystem, how big it can grow, and what permissions the cache files
have).confp.add_rule('server.storage.dirname')confp.add_rule('server.storage.maxsize')confp.add_rule('server.storage.umask')In combination, these three rules describe the structure that theserver.storageblock must have (i.e., three fields with the namesdirname,maxsize, andumask).The following config file would be accepted by this parser:server:storage:dirname:/var/share/SomeApp/SharedDirmaxsize:10GBumask:0644Test that a field or pathdoes notexistAny field or path for which there is no matching rule will raise a
parse error.  In a sense, theyamlfigfield-existence validation
is deny-by-default.  We do not need to do anything specific to assert
that a field does not exist; just don't add an existence requirement.Test that a block contains fields without specifying which fields by using wildcardsA rule path can contain wildcards.  For the sake of this example, we
need a config file to specify upload paths for each of one or more
users.  Aserver.upload_pathsblock will map from username to their
corresponding upload directory, as in the following example:server:upload_paths:alice:/home/alice/uploadsbob:/home/bob/publicSince we do not want to hardcode the usernames in the parser, we can
use a wildcard rule to accept one or more fields within a block
without specifying the specific field names:confp.add_rule('server.upload_paths.*')Such a rule asserts that theserver.upload_pathsblock contains
non-empty substructure (i.e., it is a block), but not the specific
field names within the substructure.  In the above example config
file, the wildcard woudl match bothaliceandbob, even though
neither are explicitly listed field paths.A new user could be added with their own upload path, and the same
parser would accept the config file:server:upload_paths:alice:/home/alice/uploadsbob:/home/bob/publiccarol:/home/carol/tmpNote that a wildcard rule must matchone or morefields, not zero.
If theserver.upload_pathsblock were empty, the config file would
raise an error:"server.upload_paths" must contain at least one field.  Alater
segmentdescribes how theoptionalanddefaultflags can be used with
wildcards to implement a zero-or-more match.Also note thatpartialwildcard matches are not currently supported.
A path likeserver.upload_paths.user-*intending to accept fields
likeuser-aliceanduser-bobwould instead raise an error.  As
described in adifferent later
segment, one way to implement such a
check would be to write a function that tests every field within a
block against a regular expression, and then specify that function as
a test function for theserver.upload_pathsblock.Wildcard fields can have substructure and rules can enforce matching substructureFor the sake of this example, a server hosts one or more projects,
each of which has a directory of static web pages associated with it,
and a backend database.  Our configuration file maps from one or more
arbitrary project names (e.g.,ProjectXandmeerkat_works) to
blocks that contain precisely three datapoints (1) a path to a
directory of webpages, (2) a path or URL to a database, and (3) the
type of the database (e.g.,sqlite,mysql, ormongodb).The following config file gives an example of this structure:projects:ProjectX:webpath:/home/alice/projx/htmldbpath:/home/alice/projx/project.dbdbtype:sqlitemeerkat_works:webpath:/home/bob/public/meerkat/wwwdbpath:mongodb://192.168.1.200:27017dbtype:mongodbWithyamlfig, we can specify wildcards on paths while still
requiring that any fields matching the wildcard have a required,
fixed-field substructure.  The following parser will accept one or
more blocks, each corresponding to a project name, but every one must
have the three required fields:confp.add_rule('projects.*.webpath')confp.add_rule('projects.*.dbpath')confp.add_rule('projects.*.dbtype')These rules implicitly assert that theprojectsblock exists and
contains one or more sub-blocks, with no restriction on their field
names.  The rules explicitly assert that each one of those sub-blocks
must contain exactly three fields:webpath,dbpath, anddbtype.
This parser would accept the example config file above.If a project sub-block were missing one of the three required fields
or had an extra field, an error would have been raised.Wildcards are also useful for accepting lists of valuesYAML's nested structure supports not only the mappings described above
but also lists.  Technically, YAML offers a whole lot of different
nesting types (e.g.,omap,pairs, etc.) but our python parser
represents them all as eitherdictorlistobjects, with mappings
represented asdictobjects and lists aslists.  To most easily
and succinctly accommodate both mappings and lists,yamlfigeffectively treats lists as a very specific kind of mapping, where
each field is a list index and each value is the item in the list.Consider a config file where ausersblock contains a list of one or
more authorized users:users:-alice-bob-carolThe following rule would accept such ausersblock:confp.add_rule('users.*')Note that the wildcard rule by itself just ensure that there are
subfields, not that they take the form of a list as opposed to a
mapping.  That same parser would accept a config file with a mapping:users:alice:Alice A.bob:Bob B.carol:Carol C.These two config files—the list version and the mapping
version—have very different structures, and a program would
likely be expecting one and not the other.  To ensure that a rule with
a wildcard matches only a list (and not a map) or only a map (and not
a list), we would need to use type checking, as described in alater
segment, to assert that
the type of the block is eitherlistordictrespectively.Also note that when accessing list values parsed into aconfobject,
we need to be aware of some difference in their behavior from that of
a standard python list, as described in theHandling Parsed
Objectssection.Test that a list has exactlynelementsWhile not a typical occurrence, parser rules can be configured to
ensure that a list has a specific number of elements.  The following
rules would accept a list of length 2 by explicitly requiring fields
named0and1:confp.add_rule('network.route.0')confp.add_rule('network.route.1')For the sake of the example, perhaps the application must have two
network routes, a primary and a secondary.As noted above,yamlfigtreats lists as mappings from numeric
fields to values, so the parser would accept the following config:network:route:-192.0.2.1-198.51.100.1As a side note, the same two rules would accept a config in which therouteblock contained a mapping from numeric string fields (i.e.,"0"and"1") to the two IPv4 addresses.  As we keep stressing, the
rules simply treat lists as mappings from numeric fields to the list
elements.  To differentiate a list from a mapping, we would need to
use type checking, as described in alater
segmentHandling Parsed ObjectsAs described in theBasic Usagesection, to parse a
config file, a parser'sparse_filemethod would be called with the
name of the file:conf=confp.parse_file(conffile)Assuming the parsing and validation succeeds, theconfobject would
have typeYamlConfigorYamlConfigList, depending on whether the
root-level YAML object in the config file is a mapping or a list.
Typically, a YAML-formatted config file will have a mapping as its
root-level structure, and so we will consider that common case first.Throughout this section, assume that we have successfully parsed the
following config file into aYamlConfigobject namedconf:dirname:/var/share/SomeApp/SharedDirserver:projects:ProjectX:webpath:/home/alice/projx/htmldbpath:/home/alice/projx/project.dbdbtype:sqlitemeerkat_works:webpath:/home/bob/public/meerkat/wwwdbpath:mongodb://192.168.1.200:27017dbtype:mongodbusers:-alice-bob-carol-daveWhile the focus of this section is on accessing theconfobject
afterconfpsuccessfully parses and validates the config file, for
the sake of completeness, the following rules would configure a
parser that accepts this file:confp=YamlConfigParser()confp.add_rule('dirname')confp.add_rule('server.projects.*.webpath')confp.add_rule('server.projects.*.dbpath')confp.add_rule('server.projects.*.dbtype')confp.add_rule('users.*')Fields and paths can be accessed as attributesFields in a config file can be accessed as attributes of theYamlConfigobject.conf.dirname# '/var/share/SomeApp/SharedDir'If an attribute corresponds to a block in a config file, it will
return that block as aYamlConfigorYamlConfigListobject.conf.server# <YamlConfig object at 0x[...]>conf.users# <YamlConfigList object at 0x[...]>As such, attributes can be strung together in a sequence:conf.server.projects.ProjectX.webpath# '/home/alice/projx/html'conf.server.projects.ProjectX.dbpath# '/home/alice/projx/project.db'conf.server.projects.ProjectX.dbtype# 'sqlite'conf.server.projects.meerkat_works.dbtype# 'mongodb'Note that to be accessed as an attribute, a field must be a valid
Python attribute (e.g., must be a string, cannot start with a number,
etc.).Fields and paths can be accessed via index lookupsA values stored in aYamlConfigobject can also be accessed via
index lookup.conf.server.projects['ProjectX'].dbtype# 'sqlite'proj='ProjectX'conf.server.projects[proj].dbtype# 'sqlite'conf['server']['projects']['ProjectX']['dbtype']# 'sqlite'path=['server','projects','ProjectX','dbtype']functools.reduce(lambdad,idx:d[idx],path,conf)# 'sqlite'List values can be accessed via index lookupsIndex lookups must be used to access the elements of aYamlConfigListsince attributes cannot be numbers.conf.users[0]# 'alice'conf.users[1]# 'bob'conf.users[2]# 'carol'conf.users[3]# 'carol'conf.users[-1]# 'dave'conf.users[-2]# 'carol'In a departure from standard python lists, aYamlConfigListobject
will translate to or from a string representation of an index as
needed.conf.users["1"]# 'bob'conf.users['-2']# 'carol'Once again, this is to allow—as much as possible—lists to
be treated like mappings from the list indexes to the list elements.Length checks can be used to determine the number of fieldsAs withdictandlistobjects, we can see how many elements are
in aYamlConfigandYamlConfigListobject by querying their
length.len(conf)# 3len(conf.server)# 1len(conf.server.projects)# 2len(conf.server.projects.ProjectX)# 3len(conf.server.projects.meerkat_works)# 3len(conf.users)# 4Iterators return field names forYamlConfigobjectsIterating on aYamlConfigobject will return the field names
contained within the block, like what we would get from iterating on
adictobject:list(conf)# ['dirname', 'server', 'users']list(conf.server)# ['projects']list(conf.server.projects)# ['ProjectX', 'meerkat_works']list(conf.server.projects.ProjectX)# ['webpath', 'dbpath', 'dbtype']list(conf.server.projects.meerkat_works)# ['webpath', 'dbpath', 'dbtype']Note that the order in whichYamlConfigfields are returned isthe
order the rules were added to the parser, not the order in which the
rules appear in the config file.  When a single parser rule matches
multiple fields (i.e., a wildcard rule), the fields are returned in
arbitrary order.  Note however, that around Python 3.6 and Python 3.7,
they have started being returned in the order they appear in the
config file, likely due todictobjects beginning to return keys in
the order they were inserted.Iterators return list indexesnot valuesforYamlConfigListobjectsIterating on aYamlConfigListobject issignificantly different
from iterating on a python list.  In particular, it will return the
list of index values as strings,not the actual list values:list(conf.users)# ['0','1','2','3']As noted previously, aYamlConfigListtreats lists less like lists
per-se and more like mappings from zero-based, sequential, numeric
indexes to values.  As such, its iterator returns field names that can
be used as indexes to look up values, not the values themselves.This behavior is likely unexpected at first and arguably
controversial, but was chosen for greater overall simplicity.  A lot
of code can iterate over fields, descend into blocks, and so on much
more simply, when it does not need to treatYamlConfigListobjects
as a special case, separate fromYamlConfigobjects.To get the values rather than the indexes, we recommend list
comprehension:[conf.users[idx]foridxinconf.users]# ['alice', 'bob', 'carol', 'dave']Unlike the fields of aYamlConfig, indexes of aYamlConfigListwill be returned in a specific order: sequential and increasing from
a base of zero.Optional, Default, and No-Follow RulesHaving examined how to configure a parser to require certain fields
and structure, and how values will be represented in the parsed
object, we introduce ways to make rules optional, take default values,
and have the parser ignore their substructure.A field flagged as optional can be omittedWhen instantiating and adding a new rule, we can specifyoptional=True:confp.add_rule('name')confp.add_rule('description',optional=True)The above parser would require anamefield but not a description
field, as in the following config file:name:Simple Single-File ServerThe parser will accept the file, create adescriptionfield, and assign
it the valueNone.conf.name# 'Simple Single-File Server'conf.description# NoneA program acting on theconfobject can assume that the optional
field exists, but it will have the valueNoneif it was not present
in the config file (or if it was explicitly assigned the valueNonesince the two are treated as equivalent).Optional fields can have required substructureA rule representing a nested block can be marked optional and still
have substructure with required fields.  For the sake of example, a
server requires three files in order to encrypt its communications
using SSL.  If aserver.sslblock is present in the config file,
those files must be provided, and the server will use SSL.  If the
block is omitted, the config file should still be accepted, but the
server will fall back to unencrypted communications.The following parser is configured with an optionalserver.sslblock
that, if it exists, must have three specific fields:confp.add_rule('server.addr')confp.add_rule('server.port')confp.add_rule('server.ssl',optional=True)confp.add_rule('server.ssl.key')confp.add_rule('server.ssl.cert')confp.add_rule('server.ssl.chain')In the following config, the optionalsslblock and its substructure
have been omitted:server:addr:127.0.0.1port:81Since the block was not included, thesslfield is present in theconfobject but assigned a value of None.conf.server.ssl# NoneIn the following config, the optionalsslblock and its substructure
have been included:server:addr:127.0.0.1port:81ssl:key:/etc/ssl/privkey.pemcert:/etc/ssl/cert.pemchain:/etc/ssl/full_chain.pemSince the block was included, its substructure was parsed and
validated.  Theconfobject includes the block and its substructure.conf.server.ssl# <YamlConfig object at 0x[...]>conf.server.ssl.key# '/etc/ssl/privkey.pem'conf.server.ssl.cert# '/etc/ssl/cert.pem'conf.server.ssl.chain# '/etc/ssl/full_chain.pem'The existence requirements on the substructure will only be checked
and enforced if the optional field is present.  In the following
config, the optionalsslblock is present, but it is missing one of
its required fields:server:addr:127.0.0.1port:81ssl:key:/etc/ssl/privkey.pem# cert: /etc/ssl/cert.pemchain:/etc/ssl/full_chain.pemWhenparse_fileis invoked on this config file, aParseErroris
raised:"server.ssl.cert" is missing.A default field will take a default value if omittedWhen instantiating and adding a new rule, we can specify adefault.confp.add_rule('server.addr',default='127.0.0.1')confp.add_rule('server.port')In the following config, the default rule has been omitted:server:port:81The parser will accept the file, create not only aportfield but
also anaddrfield within theserverblock, and since theaddrfield does not appear in the config, it will assign the default value
(127.0.0.1) to the field.conf.server.port# 81conf.server.addr# '127.0.0.1'Default substructure must still undergo validationThe following config rule will provide an entire server block if none
is specified in the config file:confp.add_rule('server',default={'addr':'127.0.0.1',port:81})Providing such structure is possible, but the above rule would
generate aParseErrorunless it was accompanied by rules to accept
theserver.addrandserver.portpaths.  With only the rule above,
a config file that triggered the default would raise aParseError:"server.addr" unexpected by parser.We need to add rules to prepare the parser for the substructure, as
in the following parser that accompanies the default rule with two
more:confp.add_rule('server',default={'addr':'127.0.0.1',port:81})confp.add_rule('server.addr')confp.add_rule('server.port')With these two additional rules, a config file will be accepted with
the default values ifserveris omitted, and it will require those
two values be present if aserverblock is present.  In both cases,
once parsing is successful, the program can assume thatconf.server.addrandconf.server.portexist.If we really did not want to validate the fields of the default
substructure, rather than adding rules for the fields, we could mark
the block as no-follow as described in alater
segment.Fields cannot both be optional and take a defaultTheoptionalanddefaultparameters toadd_ruleare mutually
exclusive; if both are specified, an error will be raised.
Essentially,optional=Trueacts like a default rule for which the
default value isNone.  In fact, settingoptional=Trueis the only
way for a missing field to be assigned a value ofNone, since
settingdefault=Noneis a no-op.  A value ofNonefordefaultactually signals that no default has been specified, so the field is
still required.It is unclear what the semantics would even be for an optional rule
that also takes a default, so the pairing is just not allowed.A default pathcanhave optional subpaths and vice versaIn the following parser configuration, theserverblock takes a default,
while theserver.sslblock is optional:confp.add_rule('server',default={'addr':'127.0.0.1','port':81})confp.add_rule('server.addr')confp.add_rule('server.port')confp.add_rule('server.ssl',optional=True)confp.add_rule('server.ssl.key')confp.add_rule('server.ssl.cert')confp.add_rule('server.ssl.chain')We can see what will happen in the following config, where theserverfield is omitted.  Note that this config file uses the
convention that leaving a field value blank assigns it a value ofNone(ornullin YAML terms), and that causes it to be treated as
omitted byyamlfig:server:The parser above will accept this config file.  Since theserverblock has been omitted, it will substitute its default value.  Since
theserver.addrandserver.portfields are provided by the
default, they will pass the rules requiring their existence.  Theserver.sslfield has not been provided by the default, but since
it is flagged as optional, the field will be created and assigned a
value of None:conf.server# <YamlConfig object at 0x[...]>conf.server.addr# '127.0.0.1'conf.server.port# 81conf.server.ssl# NoneThe reverse is also true.  Default fields can be included in the
substructure of an optional field, and they will take the default
values if they are omitted from the config but the optional block is
included.  Other combinations work as well (e.g., default fields
within default blocks; optional fields within optional blocks;
optional fields within default blocks within optional blocks; etc.).If it helps, we can think of optional and default flags being handled
from the top down in a cascade.  If a parent field is omitted, it will
be checked foroptionalordefaultflags first.  If it is
optional, the field will be created with aNonevalue and the
parsing will move on.  If it takes a default, the field will be
created with the default value or substructure, and the parser will
descend into that substructure, checking those fields and values before
moving on.  The parser will only encounter child fields and values
after the parent's optional or default nature has been handled.Default block and optional wildcarded path recognizezero-or-moreAs noted earlier, a wildcard rule path requires that a config file
have one or more fields matching the path.  By default, a wildcard
rule will raise an error if there are no fields matching it, but there
are times when we want to accept zero-or-more matches.The following rules configure a parser for cases where we want a block
with zero or more subfields:confp.add_rule('server.upload_paths',default={})confp.add_rule('server.upload_paths.*',optional=True)Theoptionalflag on the wildcard path will cause the parser to
allow theupload_pathsblock to contain no fields.  The default on
theupload_pathsfield will create that empty block if the field is
null.  The following config file would be accepted by this parser:server:upload_paths:By leavingupload_pathsnull, we signal that it has been explicitly
omitted, so it is given its default value (i.e., an empty mapping).
Then, since the wildcard path for the fields withinupload_pathsis
flagged as optional, that rule will be satisfied with zero fields.
TheYamlConfigobject would exist but contain zero fields:conf.server.upload_paths# <YamlConfig object at 0x[...]>len(conf.server.upload_paths)# 0For completeness sake, the same parser would also accept a config
file with one or more fields withinupload_paths:server:upload_paths:alice:/home/alice/uploadsbob:/home/bob/publicIn this case, theYamlConfigobject would contain two fields:conf.server.upload_paths# <YamlConfig object at 0x[...]>len(conf.server.upload_paths)# 2set(conf.server.upload_paths)# {'alice','bob'}This pattern—with the block taking an empty substructure as the
default and the wildcard rule flagged as optional—is the
recommended way to implement a parser that accepts zero-or-more fields
or list elements.A path marked no-follow can have any and arbitrary substructureIn some cases, we want to stop ayamlfigparser from attempting to
validate a substructure, either because the program is designed to
handle whatever is beneath that value or, more often the case, the
actual structure follows a complicated syntax, but the program will be
passing that structure to another package, and it has its own
functions for validating the input.For example, consider an example where a program needs to pull back a
list of projects from a MongoDB database.  The following config file
provides values that might be needed to (1) access the database, (2)
reference the specific collection within the database, and (3) filter
the results to only a subset of all projects, using a MongoDB query:mongodburl:mongodb://192.168.1.200:27017/collection:projectsfilterquery:{'is_private':{'$ne':true}}All three fields must exist, but thefilterqueryfield contains a
MongoDB queryasits substructure.  MongoDB queries can be expressed
as JSON objects, and YAML syntax is a superset of JSON, so the query
can be expressed as JSON/YAML right within the YAML config file.By default, theyamlfigparser will try to validate that object
(i.e., check whether the pathsfilterquery.is_privateandfilterquery.is_private['$ne']are expected by the parser).
Configuring aconfpparser to correctly validate the syntax of an
arbitrary MongoDB query is impossible and an unnecessary waste of
complexity.  As soon as the script hands the query off to MongoDB, it
is going to do a much better job of validating it.The following parser is configured to accept the above config file:confp.add_rule('mongodburl')confp.add_rule('collection')confp.add_rule('filterquery',nofollow=True)This parser will require that afilterqueryfield exists along with
themongodburlandcollectionfields, but thenofollowargument
ensures that it will not descend into the substructure within thefilterqueryfield.  No additional validation of that substructure
will take place.The value atconf.filterqueryis a standard pythondictwhich can be
passed to a MongoDBfindcommand as-is.conf.filterquery# {'is_private': {'$ne': True}}As an aside, note that until this example, we have been using YAML
block-structure syntax rather than JSON syntax, but there is no
difference between the two formats once parsed.  The config file
above could have been written equivalently as follows:mongodburl:mongodb://192.168.1.200:27017collection:projectsfilterquery:is_private:"$ne":trueA distinct alternative would have been to encode the MongoDB query
object as a string, as in the following line:filterquery:"{is_private:{$ne:{true}}}"`While a viable alternative, there are benefits tonotdoing so.  By
storing the query object as a query object, we actually do perform
some syntax checking at parse time, before handing it off to MongoDB.
We ensure that the brackets are balanced and the JSON is legal.  We
also get whatever syntax highlighting our editor provides to
YAML/JSON.  A string would simply be treated as a string by theyamlfigparser, and we would eventually have to invoke a JSON
parser ourselves.A path marked no-follow can also be optional or take defaults (but not both)Thenofollowparameter really affects the handling of the value not
the field, whereasoptionalanddefaultare parameters that affect
the handling of the field (i.e., what to do if it is omitted).If a field is omitted from a config, and if its path is marked in the
parser as both optional and no-follow, the field will be created and
assigned the valueNone.  If it were not optional, an error would be
raised.  Since the valueNoneis terminal and has no substructure,
being marked no-follow has little effect.If a field is omitted from a config, and if its path is marked in the
parser as no-follow and taking a default, the field will be created
and the default value will be substituted.  If the default value has
substructure (i.e., it is adictor alist), then the no-follow
marking would apply and no additional validation would be performed by
the parser on that substructure.A path marked no-follow cannot have any subrulesA parser configuration such as the following would raise an error:confp.add_rule('filterquery',nofollow=True)confp.add_rule('filterquery.is_private')The no-follow condition on a path means that no rules on descendant
paths will ever be checked or validated, so we prevent such rules from
being added.  For this example, aValueErrorwould be raised
explaining that"filterquery.is_private" is a descendant of a no-follow rule.Path Type CheckingWithin theyamlfigparser, after establishing that every required
field exists, that every optional or default field has been handled,
and that there are no unexpected fields, the parser's next step is to
check that any type assertions on the values for each field are
satisfied.Ensure that a field is a str (or int or bool or float, etc.)When instantiating and adding a new rule, we can specify apath_type.confp.add_rule('server.addr',path_type=str)confp.add_rule('server.port',path_type=int)In addition to requiring that theserverblock contains anaddrfield and aportfield, these rules will further check that the
values are instances of the givenpath_typetypes.The following config has a null in theaddrfield:server:addr:~port:81The above parser would raise an error:"server.addr" has type NoneType not type str.Likewise, the following config has a string in the port field:server:addr:127.0.0.1port:"81"The above parser would raise an error:"server.port" has type str not type int.The type that a value takes is determined by the underlying raw-YAML
parser thatyamlfiguses.  By defalt, we useSafeLoaderwithinPyYAML.  It recognizes the
following types:boolstrunicode(in Python 2, when the value contains non-ASCII characters)intlong(in Python 2, when the value is larger thansys.maxint)floatdate(in thedatetimepackage)datetime(in thedatetimepackage)dict(for mappings and mapping-like tags)list(for lists and list-like tags)NoneType(i.e.,path_type=type(None))Any of those types could arise in a config and be accepted or rejected
by apath_typeargument.  Additionally, it is possible to replaceSafeLoaderwith a different YAML parsing class, in which case the
set of types would depend on what types it constructed.Union types handle complex types likea numberora stringIn the following config, thetimeoutfield will be parsed as a float:server:timeout:1.2But in the following config, thetimeoutvalue will be parsed as an int:server:timeout:1Assuming the underlying server wants a float but handles the
conversion of an int all by itself, we don't really want to force a
user to add spurious decimal points (e.g., changing1to1.will
ensure the parser returns a float) just to consistently achieve a
single type across all configs.The following parser configuration rule will accept atimeoutthat
is either anintor afloat:confp.add_rule('server.timeout',path_type=(int,float))By specifying a tuple of types, we can directyamlfigto accept
values that are instances of either type.This feature was a lot more urgent in Python 2, where we almost always
wanted a string to be checked against(str, unicode), so that the
appearance of a word with an accent or umlaut in a descriptive string
wouldn't suddenly cause our config file to be rejected.  Things have
gotten calmer with Python 3 (e.g.,strvsunicodeandintvslongare no longer issues), but union types do still arise (e.g.,intvsfloat).Ensure that a path contains a map or a listConsider this parser configured to accept mappings from project names
to descriptions:confp.add_rule('projects',path_type=dict)confp.add_rule('projects.*')Contrast it with this parser configured to accept lists of authorized
user names:confp.add_rule('users',path_type=list)confp.add_rule('users.*')In both cases, the wildcard rule accepts one-or-more arbitrary fields
within the block, but as noted in anearlier
segment,
the wildcard does not distinguish a map from a list.  Thepath_type=dictconstraint is what ensures that the first parser
accepts config files with a map, like this:projects:ProjectX:"ProjectXisaneXtremeproject(formoreinfotalktoAlice)"meerkat_works:"Bob'snot-quiteskunkworksproject"Thepath_type=listconstraint is what ensures that the second parser
accepts config files with a list, like this:users:-alice-bob-carolSince few programs are written to expect either a mapping or a list,
we typically want to use type checking to ensure that a config-file
block contains the expected structure.A config file itself can be verified as either a list or a mapWhile we have so far considered YAML files with a map structure at the
root level, a YAML file could also be a list:-addr:192.0.2.200port:81-addr:192.0.2.201port:81-addr:198.51.100.15port:8080-addr:203.0.113.130port:8080In this example, these address-port pairs might be a list of mirrors,
ordered by proximity.  The following parser is configured to check that
the YAML file itself is a list, and then that each element of the list
has the proper substructure:confp=YamlConfigParser(path_type=list)confp.add_rule('*.addr',path_type=str)confp.add_rule('*.port',path_type=int)After reading and validating the above config file, this parser
returns aYamlConfigListobject:len(conf)# 4conf[0].addr# '192.0.2.200'conf[0].port# 81Note that in this example, we actually included the instantiation of
theYamlConfigParserasconfp.  All our previous examples (afterBasic Usage) assumed that step.  But when we are
making assertions about the top-level object parsed from the config
file, those are configured as part of theYamlConfigParserinstantiation.Down deep, every rule that gets added to the parser is of typeYamlConfigRule.  TheYamlConfigParserclass inherits fromYamlConfigRule, and any arguments are used to validate the
root-level object rather than any particular field or path within the
object.  The primary difference between the rootYamlConfigParserobject and theYamlConfigRuleobjects that are added to it is that
theYamlConfigParsercannot have arule_pathwhile the other
objects must.  Additionally, the root-level object cannot be optional
or take a default.  (It can be flagged no-follow, though.)A config file cannot be an atomic value; it must be a map or listAyamlfigparser will not accept a config file without any
substructure.  Technically, this is a valid YAML file:42A standard YAML parser will parsed it as anint.  However,yamlfigwill raise an error:config is a(n) int but a record or list is expected.Honestly, if our program must accept config files consisting of a
single value,yamlfigmight not be the right tool for the
situation.  If we still desired to make a go of it, we could nest that
value in a single-field mapping, like so:number:42The following parser would accept that config file, with a single
field, the value of which is a list:confp=YamlConfigParser(path_type=dict)confp.add_rule('number',path_type=int)Even more concisely, we could nest the value in a singleton list:-42The following parser would accept such a config file:confp=YamlConfigParser(path_type=list)confp.add_rule('0',path_type=int)Note the space between-and42.  Without it, the singleton list
collapses back to a single (negative) integer:-42Such a single value would not be accepted.Rule Test FunctionsWhile type checking helps validate the values in a config file, we
often want to place additional constraints on those values.  For
instance, we might want a value not only to be anintbut to fall
within a particular range.  We might want another value not only to be
astr, but also to match a regular expression.  We might want a
third value not only to be astrbut also to point to an existing
file.When adding a rule to a parser, we can specify a test function using
thetestargument, to perform additional checking of values.  Theyamlfig.testpackage contains a variety of pre-packaged test
functions for some common validation scenarios.Verify that a value matches a regular expressionThis parser rule will constrainusernameto start with a lower case
letter and be followed by zero or more lowercase letters or numbers:confp.add_rule('username',test=test.is_regex('^[a-z][a-z0-9]*$'))This config file would be accepted by such a parser:username:'carol57'This config file would be rejected:username:'CarolC.'The parse error would include the explanation"username" failed test: "Carol C." does not match /^[a-z][a-z0-9*$/.Verify that a value is an IPv4 addressThis parser rule will constrain theaddrfield of theserverblock to
be a valid dotted-quad IPv4 address:confp.add_rule('server.addr',test=test.is_ipv4_address)Test functions packaged within yamlfigTheyamlfig.testmodule that contains various common validation
tests has been directly imported as thetestobject in the examples
of this documentation, but would otherwise be accessed asyamlfig.test(e.g.,test=yamlfig.test.is_regex('^[a-z][a-z0-9]*$')).The following test functions are available in theyamlfig.testmodule:is_interval(lower, upper)verifies that the value is within the
range defined by the lower and upper bounds;is_regex(regex, invert=False)verifies that the value matches the
regular expression (or does not match it, if inverted);is_ipv4_addressverifies that the value is an IPv4 address in
dotted-quad notation;is_domain_nameverifies that the value conforms to the
specification of a DNS domain name (which, note, is a looser
constraint than that it be an actual operating and reachable domain
name);is_email_addressverifies that the value (roughly) conforms to
the specification of an email address;is_urlverifies that the value (roughly) conforms to the
specification for URLs;is_file_path(*ostests)takes one or more strings corresponding to
properties of filesystem objects, interprets the value as a
filesystem path, and verifies that the path satisfiesallof the
listed properties.  Properties include:'exists'and'!exists': the path exists (or is not);'isdir'and'!isdir': the path is a directory (or is not);'isfile'and'!isfile': the path is a file (or is not);'islink'and'!islink': the path is a symlink (or is not);'ismount'and'!ismount': the path is a mount point (or is not).Note that for higher-order functions (i.e., the ones that return the
test functions suitable fortest), not all optional parameters are
shown and described.  Check the help documentation for each function
for additional detail on usage and options.Writing our own test functionsThetestparameter toadd_ruletakes a function with three
parameters:conf,path, andvalue.  It signals acceptance of the
value by returning None, and rejection by returning a string
explaining what caused the failure.Some of the test functions packaged withinyamlfig, liketest.is_ipv4_address, directly match that specification.  Others,
liketest.is_regex, return a function that matches the specification
based on their arguments.In anearlier
segment,
we explained that rule paths could not containpartialwildcards
(e.g.,user-*to require that all fields start with a particular
prefix).  However, test functions do offer a way to recognize such
properties and more.  Consider a config file in which a block must
contain adefaultfield, and can optionally contain zero or more
fields that must all match a partial wildcard likeuser-*.Test functions offer a way to implement any test that we can write as
a Python function:defhas_default_and_user_fields(conf,path,value):ifnot'default'invalue:return'"default" field is missing'forfieldinvalue:iffield!='default'andnotfield.startswith('user-'):return'"{0}" is neither "default" nor starts with "user-"'.format(field)returnNoneconfp=YamlConfigParser()confp.add_rule('uploads',test=has_default_and_user_fields)confp.add_rule('uploads.*',path_type=str,optional=True)The following config file would be accepted by this parser:uploads:default:/var/share/SomeApp/uploadsuser-alice:/home/alice/uploadsuser-bob:/home/bob/publicThe following config file would be rejected by this parser:uploads:user-alice:/home/alice/uploadsuser-bob:/home/bob/publicThe parse error would include the explanation"uploads" test failed: "default" field is missing.Of the three parameters (i.e.,conf,path, andvalue), all of
the packaged test functions depend only on thevalue, and that will
typically be the case.  By providing the entireconfobject as well
as thepathto the value being verified,yamlfigenables the
test to evaluate the value in the context of the rest of the config
file if necessary.Warnings and CaveatsField names with leading underscores– While accessingYamlConfigfields as attributes is convenient, the drawback is
that any field names that start with a leading underscore risk
colliding with the methods and attributes that implement the class.
Consequently, if the parser encounters any field that start with'_', a warning will be raised.  The warning can be suppressed by
settingyamlfig.print_underscore_warning.off = True.  As with
non-string field names, we can always look up a field with leading
underscores via index lookup (e.g.,conf["_field"]rather thanconf._field), but the warning is intended to make us aware of the
possibility for collision.References totransformfunctions in code and documentation– The API for theYamlConfigRuleclass and theYamlBaseConfigclasses expose references to atransformfunction or the ability to invokedo_transformon the container
object.  The ability to specify a transform is planned (and
described in more detailbelow).  As the code
indicates, work on this feature was already underway when this
version was released, but consider it untested, incompletely
documented, and subject to change.Next StepsFuture WorkSeveral features are already on our list of things we would like to or
have started to implement:Allow standard fielded rules to co-exist alongside wildcard rules,
with the standard rule taking precedence if it matches and the
wildcard being used as a catch-all.  We probably still want to
disallow partial wildcards since (a) they can already be handled
with test functions, and (b) they would raise the possibility of
allowing multiple partial wildcard rules attached to the same
parent path, and that would raise all sorts of ambiguity about
what to do if multiple rules match the same field.Allow a user to more easily specify a constraint on field names,
for instance afield_typeto do the same type checking on a
field thatpath_typedoes on the value, and/or afield_regexto specify a pattern that the field must match.  Currently these
are possible, but would require the user to implement their own
rule test, as described above.Add options totest.is_file_paththat (1) allow the user to
specify a directory from which all relative paths are resolved,
and (2) allow the user to specify a path intoconfwhere such a
base directory would be stored.  These would enable support for a
config file where onehomepathfield specifies where the program
willchdirto, and then all the other paths (e.g.,dbfileorhtmldirare specified relative tohomepath).Extend therule_pathspecification to allow us to express field
types that are not strings or are strings that include 'special
characters' like whitespace, the delimiter (.), or the wildcard
(*).  Right now, we're thinking of using square brackets in a
rule path, so that the string resembles the path that would be
used to access the value once parsed (e.g.,rule_path="dbhosts['192.0.2.1'].port"would indicate the config
file had a structure where adbhostsblock contained a field
field named192.0.2.1which maps to a sub-block that has aportfield).Implement atest.is_in_choicesetwhich verifies that the value
is one of the configured options or choices.  So, if a rule for adbtypepath hadtest=test.is_in_choiceset(['sqlite', 'mysql', 'mongodb']), it would verify that thedbtypevalue took one of
those values, returning an explanatory error message if not.ExtendYamlConfigParserwith a function to write a config-file
template (or and actual config file if provided with aconfobject) to use for the values.  For each rule, it is already
possible to specify adescdescribing the purpose of the path,
and anexamplevalue.  These values could be incorporated into
the config-file template, making it somewhat self documented, and
making it easier for a program to provide its users with a
template.  In truth, we would probably want to implement this as aRepresenterclass that inherits from and extendsSafeDumper,
but with that class invoked by something likeconf.write_file(filename, conf=None).ExtendYamlConfigRulewith support for transformations.  A goal
foryamlfigwas to gather into one package all of the things
that we find ourselves doing over and over again when we read in a
config file (e.g., checking whether fields exist, that they have
the right types, and that they meet various other conditions).
Another thing we do at this stage is converting them to the
objects that we really want to use in our program.  For instance,
we don't really want the path to the log file, we want the open
filehandle to it; we don't really want the IPv4 address in
dotted-quad notation, we want theIPv4Addressobject that we can
construct with it.  Some initial groundwork for such
transformations already exists within the code, but it needs to be
built out and tested (lots and lots of testing, especially the
interaction with the write-out-configs extension described above).This list is neither exhaustive nor a promise of what is certain to
come.  Other suggestions are also welcome, of course, too.Support and CollaborationWe welcome reports of issues and other contributions through our
package's page on GitHub:https://github.com/ktkalaru/yamlfigNote that this is our first open-source project, and it was shared in
part so that we could get more experience with the standard tools and
workflows.  We aim to respond to any issues, requests, or other
feedback promptly and professionally, but some understanding may be
required since we are learning as we go.