Package: yamlicious
Yamlicious is a (work-in-progress) lightweight configuration library built on
top of YAML. It’s for folks who love to write their configuration files in
YAML, but who find themselves writing additional boilerplate each time they
want to use YAML to encode a configuration file.In addition to a parser, yamlicious provides code that:Validates the correctness of a file and prints an understandable failure
message when the file doesn’t quite matchUnderstands your environemnt, and allows you to configure different properties
for different environment dimensions without unnecessary DRY-violationReads configuration from multiple filesProvides a cascading set of configuration sources, including environment
variablesLazily loads certain parts of a large configuration. (read: when your
“configuration” really starts to feel more like “data.”)Is generically extensible to new behavior, just in case you need to add
something we haven’t thought ofYamlicious does all this by being like any other YAML parser, but by treating a
handful of specific keys in a special way. It can be configured to recognize
and evaluate any subset of thesefeature keys, allowing a developer to bake
in as much (or as little!) of the crazy capabilities as seems reasonable for
the situation. (The craziest stuff is turned off by default.)Skip toFeature Key Definitionsif you want the formality without the
English verbiage.ContentsThe Yamlicious FileEnvironment Variable SubtitutionWhen substituting into a single value stringWhen substituting into a value listWhen substituting into a key stringWhen substituting multiple list values into the same stringLoad Another FileMergeInsert and MergeMerging Entire DocumentsChanging the EnvironmentThe Default DocumentFunctional Conditional KeysLazy LoadingOrder Of OperationsFeature Key DefinitionsDocument Feature KeysenvincludeFunctional Feature Keysinsertmergemerge_overrideinsert_mergecaseotherwisecondlazylazy_lookupThe ValidatorThe CommandThe APICustom Feature KeysThe Yamlicious FileA configuration file specified in Yamliciousis just YAML. In fact, nothing stops
you from parsing a Yamlicious file with any YAML library. This can be useful for
searching, cleaning, or editing (with syntax highlighting) the file.Environment Variable SubtitutionYamlicious gives your configuration file automatic access to the environment.some:key:user_$(USER):not cooluser_kyle:coolDon’t worry. The Yamlicious API gives you (the developer) the ability to both
limit the environment variables your configuration file is allowed to access
and provide overriding values for anything in the environment. You’re neither
forced to use environment variables nor doomed to a free for all.Yamlicious supports list values in the environment, and splits environment variables
on the comma character. If a Yamlicious file substitutes a list variable into a
string, that string renders into multiple strings (one for each value in the
list variable).If a string references a variable that’s not in the environment, yamlicious
does not modify the string. It will leave the variable reference alone,$()characters included. This makes string substitution have a neat property – it
can be applied to the same document iteratively as the environment grows.List substitution behavior varies subtly by situation.When substituting into a single value stringsome_list:$(MY_LIST)renders to the following, ifMY_LIST=one,two,threeis in the environment.some_list:-one-two-threeWhen substituting into a value listsome_list:-first-$(LIST)becomessome_list:-first-one-two-threeWhen substituting into a key stringKey strings are special, because you almost certainly don’t intend to make a
list into the key of a dictionary. Instead, you likely mean to define a key in
the dictionary for each item in the list. Yamlicious provides a special
variable in the environment,_KEY, to help you out in this situation.$(LIST):$(_KEY) is in the list!becomesone:one is in the listtwo:two is in the listthree:three is in the list_KEYis set to the first-level key in the document, regardless of whether
the key was derived from string substitution. To get the second-level key, use__KEY, and so forth:$(LIST):second_level_key:$(_KEY) on top and $(__KEY) on bottombecomesone:second_level_key:one on top and second_level_key on bottomtwo:second_level_key:two on top and second_level_key on bottomthree:second_level_key:three on top and second_level_key on bottomWhen substituting multiple list values into the same stringThis is interpreted as a dot product. Yamlicious will substitute every
combination of variables between the two lists.IfBOYS=joey,johnny,bobbyandGIRLS=sally,marythen:"$(BOYS)likes$(GIRLS)"becomes:-joey likes sally-joey likes mary-johnny likes sally-johnny likes mary-bobby likes sally-bobby likes maryNote – the rest of the “positional” list substitution rules (defined in the
immediately previous sections) apply to dot product substitutions.Load Another FileSometimes, it makes sense to define sub-configuration somewhere outside the
main configuration file. (e.g., secrets go somewhere special.) Yamlicious gives
you theinsertkey to accomplish this.Note:insert_is afunctional feature key. (Defined more in theFunctional
Feature Keyssection.) These are keys that participate in something like a
function call – the entire map that contains a functional key evaluates to
functional behavior applied to the key’s value. (No project is complete without
a smidge of functional programming.) You can only use one of these keys in a
map at a time because yamlicious replaces the key-containing map with a
document – the result of the function applied to the key’s value. Multiple
keys is an abiguous definition.some_place:placed_here:_insert:other/file.yamlIn this case, the rendered YAML output ofother/file.yamlis placed under
theplaced_herekey.some_place:placed_here:contents of:-that other file-which can be arbitrary YAMLYou can use variable substitution with theinsertfeature to get conditional
configuration.user_settings:_insert:$(USER)/conf.yamlMergeYamlicious allows you tomergean external file into a bit of config.merged_settings:_merge:-some_list:['thing']some_thing:'thing'-_insert:some_other_place.yamlWhen you ask Yamlicious to do this, it will use a strategy I callsafe deep merge
with list append. Yamlicious merges dictionaries recursively by combining their
key-value pairs. It merges lists by list addition. It refuses, however, to
merge anything else. (Anything else would be shoot-self-in-foot territory, and
I’d rather not encourage it.)ifsome_other_place.yamllooked like this:some_list:['second_thing']some_other_thing:'thing'The above configuration would render as follows:merged_settings:some_list:['thing','second_thing']some_thing:'thing'some_other_thing:'thing'If you’re looking to implement the commondefault overridepattern, specifyThe Default Documentas part of the Yamlicious API. That feature is specifically
built to help you not have to allow arbitrary overrides when including files.
If you absolutely must allow overrides, use themerge_overridekeyword,
but note that it is turned off by default.Insert and MergeLoading several files and merging them is a common pattern, and it would be
nice if folks didn’t have to be verbose if that’s the behavior they’re looking
for. This is what theinsert_mergekey is for.merged_stuff:_insert_merge:-first/place.yaml-second/place.yaml-third/place.yamlThis key will load each file in order and merge that file into the previous
file.Merging Entire DocumentsIf you’d like to merge an entire document with your own, use theincludefeature key.Note:includeis adocument feature key. (Defined more in theDocument
Feature Keyssection.) Unlike functional feature keys, which apply behavior
to any map embedded anywhere in the document hierarchy, document feature keys
apply behavior to the entire document, and therefore must exist at the top of
the YAML document.Changing the EnvironmentYou can also use theenvdocument key to place new variables into the
environment._env:COOLEST_PERSON:kyleThese variables can be used either inthe same document(although the utility
of that is not immediately obvious, other than for mitigating DRY violation)
or, more importantly,in documents that include it. Yamlicious supports this
by taking special care to re-run string substitution each time it changes a
document’s environment. (Remember, string substitution is idempotent.)This behavior is somewhat dangerous if the included document defines a variable
that’s already defined in the including document. If the including document
uses string subtitution to define included document paths, those substitutions
can happen using only theinitialversion of the environment (before it is
mutated by the act of inclusion). If the included document then changes any key
that’s used in the process of inclusion, things get hard to reason about.Rather than allow such craziness, Yamlicious bans it. That is, it does not
allow multiple documents included in the same parent document to define
differing versions of the same environment variable. It does allow actual
environment variables to coexist with (and override) those defined inenv.
Not allowing this would be brittle and would remove a very common use case,
where setting an environment variable changes some sort of important behavior.The Default DocumentYamlicious merge-overrides the configuration document it renders with adefault documentthat it is configured to use.This is the only place that, by default, uses the merge-override (rather than
safe merge) behavior. For that reason, it’s best to use the default document
feature to specify override behavior. If you’re wanting override behavior that
can’t be done by using the default document, chances are you’re doing something
that’s either too complex or wrong. If you insist, there’s alwaysmerge_override.Functional Conditional KeysTo specify a condition in-line, you can use thefunctional conditionalfeature keys (caseandcond), each inspired by Lisp. This adds a bit too
much Turing completeness to the project for the taste of most, so these are
disabled by default.case_configuration:'_case':-'$(USER)'-{'kyle':'isawesome'}-{'_otherwise':'isnotawesome'}cond_configuration:'_cond':-{"$(ENV)in['test','prod']":'go!'}-{true:'undefined'}Note the use of the python expression. This is mostly for convenience and
terseness. Nobody wants to write a boolean expression in YAML, and I don’t
particularly want to implement it, either, so Yamliciouseval()s every single
string that it finds below either functional conditional key.List substitution works in both kinds of functional conditional. For example,
ifGOOD_USERS=kyle,anthony, then the following expressionaccess_configuration:'_case':-{'$(GOOD_USERS)':'go!'}-{'_otherwise':'stay.:('}evaluates toaccess_configuration:'_case':-{'kyle':'go!','anthony':'go!'}-['_otherwise','stay.:(']Yamlicious is careful to “do the right thing” here. While there is no defined
order in how it matches either the key'anthony'or'kyle', it will try
to match both before falling back to theotherwisekey.Be careful to not do something like this unless you really mean it:access_configuration:'_case':-{'$(GOOD_USERS)':'$(_KEY)'}-{'_otherwise':'stay.:('}While it will technically work, Yamlicious offers no definition for what the
above expression evaluates to – the order of iteration for a map/dictionary is
an implementation detail.Lazy LoadingIf you notice an explosion in the number of Yamlicious files that your program
includes, and you also notice that only a few of them ever get used, you’ll
likely want to conditionally load said files only when they’re needed. Yamlicious
provides two lazy loading keys to help you with this.Thelazykey changes nothing about the semantic meaning of the document it
points to. It does change the time when functional key evaluation happens.
Yamlicious evaluates embedded functional keys atlookup time, rather than
during the depth-first functional key evaluation of the entire document.In this example_lazy:one:_insert:some/other/file.yamlTheinsertevaluation happens only when someone tries to look at theonekey.Thelazy_lookupkey delays functional key evaluation just likelazy, and
it also allows you to use string substitution of the special variable$(_KEY)to define how every key in the document is looked up. Rather than
defining a document forevery keyin the map, you defineone expressionthat, after string subtitution, can evaluate toany key.To get the most power, pair lazy lookup with file inclusion. Here’s an example
inspired by YAML configuration of SQL tables.tables:_lazy_lookup:_insert_merge:-generic/schema/$(_KEY).table.yaml-$(SYSTEM)/schema/$(_KEY).table.yaml-$(INSTITUTION)/schema/$(_KEY).table.yamlNote that there’s nothing that prevents lazy-loaded documents from merging with
one another. If you’re feeling particularly masochistic, you can define this
confusing yet equivalent thing.tables:_merge:-_lazy_lookup:_insert_merge:-generic/schema/$(_KEY).table.yaml-$(SYSTEM)/schema/$(_KEY).table.yaml-_lazy_lookup:_insert_merge:-$(INSTITUTION)/schema/$(_KEY).table.yamlOrder Of OperationsYamlicious goes through the following phases when processing a document:String substitution.Document key evaluation.String substitution.Functional key evaluation (depth-first).Feature Key DefinitionsEnough with your words. Let’s define this stuff explicitly.Document Feature KeysThese keys must be placed at thetoplevel of a document, and affect the
entire document that they’re placed inside. They disappear when rendered.env_env:<variable>:<value>...Sets document environment variables to given values.include_include:-<file-path>-<file-path>-...Loads and safe-merges several files into the document.Functional Feature KeysThe key must exist by itself in its containing dictionary. The feature key,
itself, describes a transformation operation on the given document._<feature-key>:<document>insert_insert <file-path>Evaluates to the loaded and processed configuration document found atfile_path.merge_merge:[<document>,<document>,...]Uses safe-merge-with-list-append to merge given documents together. Can safely
merge dictionaries and lists, but nothing else.merge_override_merge_override:[<document>,<document>,...]Uses deep-merge to merge given documents together. Can safely merge anything.
For scalar values, documents further down the list override documents earlier
in the list.insert_merge_insert_merge [ <file-path>, <file-path>, ... ]Loads files and then merges them with safe-merge-with-list-append.case(NOT IMPLEMENTED)'_case':-<key-python-expression>-{<match-python-expression>,<outcome-python-expression>}...Functional case. Evaluates to the first outcome expression whose match
expression is python-equal to the key expression.otherwise(NOT IMPLEMENTED)'_otherwise':<expression>Evaluates to a case condition that always matches.cond(NOT IMPLEMENTED)'_cond':-{<boolean-python-expression>,<outcome-python-expression>}...Functional cond. Evaluates to the first outcome expression whose boolean
expression is true.lazy(NOT IMPLEMENTED)'_lazy':<document>Evaluates to document, but where each of the keys in document is lazy-loaded.lazy_lookup(NOT IMPLEMENTED)'_lazy_lookup':<value-expression>Evaluates to a lazy-loaded dictionary, where every key is evaluated at lookup
time by evaluating the value-expression, which is allowed to use the_KEYenvironment variableThe ValidatorTBDThe CommandYamlicious comes with a convenient command,yamlicious, that reads input
from stdin and writes to stdout. It uses a default configuration, along with
all environment variables, in order to process the yaml document fed to it on
standard in:[10:49:46][kderr@Kyles-MacBook-Pro][~/Repositories/derrley/yamlicious]
$ cat /tmp/test
Hello: "$(PWD) is the current wd"


[10:49:52][kderr@Kyles-MacBook-Pro][~/Repositories/derrley/yamlicious]
$ cat /tmp/test | yamlicious
{Hello: /Users/kderr/Repositories/derrley/yamlicious is the current wd}The APITBDCustom Feature KeysTBD