Package: awesomeyaml
AwesomeYamlAwesomeYaml is a Python-based library which extends standard yaml with a set of custom tags and a processing pipeline designed to make the process of building and manipulating yaml-based config files easier.
We use a popularpyyamlpackage as the core component to read and write yaml files.Note:AwesomeYaml isNOTdesigned to be safe! Be cautious when using it with externally provided content.ContentOverviewCapabilitiesInstallationRunning TestsGenerating documentaionQuick referenceLow-level APIDetailed descriptionYamlMergingYaml streamsEvaluationMetadataPythonBasic usageCommand lineAccessing dataTags summaryContributingReporting issuesLicenseCapabilitiesBriefly speaking,awesomeyamlis capable of:merging multiple (partial) config files into a common config object, allowing for:cross-referencing elements from different files and expressing dependenciescontrolling the merging process by defining 3 priorities and allowing the user to set them arbitralily for any entity within the config treemoving and deleting entire subtrees during the merging process for easier wrapping of componentsincluding files, supporting nested and within-the-tree includesdynamically evaluating content of the nodes in the config tree, supporting:f-string-like mechanism to evaluate stringsimporting and embedding python entities directly in the evaluated config treecalling python entities with the provided arguments (or binding arguments to them)evalauting arbitrary python code (UNSAFE!)attaching user-defined metadata to any node in the config tree via a generic yaml-compatible mechanismInstallationAwesomeyaml requirespyyamlandPython>=3.6.
You can installawesomeyamlfrom PyPI by simply issuing:python-mpipinstallawesomeyamlAlternatively, you can also perform installation by cloning this repo and installing directly from it with pip:gitclonehttps://github.com/SamsungLabs/awesomeyaml.git
python-mpipinstall./awesomeyamlOptionally, you can add-eoption to allow for in-place (also called editable or developer) installation:python-mpipinstall-e./awesomeyamlor, if you don't need the repo to be laying around simply install directly from github:python-mpipinstallhttps://github.com/SamsungLabs/awesomeyaml/tarball/master#egg=awesomeyamlRunning testsWe use a standardunittestpython package for writing tests.Sinceawesomeyamlis a pure python library with minimal dependencies and does not directly depend on any OS or HW specific components, it is rather unlikely that a version tested on one machine would break on another.
However, if you want to be extra safe, you can run tests provided with the repo by using the following command (run from the cloned directory):python -m unittest discover -v -s . -p '*_test.py'You should see output ending with something similar to:----------------------------------------------------------------------
Ran 118 tests in 0.131s

OKIf any errors have occurred, they should be mentioned at the end instead ofOK.
Pleaseraise an issueif you encounter any.Note:the tests are only shipped with the repo, so you need to clone it in order to run them. Installing the cloned code is not required as the testing code always uses the code provided in the repo (by alteringsys.path).Generating documentationTo automatically generate API documentation for the library, make suresphinxandsphinx_rtd_themeare installed first.
You can easily install them withpip:pipinstallsphinxsphinx_rtd_themeNote:sphinx_rtd_themeis only required if you want to use the readthedocs theme (the default one for this project).
You can change the theme used by modifying yourdocs/source/conf.pyfile.After installingsphinx, go todocs/and executemake html.
Your newly generated documentation should be available atdocs/build/index.html.Quick referenceUseawesomeyaml.Configfor default, out-of-the-box behaviour.
Specifically, to build a config from a sequence of yaml sources, use:importawesomeyamlasaywithopen('file2.yaml','r')asf:# read yaml from: filename, file object and inline yamlcfg=ay.Config.build('file1.yaml',f,'field: !!null')alternatively, you can let awesomeyaml parse part of your command line arguments and handle filenames and inline overwrites:importargparseimportpathlibimportawesomeyamlasayparser=argparse.ArgumentParser()# add some custom argumentsparser.add_argument('--debug',action='store_true')# identify arguments which should be used to build configparser.add_argument('yamls',action='append')args=parser.parse_arguments()# optional function to lookup yaml files in a non-standard way_lookup_dirs=['.','~/.project']deflookup(filename):fordin_lookup_dirs:candidate=pathlib.Path(d).joinpath(filename).expanduser()ifcandidate.exists():returnstr(candidate)returnfilenamecfg=ay.Config.build_from_cmdline(*args.yamls,filename_lookup_fn=lookup)Access fields in a dict by either standard indexing operator or by attribute names:cfg['field']is the same ascfg.field; consequently, access awesomeyaml API via a proxy object (namespace), e.g.:cfg.ayns.children()Low-level APIUseawesomeyaml.Builderto merge multiple yaml sources without evaluating dynamic nodes.
Useawesomeyaml.EvalContextto evaluate merged config:importawesomeyamlasay# merge multiple filesbuilder=ay.Builder()builder.add_source('file1.yaml')builder.add_multiple_sources('file2.yaml','field: !!null')cfg=builder.build()# evaluate dynamic nodes in cfgeval_ctx=ay.EvalContext(cfg)cfg=eval_ctx.evaluate()# at this point cfg should be roughly the same as cfg from the first exampleUseawesomeyaml.yamlfor core functionality of reading and writing yaml files and to add support for custom tags.importawesomeyamlasayimportawesomeyaml.yamlasayyaml## Read and write files with support for awesomeyaml tags#withopen('file1.yaml','r')asf:# optionally pass filename as well to make# e.g. include nodes know were they come from,# otherwise everything is done w.r.t to the# current working directory# you can also pass awesomeyaml.Builder instance# for more complete functionality,# this is what happens normally when using# higher-level functionsobj=ayyaml.parse(f.read())ayyaml.dump(obj,'file1_copy.yaml')## define custom yaml constructor## returned value should be derived from `awesomeyaml.nodes.ConfigNodeclassMyNodeType(ay.nodes.ConfigNode):passdefmy_node_constructor(loader,node):# expects scalar valuereturnMyNodeType(loader.construct_scalar(node))# ayyaml.yaml is the standard yaml package used by awesomeyamlayyaml.yaml.add_constructor('!mynode',my_node_constructor)See thesummary of the extended tagsfor an overview of extra tags implemented by awesomeyaml.Detailed descriptionIntroductionAwesomeyaml was originally developed to make it easier to work with multiple configuration files, therefore it shines especially when configuration has a modular nature -- i.e., different parts of the system can be configured independently from each other, but at the same time might introduce requirements for each other, etc.
A prominent example of such a system could be a machine learning project which supports multiple models, datasets and optimization techniques.
Usually, each of these elements can be configured independently from the rest, e.g., the dataset configuration can include parameters for a common preprocessing done for this dataset regardless of what model is used.
Analogically, the model configuration can also be done mostly independently from the dataset in use, however, it does contain some parameters which are strictly determined by the dataset - e.g., for an image classification network that would be a number of classes for the last layer.As mentioned before, to help dealing with situations like that, awesomeyaml is designed to work with configurations which are split into multiple files.
For example, following the machine learning analogy, the configuration could be split in the following way:+ configs
|   + datasets
|   |    dataset1.yaml
|   |    dataset2.yaml
|   |
|   + models
|   |    model1.yaml
|   |    model2.yaml
|   |
|   + optimizers
|        optim1.yaml
|        optim2.yaml
|
+ train.pywe would like to be able to calltrain.pywith a set of 3 config files by selecting any one of datasets, models and optimizers, e.g., let us train model1 with dataset2 and optimizer2:python train.py configs/models/model1.yaml configs/datasets/dataset2.yaml configs/optimizers/optim2.yamlor use the same setting but try a different optimizer:python train.py configs/models/model1.yaml configs/datasets/dataset2.yaml configs/optimizers/optim1.yamlIn both cases, the command line arguments define a sequence of config files which need to be processed to construct the final form of the config - we call this processbuildingand split it into two parts:1)mergingof the provided config files, and2)evaluatingdynamic elements.
The following code snippet presents the basic idea behind building a config object:defbuild_config(*yaml_sources):merged_cfg=merge(yaml_sources)evaluated_cfg=evaluate(merged_cfg)returnevaluated_cfgMerging of the yaml files is handed byawesomeyaml.Builderand evaluation is done byawesomeyaml.EvalContext, withawesomeyaml.Configbeing the top-level class encapsulating both steps in a standard way presented in the code snippet above.The following documentation is split into two parts.
The first talks about working with yaml files and how awesomeyaml functionality helps with that.
The second concerns awesomeyaml Python's API and how to achieve the behaviour explained in the first part.Yaml sideMerging yaml filesLet us begin by providing some examples of how the config files could look like/ For example, consider the content of themodel1.yamlfile like the following:---model:name:MyAwesomeCNNchannels:64stacks:3classes:!requiredThe file is a pretty straightforward yaml file which defines a set of parameters grouped under a common"model"key.
The only unusual thing is the non-standard!requiredtag - this tag is provided by awesomeyaml and can be used to identify parameters which are introduced (and required) by a specific part of the system but should be defined by other parts of the system.
In our case specifically, we want to say that our model needs to know how many classes are to be used but this value should be defined somewhere else (as mentioned earlier, this is defined by the choice of a dataset).
Following this, we could further define thedataset1.yamlfile as:---dataset:name:cifar100folder:~/data/cifar100process:random_vflip:0.5cutout:3normalize:[[0.4914,0.4822,0.4465],[0.2023,0.1994,0.2010]]model:classes:100In both cases the final config, which will be used by the training script, will bebuildby reading andmergingthe content of the consecutive config files in the provided sequence.Merging is performed by reading and combing files sequentially, strictly in a user-defined order (more about it can be found in the Python part).
We carefully define the rules of combing elements and provide ways of manipulating them which are explained in details later.
However, the most typical (and intuitive) example would be merging of two dictionaries where elements with common keys are overwritten with their value from the newer dict and all the others are taken from either of the dicts (no elements are removed).
This is what would happen most of the time when merging contentmodel1.yamlwithdataset1.yaml, since both files contain mostly simple dictionaries.
Specifically, if the merging order is defined as:model1.yaml <- dataset1.yamlthen the content ofdataset1.yamlwill be used to update content ofmodel1.yamland the expected result would be:---dataset:name:cifar100folder:~/data/cifar100process:random_vflip:0.5cutout:3normalize:[[0.4914,0.4822,0.4465],[0.2023,0.1994,0.2010]]model:name:MyAwesomeCNNchannels:64stacks:3classes:100Please note that the value ofmodel.classeswas overwritten with from the original!requiredto100.
ALso, please note that in general merging operation is non-commutativie, meaning that mergingA <- BandB <- Acan result in different outcomes.
For example, in our example above, if the merging order was swappedt he value100would be overwritten with!required.Yaml streamsYaml includes a built-in mechanism for including multiple "files" inside a single physical file.
This is achieved by separating contents with a line containing---.
Awesomeyaml supports reading yaml sources with more than one "file" inside them - we call such entities yaml streams (the term also covers multiple yaml sources defined outside of yaml).Yaml streams can be nested.
For example, consider a user-defined list of files:['file1.yaml', 'file2.yaml'].
This list is a yaml stream with two sources.
Now, supposefile1.yamlhas the following structure:---!includefile3.yaml---foo:!include[file4.yaml,file5.yaml]Here,!includeis one of the extended tags introduced by awesomeyaml.
The tag read content of a file and embeds it in the parent file - either as a top-level entitiy (file3.yaml) or as a subelement of a node (file4.yamlandfile5.yaml).
If more than one file is provided to!include, the files are merged recursively before being embedded.
Putting it all together, after considering nested stream, our example expands from a single stream of two files to nested streams which need to be merged in a correct order to eventually form a single, final config.
The snippet below roughly illustrates how this process progresses:1. file1.yaml <- file2.yaml
2. [file3.yaml <- { foo: [file4.yaml <- file5.yaml] }] <- file2.yaml
3. file3.yaml <- { foo: content } <- file2.yaml
4. final_configEvaluation of nodesApart from merging, another important aspect of awesomeyaml is support for lazily-evaluated nodes.
The requirement behind laziness comes from the fact that when a single yaml file is parsed - that is, before merging - the final content of the config object is not yet defined, so any node who content depends on another node's value could evaluate to an undesired value.
This is also one of the reasons why evaluation happens after merging is performed.Consider the following example, where we add a short section to the config file defining the experiment directory:---fs:root:~/.projectexp_folder:!path[!xreffs.root,!xrefmodel.name]We define the experiment folder (fs.exp_folder) with respect to the root folder (fs.root) by appending the selected model's name.
By doing so, each model will have it's own experiment directory.
Obviously, we could create a deeper structure by considering other elements, like dataset, but for the purpose of this example one extra level is enough.The!pathnode in this example is used to define and manipulate paths - in this specific case, during evaluation phase, it will try joining the two elements in its list to form the final path.
The!xrefargument, on the other hand, is used to cross-reference another node in the config - that is, use its value in another place.
Both of these nodes are examples of nodes which are evaluated lazily.
You can see the reason why in the example itself - even though the first parameter infs.exp_foldercould be evaluated eagerly, the second one references a node which does not exist in this file - the expectation is that it will be defined later.Note:For more user-friendly behaviour, the example could also setmodel.nameto!requiredto make sure that the error messages are more meaningful.Therefore, eager execution would always result in an error.
What is more, even if the missing value was defined, we still would like to allow the user to overwrite some of the values later.
All this makes the value of the entirefs.exp_folderundefined at the moment of parsing of the yaml file.For more information about what nodes are executed lazily and what they do, please see thesummary of tags.Adding user-defined metadataAwesomeyaml allows its users to assign arbitrary metadata to any of the nodes in the config - this is done by (ab)using yaml tag mechanism.
At the core level, metadata are stored as a postfix to any tag which contains hex-encoded result of pickling python literals - this tag suffix can potentially be added to any other tag.Note:although not directly enforced, official support for metadata is only limited to the cases where the top level element is adict. You can try encoding other literals and there's a chance that everything will be fine (since internally metadata are unused), but we do not guarantee anything. Also, as explained later, our syntax extension only supportsdicttypes.To further explain the process of encoding metadata, consider the following yaml file which extend the previous example with some caching parameters:---fs:root:~/.projectexp_folder:!path[!xreffs.root,!xrefmodel.name]cache:!path[!xreffs.root,cache]cache_size:16GBcaching_policy:lruLet's say, that thefs.caching_policyis an argument expecting one of the following values:['lru', 'fifo', 'filo']and the user would like to embed this information (to later handle validation of the passed value) in a form of the following metadata dict:{'available_options':['lru','fifo','filo']}In order to form a valid yaml document, the python code defining the metadata needs to be encoded by pickling the desired value and encoding the resulting stream of bytes using hexadecimal literals.
For the example above, this goes roughly like:importpicklemetadata={'available_options':['lru','fifo','filo']}encoded=pickle.dumps(metadata).hex()print(encoded)# prints '80037d71005811000000617661696c61626c655f6f7074696f6e7371015d71022858030000006c7275710358040000006669666f7104580400000066696c6f710565732e'After the encoded metadata are obtained, they can be added to any tag as a suffix, separated by:from the main tag.
Since in the example above the target node does not have any tag, we can add a simple!metadatatag which does nothing except serving as a way to add metadata
to an otherwise standard node.
The final form would then look like:fs:root:~/.projectcache:!path[!xreffs.root,cache]cache_size:16GBcaching_policy:!metadata:80037d71005811000000617661696c61626c655f6f7074696f6e7371015d71022858030000006c7275710358040000006669666f7104580400000066696c6f710565732elruAlthough, encoding metadata in this form has the benefit of being compatible with standard yaml, it is obvious that this form is neither easy to write nor read by humans.
Therefore, even though at its core awesomeyaml will use the aforementioned mechanism to store and read metadata, it also provides a human-friendly way of defining metadata by extending yaml syntax a little bit.
Specifically, instead of using!tag:encoded_metadatasyntax, the user can:!tag{{ metadata }}non-standard syntax, wheremetadatais content of the metadata dict in plain python.Using the extended metadata syntax, the example above becomes:fs:root:~/.projectcache:!path[!xreffs.root,cache]cache_size:16GBcaching_policy:!metadata{{'available_options':['lru','fifo','filo']}}lruPython sideBasic usageFollowing the example command lines above, the desired behaviour could be achieved by usingawesomeyaml.Config.buildstatic method, for example:# train.pyimportawesomeyamlasaycfg=ay.Config.build('configs/models/model1.yaml','configs/datasets/dataset2.yaml','configs/optimizers/optim2.yaml')or, in a more generic way:# train.pyimportsysimportawesomeyamlasaycfg=ay.Config.build(*sys.argv[1:])Handling command-line argumentsPlease note that awesomeyaml does not perform any sophisticated command line parsing, so the example above withsys.argvcan easily break it.
Specifically, theawesomeyaml.Config.buildmethod expects all its arguments to be either of:a filenamea file objectyaml stringBecause of that, it's advised to useargparseto extract arguments which are meaningful for awesomeyaml and preprocess the rest independently.However, there is one extra thing awesomeyaml is capable of with regard to command line processing - it can construct valid yaml from one-liners which can be used to overwrite a specific value. Consider the following example: we'd like to trainmodel1withdataset2andoptim2(as in the examples before) but we want to experiment with different number of channels. It is impractical to create a new config file each time we want to change just a single value, so we try to overwrite the default values by appending some yaml to the command line:# try the model with 16 channels, instead of 64pythontrain.pyconfigs/models/model1.yamlconfigs/datasets/dataset2.yamlconfigs/optimizers/optim2.yaml"model: { channels: 16 }"The example above works by first merging content defined in the files and then with the content defined with inlined yaml.
However, specifying a full yaml structure to simply overwrite a single value introduce unnecessary burden and makes it hard to quickly change some values from the command line.
Therefore, the user is free to use a shorted form where the target node is specified with syntax similar to accessing attributes of the evaluated config (seebelow) and its new value is given after=.
For example, the number of channels in our previous example could be overwriten with:model.channels=16Note:It is important to make sure that the entire expression is passed as a single argument. This can be achieved by either not putting spaces around=or using quotes.However, expressions like the one above are not valid yaml and hence cannot be used directly withawesomeyaml.Config.build.
Instead, awesomeyaml provides another functionawesomeyaml.Config.build_from_cmdlinewhich is capable of identifying and transforming those expressions into their full-yaml form (like the one in the first example) suitable to be given to the standardbuildfunction.Seequick referencefor a short example of how handling of the command line arguments could be done.Accessing evaluated fieldsAwesomeyaml implements thebunchpattern for all itsdict-like objects, including the main config object.
Therefore, after the config object is built, its content can be accessed either by using standard indexing operator (like normaldict) or through attribute names.
Following our example from above, the following lines are analogical:obj1=cfg['a']obj2=cfg.aassertobj1isobj2The bunch pattern is applied recursively to all dictionaries so it's possible to chain attributes.However, using the bunch pattern introduces certain problems.
Specifically, because it is possible in Python to shadow methods and other class-level attributes with instance-level attributes, it is fairly easy to introduce undesired conficts between attributes introduced by reading the user's config file and either internal or external API of awesomeyaml's entities.For example,ComposedNodeclass, which is the base class for both lists and dicts within awesomeyaml, provides a common interface for iterating and manipulating sub-nodes of a node. One of the methods introduced by this class is calledchildrenand can be used to simply iterate over sub-nodes.
However, it is quite likely that a key"children"would appear inside the user's config file,
e.g., if the config file describes some tree-like structure.
In that case, any code which would accessobj.childrenwith the intention of calling the awesomeyaml function would instead access the value read from the config file, most likely resulting in an error.To avoid name conflicts and still be able to use the bunch pattern, we decided to introduce two rules:the bunch pattern does not pick up names starting with_, that means that any internal method, attribute etc. can be safely accessed by its names; from the end-user's perspective, that also means that if a name starting with an_appears in the.yamlfile, it can only be accessed by the standard indexing operator, e.g.:cfg['_some_field']all other methods and attributes which are part of the awesomeyaml API are put underaynsnamespace - in order to access them, one needs to go through the proxy calledayns, e.g. to access the previously mentionedchildrenmethod, the relevant call would look like:cfg.ayns.children()Note:it is still possible to shadowaynsname with a value from a config file as there are no checks to prevent this, therefore the user should be careful about (not) using this name - including it in a config file would result in undefined behaviour.Summary of extended tagsThis section includes a brief summary of the new tags introduced by awesomeyaml, each with a short description and a link to more information.
Please note that many tags supports more than one syntax and the behaviour of some can be achieved in different way than using the tag.
In either case, more information can always be found in the provided link to documentation.Note:For the links to work, pleasegenerate html documentationTagDescriptionDoc!append [list]Appends to an existing list on mergeAppendNode!bind:name { args }Binds argumentsargsto a python entity callednameBindNode!call:name { args }Calls a python entity callednamewith argumentsargsCallNode!eval codeEvaluates arbitrary pythoncodeEvalNode!fstr strEvaluates an f-string expressionstrFStrNode!import nameEmbeds a python entity callednameImportNode!include fileIncludes another configfilerecursivelyIncludeNode!path:ref [joins]Computes a path, starting with a reference pointrefand then following a list of folders injoinsPathNode!prev nodeMoves the content ofnodeto a new place on mergePrevNode!requiredSpecifies that a value should be provided laterRequiredNode!xref nodeCross-references another nodeXRefNode!nullEvaluates toNoneN/A!metadata:strEmbeds metadata encoded instrto a nodeMetadata!weakChanges a node's priority to lowMerging!forceChanges a node's priority to highMerging!delReplaces the content of a matching node on mergeMerging!mergeMerges the content of a matching node on mergeMergingContributingAll contributions are welcome, please open a pull request with your changes!If a substantial change is accepted and merged into the codebase the author might be asked to own contributed pieces of code and become responsible for reviewing/maintaining those parts.
Lack of commitment to fulfil this obligation might result in reverting any changes, arbitrary changes of ownership, or any other actions deemed necessary to allow for healthy development of the package.When making your changes, please follow the coding style used throughout the project (PEP-8).Reporting issuesPlease open an issue on GitHub and provide minimal failing example, together with information about the specific version of the package (ideally git commit), Python version, pyyaml version, libyaml version (if used), and OS used.LicenseThe package is released under Apache License 2.0.
See LICENSE file for more information.