Package: replacement
ReplacementReplacement is a python utility that parses a yaml template and outputs text.Installingpip3 install replacementor, if you usenix:nix-env --install -A 'nixpkgs.python3Packages.replacement'IntroductionNOTE: the examples given here are further documented in thetests.Atemplateis a YAML file containing areplacementobject.A replacement object contains a list ofblocks.1. basic templateA template:---# simplistic template example# tests/hello.yamlreplacement:-text:textinput:hello world...Execute a template usingreplacement:$replacement-ttests/hello.yaml
helloworldBlocks always begin with adirective, which specifies:the data type the block willyield, such astextor adictthe input data type, such asfileoryamlIn the block above,text: textis the directive;
it specifies the block takestextinput and yieldstextoutput.2. reading from a fileFilehello.out:hello world---# read from file# tests/file.yamlreplacement:-text:fileinput:hello.out...$replacement-ttests/file.yaml
helloworldNotice that:The directive in the above block istext: file;
this means: "yield text, input from a file".Theinputkey is used for the filename to be read;
in the first exampleinputwas used to store the literal text input.File paths are relative to the path of the template.intermission: schemaA schema of all the replacement directives is contained inschema.yaml.Here is a snippet showing just thedirectiveandinputportions of a block:---### format of a block##block:-yield:inputType# this is a directiveinput:inputVal### definition of schema elements##schema:# 'yield' : what to output (how to format output)yield:-text# text: a list of newline-separated strings-dict# a dictionary of key-value pairs-meta# output nothing: set a metadata (substitution dictionary) variable# 'input' : where to source datainputType:# we can input everything we output (see 'yield' above)-text-dict-meta# a key-value pair retrieved from substitutions dictionary# we can also input from other sources-file# open a file on disk-eval# python3 eval() statement-func# import and then call call a function-exec# subprocess execution (usually bash)inputVal:input_specific...3. metadata and substitutionThemetadirective specifies that the output of a block should be inserted
into the "substitutions dictionary" (aka:meta).this implies input must be a valid dictionaryreplacement will parse YAML or JSON (which is a subset of YAML)Theprockeyword specifies that block output should be processed
(string substitution).
Validprocdirectives are:format::str.format()substitute::string.Template().substitute()safe_substitute::string.Template().safe_substitute()---# metadata# tests/metadata.yamlreplacement:# parse 'input' and insert the resulting dictionary into 'meta'-meta:dict# metadata may be given as an objectinput:version:1.1tag:my_awesome_tag-meta:text# metadata may be given as text which can be parsed as valid YAMLinput:|---message: hello world...-meta:text# metadata may also be given as JSONinput:|{ "hi": 5 }# use 'proc' to specify that 'str.format(**meta)' should be run on output-text:textproc:formatinput:|v{version} tag "{tag}"-text:textproc:substituteinput:|message $message-text:textproc:safe_substituteinput:|hi $hithis value may not exist - $nonexistent...$replacement-ttests/metadata.yaml
v1.1tag"my_awesome_tag"messagehelloworld
hi5thisvaluemaynotexist-$nonexistentMetadata can also be read from a file.Filea.json:{"hello":"world","hi":5,"list":[1,2,3,4]}---# metadata# tests/meta_file.yamlreplacement:# parse file, inserting result into metadata dictionary-meta:fileinput:a.json# string substitution with 'proc'-text:textproc:formatinput:|hello {hello}hi {hi}list {list}...$replacement-ttests/meta_file.yaml
helloworld
hi5list[1,2,3,4]3a. value -> dictionary usingkey---# insert contents of file as metadata# tests/file_as_meta.yamlreplacement:-text:dictinput:# produces {"hello": "hi"}-dict:textkey:helloinput:|hi-dict:filekey:contentsinput:hello.out-meta:filekey:datainput:prep.out-dict:dictprep:substituteinput:also:$data-meta:filekey:largefileinput:recurse.out-dict:dictprep:formatinput:hello:worldhi:|{largefile}...---
hello:hi
contents:helloworld
also:helloworld
hi:|-recursedinlinev1.1tag"my_awesome_tag"messagehelloworldhi5thisvaluemaynotexist-IexistIpromise!
...4. nestingBlocks are executed in sequence, and may nest.nested blocks may extend/altermeta, which will be seen by
later blocks or child blocks, butnotparent blocks.---# nesting and dictionaries# tests/nesting.yamlreplacement:# parse 'input' and insert the resulting dictionary into 'meta'-meta:dictinput:version:1.1# same thing, but parse *text* input, instead of a dictionary-meta:textinput:|# note the '|'---tag: my_awesome_tag...# use 'proc' to specify that 'str.format(**meta)' should be run on output-text:textproc:formatinput:|v{version} tag "{tag}"-text:textinput:# metadata additions/changes seen by later blocks and any children,# but seen outside this list-meta:dictinput:version:1.0-text:textproc:formatinput:|#v{version} tag "{tag}" (version clobbered in inner scope)-text:fileinput:hello.out# contains 'hello world'-text:textproc:formatinput:|outer still v{version}...$replacement-ttests/nesting.yaml
v1.1tag"my_awesome_tag"#v1.0 tag "my_awesome_tag" (version clobbered in inner scope)helloworld
outerstillv1.15. preprocessingSubstitution can also be performed on the values of the block itselfbeforeit is parsed.
The keywordprepis used, with the same semantics asproc(above):---# preprocessing# tests/prep.yamlreplacement:-meta:dictinput:filename:hello.out# preprocessing will substitute {filename} before evaluating 'file' input-text:fileprep:formatinput:|{filename}...$replacement-ttests/prep.yaml
helloworld6. access to pythoneval---# use of eval# tests/an_eval.yamlreplacement:# 'eval' returning a dictionary that can me appended to 'meta'-meta:evalinput:|{"hello": 5 + 1}-text:textprep:formatinput:|hello {hello}# eval returning a scalar value-text:evalprep:formatinput:|{hello}**3...$replacement-ttests/an_eval.yaml
hello62167. pythonexecSometimes it is advantageous to manipulate the runtime environment.In the below example, this is being used to import a module which will
be needed by a subsequency call toeval.---# use of eval with an exec statement to execute an import call# tests/eval_exec.yamlreplacement:-meta:execinput:|global IPv4Networkfrom ipaddress import IPv4Network-meta:evalinput:|{'gateway': str([h for h in IPv4Network('192.168.1.0/24').hosts()][-1])}-text:textprep:formatinput:|my gateway is {gateway}...$replacement-ttests/eval_exec.yaml
mygatewayis192.168.1.254NOTE: pythonexecis very powerful,
pleaseavoidrunningreplacementas a privileged user.8. imports and function executionThefuncinput directive can be used to find and call an external function
(seedemo.pyfor reference):---# function execution# tests/func.yamlreplacement:# run a function returning a dictionary, and merge that into 'meta'-meta:funcargs:existing:{'original':'thesis'}# NOTE that this is sensitive to PYTHONPATHinput:|ret_a_dict ./demo.py-text:textprep:formatinput:|original {original}secret {secret}# run a function returning a dictionary and export return as JSON-text:funcoptions:-json# emit JSON rather than YAML (the default)args:existing:{'original':'thesis'}input:|ret_a_dict ./demo.py# run a function returning an IOStream-text:funcargs:{}input:|ret_a_stream ./demo.py# previous use of non-portable BRAINDEAD python dot.notation# ... will break if 'replacement' (the script) is MOVED vis-a-vis demo.py# (of OF COURSE the CWD is blithely ignored ... why would ANYONE use it LOL)# TLDR: use the '[symbol] [path]' notation just above.# The _original_ call (broken) was:# tests.demo.ret_a_stream# A function returning a list of lines of text# (using the proper, cleaner import strategy)-text:funcargs:an_arg:["question"]input:|ret_a_list ./demo.py# a static function inside a class-text:funcinput:|aClass.invented_list ./demo.py$replacement-ttests/func.yaml
originalthesis
secret42{"secret":42,"original":"thesis"}1.hello2.world42meaning
question
hello
from
staticmethod9. RecursionOr, how I learned to stop worrying and have templates import other templates.---# recursion (nested 'replacement' templates)# tests/recurse.yamlreplacement:# parse a 'replacement' template inline-replacement:textinput:|---replacement:- text: textinput: |recursed inline...-meta:dictinput:nonexistent:"IexistIpromise!"# parse a replacement template from a file# NOTE *relative* path)# NOTE our 'meta' dictionary is propagated to child replacement being parsed-replacement:fileinput:metadata.yaml...$replacement-ttests/recurse.yaml
recursedinline
v1.1tag"my_awesome_tag"messagehelloworld
hi5thisvaluemaynotexist-IexistIpromise!DevelopmentThe recommended development environment is:Pythonvenvwithnix:nix-shell --pure  # activates a venvalternately:python -m venv venv
 venv/bin/activate
 alias pip="python -m pip"Install requirements:pip install -r requirements.txt
 pip install -e .Test and build:pytest
 ./create_package.shNOTESThis documentation is itself built as a template, so that the content
of.yamltests and.outresults is kept in sync with thetestsdirectory.
SeeREADME.template.yamlandgen_readme.sh.Project TODOProject TODO listsubprocess execution and output captureaccept template from STDIN, not just a template filePackaging:proper test runner (perhapstox?)add test runner to all the buildsDependency output command: runs all preprocessing and outputs a list of
file dependencies.
For use e.g. in Makefiles.Expressthe schemaformally and write validation code for it.