Package: ynot
y!==Language for processing structured data from sources that can providedata in ``yaml`` or ``json`` format.Why not?========That's the way ***y!*** is pronounced.And that's the question I asked myself when I had the the idea toimplement a data processing language quite different from the ones Iknow so far:**y**.aml-based **no**-XML **t**.ransformation***y!*** is the answer to the question ***"why not?"***~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~So what does ***y!*** focus on?-  ***y!*** is an incredibly simple language for processing structureddata (``json``, ``yaml``, ...).-  Therfore it is also perfectly suited for processing output fromvarious NOSQL databases! And with little effort even fromrelational databases.-  ***y!*** focuses on quickly and easily producing output.This output can be:-  Text-  Structured data-  ***y!*** represents the power of ``yaml``-  ***y!*** represents the power of ``jsonpath``-  ***y!*** adopts the power of xslt (and more) without adopting itscomplexity-  ***y!*** supports self-verification of programs by simply providingsamples for input and output.-  No need for using testing-frameworks, writing unit-tests or anyother hassle.-  ***y!*** supports producing well-formatted documentation without anytools-magic.-  It doesn't require any more than a command-line flag.***y!*** Installation=====================You need python 2.7. Probably any python >= 2.7 will be supported butright now it wasn't tested at all for any version but 2.7| To install ***y!*** simply enter| ``pip install --upgrade ynot``| on your command line and you should be able to use it.You can check if the installation succeeded:::$ ynot --versionynot 0.2.2The output may look different dependent on where your installerinstalled it.Now try::$ ynot -hUsage: ynot [OPTION] -t trafoFile document...Apply transformation to yaml documentsOptions:--version             show program's version number and exit-h, --help            show this help message and exit-t TRAFOFILENAME, --trafo=TRAFOFILENAMEFile transformation is read from-l LOGLEVEL, --log-level=LOGLEVELLog level. Choices: [u'DEBUG', u'INFO', u'WARN',u'ERROR']; Defaults to INFO--verify              Verify transformator file TRANSFORMATOR--dry-run             Only validate and verify. No document processing--encoding=ENCODING   endoding of input files. Default: [utf-8]Again the output may look slightly different on your system.Quick Start===========Unfortunately tradition requires us to start with a *hello world*application:Hello World-----------``$ cat hello_world.ynot``.. code:: yamlactions:- print: Hello World::$ ynot -t hello_world.ynotHello World| Not very interesting, right?| Not useful at all, right?But quite simple, right?Process input data------------------Now let's do something a bit more useful. Let's process data - that'swhat ***y!*** is made for:Let's say we have an input that represents multiple text documents withsections and chapters:``$ cat sample_simple.yaml``.. code:: yaml- title: Some document titlesections:- title: Some section titlechapters:- title: Some chapter titletext: |Some long textwith lots of paragraphs- title: Some other chapter titletext: |Some long textwith lots of paragraphs- title: Some other section titlechapters:- title: Some chapter title for some other sectiontext: |Some long textwith lots of paragraphs- title: Some other document title| ... just regular ``yaml`` - nothing magic.| Now we want to print all the titles and nothing else.For the given input file we expect the following output:::Some document titleSome section titleSome chapter titleSome other chapter titleSome other section titleSome chapter title for some other sectionSome other document titleThe program for achieving that looks as simple as this:``$ cat sample_simple.ynot``.. code:: yamlactions:- for:path: '..title'actions:- print: '@y!{.@}'You already may have noticed that ***y!*** programs are ``yaml`` files.Following a particular schema that we'll see later on.The ``for`` action introduces an iterator. It iterates over all nodesaddressed by ``path`` (``jsonpath`` expression) and performs the defined``actions`` on them.One of the possible actions is ``print`` as we already saw in the helloworld program.Here we see that we are not limited to printing static text, but we canrefer to any node of the document by using the special template syntax``@y!{whatever}``, where ``whatever`` again is nothing else but a``jsonpath`` expression.``jsonpath`` expressions are evaluated relative to the path of the nodeaddressed by parent actions like ``for``, unless they start with ``$`` -then they are absolute ``jsonpath`` expressions, starting at thedocument's root.Now let's try it:::$ ynot -t sample_simple.ynot sample01.yamlSome document titleSome section titleSome chapter titleSome other chapter titleSome other section titleSome chapter title for some other sectionSome other document titleLooks good so far.But while developing and testing the program we don't always want tomanually check if the output is correct, do we?With verification~~~~~~~~~~~~~~~~~| ***y!*** has a very simple and straightforward solution.| You can add samples to the program:``$ cat sample_verficication_succeeding.ynot``.. code:: yamlactions:- for:path: '..title'actions:- print: '@y!{.@}'samples:sample1:input:- title: Some document titlesections:- title: Some section titlechapters:- title: Some chapter titletext: |Some long textwith lots of paragraphs- title: Some other chapter titletext: |Some long textwith lots of paragraphs- title: Some other section titlechapters:- title: Some chapter title for some other sectiontext: |Some long textwith lots of paragraphs- title: Some other document titleoutput: |Some document titleSome section titleSome chapter titleSome other chapter titleSome other section titleSome chapter title for some other sectionSome other document title... and simply verify the program against expected output for giveninput by just invoking it without input files or with the ``--dry-run``option:``$ ynot -t sample_verification_succeeding.ynot --dry-run``*Oh! No output!*| That's intended. When everything is right it doesn't output anything.| Let's prove that in case of problems they are reported.With failing verification~~~~~~~~~~~~~~~~~~~~~~~~~So we change the program slightly by prepending *``title:``* to theactual title:``$ cat sample_verification_failing.ynot``.. code:: yamlactions:- for:path: '..title'actions:- print: 'title: @y!{.@}'samples:sample1:input:- title: Some document titlesections:- title: Some section titlechapters:- title: Some chapter titletext: |Some long textwith lots of paragraphs- title: Some other chapter titletext: |Some long textwith lots of paragraphs- title: Some other section titlechapters:- title: Some chapter title for some other sectiontext: |Some long textwith lots of paragraphs- title: Some other document titleoutput: |Some document titleSome section titleSome chapter titleSome other chapter titleSome other section titleSome chapter title for some other sectionSome other document titleNow we can see that the actual output doesn't match the expected one:::ynot -t samples/trafos/sample_verification_failing.yaml --dry-runERROR:ynot.globals:Verifying sample sample1 failedExpected:Some document titleSome section titleSome chapter titleSome other chapter titleSome other section titleSome chapter title for some other sectionSome other document titleGot:title: Some document titletitle: Some section titletitle: Some chapter titletitle: Some other chapter titletitle: Some other section titletitle: Some chapter title for some other sectiontitle: Some other document titleERROR:ynot.globals:Verifying sample sample1 failed for trafo <undefined>Nice, isn't it?You can add as many samples as you want - all of them will be processedand verified.Actions=======``print``^^^^^^^^^We already saw this action in action.``write``^^^^^^^^^Same as ``print`` but without trailing newline.``log``^^^^^^^Allows writing logging information (currently on INFO level - probablythis will be made configurable).What can be logged is intentionally limited to some attributes of thecurrent node:-  path-  pathstack-  node-  documentYou can refer to these context attributes using python's stringformatting capabilities (see `Python 2.7.14documentation <https://docs.python.org/2/library/stdtypes.html#str.format>`__):.. code:: yamlactions:- log: 'current path: {path}, current node value: {node}'Since log messages are written to stderr the output verification is notaffected by adding log actions.``call``^^^^^^^^There is a simple concept or ``routines`` that can be defined on toplevel of the transformator file.All routines defined there can be called from ``actions`` as well asfrom ``routines``.Details are explained in the routines section below.Routines========-  Routines are defined on top level of the ``ynot`` yaml-file.-  Any ``routines`` key can be used as a parameter for the ``call``action.-  A ``routines`` key maps a list of actions.-  ***y!*** will also support parameters for routines, but that's notyet implemented.-  These actions can call routines recursively. The following sampledemonstrates this.``$ cat sample_routines.ynot``.. code:: yamlactions:- for:path: '$'actions:- log: 'path: "%(path)s"'- print: Depth First- print: ===========- call: print_listroutines:print_list:- log: "print_list(path='%(path)s')"- for:path: '[*]'actions:- call: print_map- call: print_titleprint_map:- log: "print_map(path='%(path)s')"- for:path: '.*'actions:- call: print_listprint_title:- log: "print_title(path='%(path)s')"- for:path: '.title'actions:- print: '@y!{.@}'samples:sample1:input:- title: Some document titlesections:- title: Some section titlechapters:- title: Some chapter titletext: |Some long textwith lots of paragraphs- title: Some other chapter titletext: |Some long textwith lots of paragraphs- title: Some other section titlechapters:- title: Some chapter title for some other sectiontext: |Some long textwith lots of paragraphs- title: Some other document titleoutput: |Depth First===========Some chapter titleSome other chapter titleSome section titleSome chapter title for some other sectionSome other section titleSome document titleSome other document titleDocumenting-----------Generating gfm markdown from ``.ynot`` transformators out of the box bya simple command line option.Just try it with the routines sample:``ynot routines.ynot --doc > routines.md````cat routines.md``When you view the output in any markdown viewer that's able to presentgfm markdown you'll see this result:--------------.. routinesynot:routines.ynot.. routines-1:Routines========-  `Samples <#samples>`__-  `Sample document 1 <#sample01>`__-  `Sample document 2 <#sample02>`__-  `routines.ynot <#transformator>`__Transformator that shows how routines can be used-------------------------------------------------This transformator lists all titles in *depth-first* mode.Routines can be invoced recursively - that does the trick.Samples-------sample01Sample document 1~~~~~~~~~~~~~~~~~This document contains two documents:-  the first one has two sections,-  the first section has two chapters-  the second section has only one chapters-  the second one has nothing.Input^^^^^.. code:: yamlinput:- title: Document 1sections:- title: Section 1.1chapters:- title: Chapter 1.1.1- title: Chatper 1.1.2- title: Section 2chapters:- title: Chapter 1.2.1- title: Chatper 1.2.2- title: Document 2sections:- title: Section 2.1chapters:- title: Chapter 2.1.1- title: Chapter 2.1.2- title: Section 2.2chapters:- title: Chapter 2.2.1- title: Chapter 2.2.2Output^^^^^^::Depth First===========Chapter 1.1.1Chatper 1.1.2Section 1.1Chapter 1.2.1Chatper 1.2.2Section 2Document 1Chapter 2.1.1Chapter 2.1.2Section 2.1Chapter 2.2.1Chapter 2.2.2Section 2.2Document 2sample02Sample document 2~~~~~~~~~~~~~~~~~This document contains two documents:-  the first one has two sections,-  the first section has two chapters-  the second section has only one chapters-  the second one has nothing... input-1:Input^^^^^.. code:: yamlinput:- title: Some document titlesections:- title: Some section titlechapters:- title: Some chapter titletext: 'Some long textwith lots of paragraphs'- title: Some other chapter titletext: 'Some long textwith lots of paragraphs'- title: Some other section titlechapters:- title: Some chapter title for some other sectiontext: 'Some long textwith lots of paragraphs'- title: Some other document title.. output-1:Output^^^^^^::Depth First===========Some chapter titleSome other chapter titleSome section titleSome chapter title for some other sectionSome other section titleSome document titleSome other document titleTransformator-------------.. code:: yamlid: routines.ynotname: Routinestitle: Transformator that shows how routines can be useddescription: 'This transformator lists all titlesin _depth-first_ mode.Routines can be invoced recursively - that does the trick.'actions:- log: 'Root path: {path}"'- for:path: $actions:- print: Depth First- print: ===========- call: print_listroutines:print_list:- log: print_list(path='{path}')- for:path: '[*]'actions:- call: print_map- call: print_titleprint_map:- log: print_map(path='{path}')- for:path: '*'actions:- call: print_listprint_title:- log: print_title(path='{path}')- for:path: titleactions:- print: nullsamples:sample1:id: sample01title: Sample document 1description: "This document contains two documents:\n* the first one has two sections,\n  * the first section has two chapters\n\\  * the second section has only one chapters\n* the second one has nothing.\n"input:- title: Document 1sections:- title: Section 1.1chapters:- title: Chapter 1.1.1- title: Chatper 1.1.2- title: Section 2chapters:- title: Chapter 1.2.1- title: Chatper 1.2.2- title: Document 2sections:- title: Section 2.1chapters:- title: Chapter 2.1.1- title: Chapter 2.1.2- title: Section 2.2chapters:- title: Chapter 2.2.1- title: Chapter 2.2.2output: 'Depth First===========Chapter 1.1.1Chatper 1.1.2Section 1.1Chapter 1.2.1Chatper 1.2.2Section 2Document 1Chapter 2.1.1Chapter 2.1.2Section 2.1Chapter 2.2.1Chapter 2.2.2Section 2.2Document 2'sample2:id: sample02title: Sample document 2description: "This document contains two documents:\n* the first one has two sections,\n  * the first section has two chapters\n\\  * the second section has only one chapters\n* the second one has nothing.\n"input:- title: Some document titlesections:- title: Some section titlechapters:- title: Some chapter titletext: 'Some long textwith lots of paragraphs'- title: Some other chapter titletext: 'Some long textwith lots of paragraphs'- title: Some other section titlechapters:- title: Some chapter title for some other sectiontext: 'Some long textwith lots of paragraphs'- title: Some other document titleoutput: 'Depth First===========Some chapter titleSome other chapter titleSome section titleSome chapter title for some other sectionSome other section titleSome document titleSome other document title'--------------Currently under implementation==============================Variables support-----------------Variables can be set during execution. They are saved in the currentnode's context.When accessing a variable, variables from all parent nodes' contexts arevisible as well.Variables can be accessed programmatically or as part ofvalue-templates.-  Value template to be substituted by a variable value: ``${...}``-  Path-match template to be substituted by a single match result:``@y!{...}``-  Path-multimatch template to be substituted by thestring-representation of multiple matches: ``@y!*{...}``.. code:: yamlactions:- set:key: myMagicNumbervalue: 42- print: '${myMagicNumber}'Call parameter support----------------------.. code:: yamlactions:- call:routine: some_routineparameters:paramX: some valueparamY: some valueNot yet implemented===================... nor verified ...Sorting-------Maybe sufficiently supported out of the box by jsonpath_ng extensions.To be verified ...Alternatively something like this might be implemented.. code:: yamlactions:- for:path: '$.some.path'sorting:order: descendingcriteria:- '.some_field'- '.some_other_field'actions:- print: whateverGrouping--------.. code:: yamlactions:- for:path: '$.some.path'grouping:criteria:- '.some_field'- '.some_other_field'having:- 'whatever'- 'whatever'actions:- print: whateverConditional processing----------------------.. code:: yamlactions:- if:- and:- value1: some valuecomparator: equalsvalue2: some other value- value1: some valuecomparator: equalsvalue2: some other value- or:- value1: some valuecomparator: equalsvalue2: some other valuethen:- print: matched if branchelse:- print: matched else branchTransormator/Document processing order--------------------------------------... for multi-document transformator yamlsAllow command line switch like ``--processing-order=templates-first`` or``--processing-order=documents-first``.. command-line-option-for-defining-template-patterns:Command line option for defining template patterns.---------------------------------------------------Something like ...``template-pattern='{separator: "@", idpattern: "[{}+]"}'``Connecting to data sources--------------------------like *Elasticsearch*, *Redis*, *Cassandra*, *MySQL*, *Kafka*, ... ......