npm package: @stackstorm/st2flow-yaml
Description: ```bash npm i @stackstorm/st2flow-yaml -S ``` > **Critical Terminology:** >  > - **AST:** **A**bstract **S**yntax **T**ree - a tree of tokens generated by a parser > - **POJO:** **P**lain **O**l' **J**avaScript **O**bject - the name says it all
Latest version: 1.0.0-pre.5

# @stackstorm/st2flow-yaml

```bash
npm i @stackstorm/st2flow-yaml -S
```
> **Critical Terminology:**
> 
> - **AST:** **A**bstract **S**yntax **T**ree - a tree of tokens generated by a parser
> - **POJO:** **P**lain **O**l' **J**avaScript **O**bject - the name says it all


This is a custom YAML parser, tokenizer, serializer, and AST manipulation utility. This module is comprised of the following sub-modules, which are described in further detail below:

- **token-set.js** - YAML parser and serializer, a thin wrapper around the popular [yaml-ast-parser][1].
- **crawler.js** - utility for reading and writing to the AST. This is the primary liaison between the AST and business code and is built for consumption.
- **(private) objectifier.js** - internal utility for converting the AST to an easy-to-consume POJO.
- **(private) token-factory.js** - internal utility for creating AST tokens from raw JavaScript data.
- **(private) token-refinery.js** - internal utility for reindexing the AST whenever mutations are made.

> **NOTE:** It will help to look at the actual code while you read along. Not all methods are documented - only the one's you should care about.

## token-set.js

Given a YAML string, parses the string into an AST. This works by first parsing the string using the [yaml-ast-parser][1]. However, the yaml-ast-parser discards whitespace, comments, colons, and other special characters. To fix this, we recurse over the token tree and parse the _"space in between"_ tokens. To illustrate this, consider the following yaml:

```yaml
version: 1.0

# comment
description: >
  This is a description
```

Parsing the above YAML results in the following AST (trimmed for brevity):

```
{
  kind: 2,
  mappings: [{
  	kind: 1,
  	key: { startPosition: 0, endPosition: 7, kind: 0, value: 'version' },
  	value: { startPosition: 9, endPosition: 11, kind: 0, value: '1.0', valueObject: 1 }
  }, {
  	kind: 1,
  	key: { startPosition: 24, endPosition: 35, kind: 0, value: 'description' },
  	value: { startPosition: 41, endPosition: 62, kind: 0, value: 'This is a description' }
  }]
}
```

As you can see, there is a "gap" between a token's `endPosition` and the next token's `startPosition`. We manually parse this gap, assigning each token a "prefix": _**an array of tokens representing the "gap" before the respective token**_. This prefix is crucial for maintaining source identity so that the following is always true:

```js
tokenSet.toYAML( tokenset.fromYAML(yaml_string) ) === yaml_string
```

### TokenSet API and Usage

```js
import { TokenSet } from '@stackstorm/st2flow-yaml';

const tokenSet = new TokenSet(yaml: string);

// parses the yaml string into an AST, replacing the previous AST
// this method is called during construction
tokenSet.fromYAML(yaml: string);

// converts the AST into a YAML string (this is where the prefix is crucial)
tokenSet.toYAML();

// converts the AST into a plain (easy-to-consume) JavaScript object
tokenSet.toObject();

// reindexes the AST after mutations are made
// consumers are responsible for calling this method after mutating the AST
tokenSet.refineTree();
```


## crawler.js

This is a utility for reading and manipulating an AST. Consumers should never need to know how to read or mutate an AST directly. Instead, this module creates a "bridge" between the AST and business code by allowing consumers to interact with an AST as though it was a POJO. To illustrate this, consider the following YAML:

```yaml
version: 1.0,
tasks:
  task1: 
  	 action: core.local cmd="echo 'Task 1'"
  	 next:
  	 	- do: task2
  	 	  when: <% succeeded() %>
  task2:
    action: core.local cmd="echo 'Task 2'"
```

The above YAML is represented as the following POJO, which is much easier to work with than the AST:

```js
{
  version: 1,
  tasks: {
    task1: {
      action: 'core.local cmd="echo \'Task 1\'"',
      next: [
        { do: 'task2', when: '<% succeeded() %>' }
      ]
    },
    task2: {
      action: 'core.local cmd="echo \'Task 2\'"'
    }
  }
}
```

Whenever a mutation should be made to the AST, consumers can _describe_ the mutation as if they were modifying the above POJO. This is explained in the next section.

### crawler API and Usage

The `crawler` is a static objec for reading and mutating an AST. Each method expects an instance of `TokenSet` as the first parameter and a string `path` as the second parameter. The path uses `deep.dot.syntax` (or `['deep', 'dot', 'syntax']`) to reference a key within a POJO. Continuing with the YAML/POJO example from above:


```js
import { crawler } from '@stackstorm/st2flow-yaml';

crawler.getValueByKey(tokenSet, 'tasks.task2');
// -> { action: 'core.local cmd="echo \'Task 2\'"' }

// Array items should also use dot syntax, not bracket syntax
crawler.getValueByKey(tokenSet, 'tasks.task1.next.0.do');
// -> 'task2'

// Replace the value of a token - the value can be any data type
crawler.replaceTokenValue(tokenSet, 'tasks.task1.action', 'aws.lambda');

// Add a new key/value pair to an existing object - the value can be any data type
crawler.addMappingItem(tokenSet, 'tasks.task3', { action: 'core.local' });

// Remove a key/value pair from an object
crawler.deleteMappingItem(tokenSet, 'tasks.task3');

// Modify a collection using a technique analogous to `Array.prototype.splice`
crawler.spliceCollection(tokenSet, 'tasks.task1.next', 1, 0, { do: 'task3' });
```

## objectifier.js (private)

> This module is intended for internal use only and is not directly exposed to the outside world.

Given a token, recursively converts the token to a POJO for easy consumption. This is used in the `TokenSet#toObject` method to convert the AST into a plain object (see the `crawler#getValueByKey` method for more details). Most of the time you want to pass the root tree for a tokenSet:

### Objectifier API and usage

```js
import Objectifier from './objectifier';

const objectifier = new Objectifier(tokenSet.anchors);

objectifier.getTokenValue(tokenSet.tree);
// -> { version: 1, description: 'This is a description', ...}
```


## token-factory.js (private)

> This module is intended for internal use only and is not directly exposed to the outside world.

Given plain JavaScript data, creates the appropriate tokens for manipulating the AST. You can pass in any data type and expect the appropriate token in return. This is primarily used by the crawler for mutating the AST.

```js
import factory from './token-factory';

factory.createToken('string');
// -> { kind: 0, value: 'string' }

factory.createToken(1234);
// -> { kind: 0, value: '1234', valueObject: 1234 }

factory.createToken(true);
// -> { kind: 0, value: 'true', valueObject: true }

factory.createToken({ an: 'object' });
// -> { kind: 2, mappings: [{ 
          kind: 1, 
          key: { kind: 0, value: 'an' },
          value: { kind: 0, value: 'object' }, 
        }]
      }

factory.createToken([ 'an', 'array' ]);
// -> { kind: 3, items: [
          { kind: 0, value: 'an' },
          { kind: 0, value: 'array' }, 
        ]
      }
```

## token-refinery.js (private)

> This module is intended for internal use only and is not directly exposed to the outside world.

Given the raw yaml data from the previous parsing, recursively "refines" all tokens in the tree, updating the `startPosition`, `endPosition`, `jpath`, and prefixes for all tokens. This utility should be used any time the AST is mutated and must be called manually. This is primarily used by the `TokenSet#refineTree` method to refine the entire AST on demand.

### Recommended (indirect) use:

Call `TokenSet#refineTree` method (with no parameters) any time the AST is mutated:

```js
import { crawler } from '@stackstorm/st2flow-yaml';

crawler.addMappingItem(tokenSet, 'tasks.task1', { action: 'core.local' });
tokenSet.refineTree();
```

### Direct use:

The `Refinery#refineTree` method returns an object with the new `tree` and `tail` for a tokenSet instance. The TokenSet instance must be updated with the returned data.

> **Note:** you should never have to do the following. It is only here for the sake of documentation. 99.99% of the time you will use the recommended technique above.

```
import { crawler } from '@stackstorm/st2flow-yaml';
import Refinery from './refinery';

crawler.addMappingItem(tokenSet, 'tasks.task1', { action: 'core.local' });
const refinery = new Refinery(tokenSet.yaml, tokenSet.head, tokenSet.tail);
const { tree, tail } = refinery.refineTree(tokenSet.tree);

tokenSet.tree = tree;
tokenSet.tail = tail;
```

[1]: https://www.npmjs.com/package/yaml-ast-parser