npm package: @lpezet/etl-js-cli
Description: Command Line Interface for ETL-JS.
Author: Luke Pezet
Latest version: 3.1.0

# ETL-JS CLI

Extract, Transform, and Load sharable and repeatable from command line.

[![NPM Version][npm-image]][npm-url]
[![Linux Build][travis-image]][travis-url]
[![Windows Build][appveyor-image]][appveyor-url]
[![Test Coverage][coveralls-image]][coveralls-url]
[![Known Vulnerabilities][vulnerabilities-image]][vulnerabilities-url]

```bash
mkdir my-etl && cd my-etl
# Initialize
etl-js init
# Use local executor
sed -i "s/executor: remote1/executor: local1/" settings.yml
# Create a simple template downloading Orion nebula from NASA site
echo -e "etlSets:\n default:\n  - step1\nstep1:\n files:\n  /tmp/orion-nebula.jpg:\n   source: https://www.nasa.gov/sites/default/files/thumbnails/image/orion-nebula-xlarge_web.jpg" > download_orion_nebula.yml
# Run template
etl-js run download_orion_nebula.yml
```

The image is downloaded locally to `/tmp/orion-nebula.jpg`.

# Table of Contents

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

- [Installation](#installation)
- [Features](#features)
- [Concept](#concept)
- [Getting started](#getting-started)
  - [First run](#first-run)
  - [Environment variables](#environment-variables)
  - [Variables](#variables)
- [Examples/Tutorials](#examplestutorials)
- [License](#license)
- [Publishing](#publishing)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

# Installation

```bash
npm install --global @lpezet/etl-js-cli
```

# Features

- Template-based process using YML to express steps and activities as part of ETL
- Built-in modules to leverage already installed software (e.g. mysql, mysqlimport, etc.)
- Dynamic behavior through the use of _tags_ in activities.

# Concept

This command line tool lets you tap into the power of [ETL-JS](https://github.com/lpezet/etl-js).
The idea is to be able to share and easily repeat activities, and leverage existing tools as much as possible.

Steps and activities are basically specified in YML like so:

```yml
etlSets:
  default:
    - activity1
    - activity2
  somethingelse:
    - activity3

activity1:
  commands:
    my_command:
      command: echo "Hello..."

activity2:
  commands:
    something:
      command: echo "World!"

activity3:
  commands:
    bye_bye:
      command: echo "Bye bye!"
```

For more details, have a look at [ETL-JS](https://github.com/lpezet/etl-js).

# Getting started

You can get starting right away and figure things out along the way. If anything is unclear or confusing, it is best to take a look at [ETL-JS](https://github.com/lpezet/etl-js). The sample template only uses the [Commands Mod](https://github.com/lpezet/etl-js/blob/master/Mods.md#commands-mod).

Once `etl-js-cli` has been installed, run the following:

```bash
mkdir etl-js-test
cd etl-js-test
etl-js init
```

At this point, 2 new files have been created in your current directory:

- **etl.yml**: a sample ETL template to show some of ETL JS features
- **settings.yml**: the settings `etl-js-cli` will be loading to run any ETL template.

You can ope `etl.yml` in your favorite editor and see its content.
It should have something similar to this:

```yml
etlSets:
  default: ["hello", "world"]
  envTest: ["envTest"]
  varTest: ["varTest"]

hello:
  commands:
    say_hello:
      command: echo "Hello..."
world:
  commands:
    say_world:
      command: echo "...world!"
envTest:
  commands:
    with_env:
      command: 'echo "The value for env variable ''TESTENV'': {{env.TESTENV}}"'
varTest:
  commands:
    001_create_var:
      command: printf "hello"
      var: TESTVAR
    002_use_var:
      command: 'echo "The value for var TESTVAR: {{vars.TESTVAR}}"'
```

It basically provides 3 different ETL Sets:

- **default**: if no special argument is passed to `etl-js-cli`, this is what it will run by default.
- **envTest**: this ETL process simply demonstrates how environment variables can be used in the template.
- **varTest**: this ETL process demonstrates how variables generated by other steps can be using within the template.

## First run

To execute your first ETL template simply run the following:

```bash
etl-js run etl.yml
```

This will effectively execute the `default` ETL set. Be default, the complete result of all activities execute by this ETL Set is outputed, along with other log messages:

```js
{ exit: false,
  activities:
   [ { activity: 'hello',
       steps:
        { commands:
           { exit: false,
             skip: false,
             results:
              [ { command: 'say_hello',
                  results:
                   { exit: false,
                     pass: true,
                     skip: false,
                     _stdout: 'Hello...\n',
                     _stderr: '',
                     result: 'Hello...\n' },
                  exit: false,
                  skip: false } ] } },
       exit: false,
       skip: false
     },
     { activity: 'world',
       steps:
        { commands:
           { exit: false,
             skip: false,
             results:
              [ { command: 'say_world',
                  results:
                   { exit: false,
                     pass: true,
                     skip: false,
                     _stdout: '...world!\n',
                     _stderr: '',
                     result: '...world!\n' },
                  exit: false,
                  skip: false } ] } },
       exit: false,
       skip: false } ] }
```

This ETL Set consists of two simple commands echo-ing "Hello...world".

## Environment variables

To see how environment variables work, first run the following:

```bash
etl-js run etl.yml envTest
```

By specifying `envTest`, we are asking `etl-js-cli` to only run the ETL Set `envTest`.
The final output should look like the following:

```js
{ exit: false,
  activities:
   [ { activity: 'envTest',
       steps:
        { commands:
           { exit: false,
             skip: false,
             results:
              [ { command: 'with_env',
                  results:
                   { exit: false,
                     pass: true,
                     skip: false,
                     _stdout: 'The value for env variable \'TESTENV\': \n',
                     _stderr: '',
                     result: 'The value for env variable \'TESTENV\': \n' },
                  exit: false,
                  skip: false } ] } },
       exit: false,
       skip: false } ] }
```

You should notice here that the standard output did not resolve the value of the environment variable TESTENV.
This is because when running the previous command, we did not have TESTENV environment variable set.
We can set it using the `env` command in linux like so:

```bash
env TESTENV="Hello world!" etl-js run etl.yml envTest
```

The output should then look like the following:

```js
{ exit: false,
  activities:
   [ { activity: 'envTest',
       steps:
        { commands:
           { exit: false,
             skip: false,
             results:
              [ { command: 'with_env',
                  results:
                   { exit: false,
                     pass: true,
                     skip: false,
                     _stdout: 'The value for env variable \'TESTENV\': Hello world!\n',
                     _stderr: '',
                     result: 'The value for env variable \'TESTENV\': Hello world!\n' },
                  exit: false,
                  skip: false } ] } },
       exit: false,
       skip: false } ] }
```

## Variables

The `varTest` ELT Set in `etl.yml` is as followed:

```yml
varTest:
  commands:
    001_create_var:
      command: printf "hello"
      var: TESTVAR
    002_use_var:
      command: 'echo "The value for var TESTVAR: {{vars.TESTVAR}}"'
```

It basically consists to 2 commands:

- **001_create_var**: This command will echo a piece of text and save (its output) into a variable named `TESTVAR`.
- **002_use_var**: This command will echo another piece of text which includes a _tag_ for the variable `TESTVAR`.

You can run this ETL set and see its output:

```bash
etl-js run etl.yml varTest
```

The final output should look like this:

```js
{ exit: false,
  activities:
   [ { activity: 'varTest',
       steps:
        { commands:
           { exit: false,
             skip: false,
             results:
              [ { command: '001_create_var',
                  results:
                   { exit: false,
                     pass: true,
                     skip: false,
                     _stdout: 'hello',
                     _stderr: '',
                     result: 'hello' },
                  exit: false,
                  skip: false },
                { command: '002_use_var',
                  results:
                   { exit: false,
                     pass: true,
                     skip: false,
                     _stdout: 'The value for var TESTVAR: hello\n',
                     _stderr: '',
                     result: 'The value for var TESTVAR: hello\n' },
                  exit: false,
                  skip: false } ] } },
       exit: false,
       skip: false } ] }
```

You can see how the variable has been resoled using the output of the first command.

**NB**: Something worth noticing here is that the `echo` command always adds a newline at the end of a text by default. Simply calling `echo hello` doesn't display just `hello` but `hello\n`. Here we are using `printf` instead which does not behave like `echo` and does not generate this newline.

# Examples/Tutorials

Examples and tutorials can be found [here](examples/README.md).

# License

[MIT](LICENSE)

[npm-image]: https://badge.fury.io/js/%40lpezet%2Fetl-js-cli.svg
[npm-url]: https://npmjs.com/package/@lpezet/etl-js-cli
[travis-image]: https://travis-ci.org/lpezet/etl-js-cli.svg?branch=master
[travis-url]: https://travis-ci.org/lpezet/etl-js-cli
[coveralls-image]: https://coveralls.io/repos/github/lpezet/etl-js-cli/badge.svg?branch=master
[coveralls-url]: https://coveralls.io/github/lpezet/etl-js-cli?branch=master
[appveyor-image]: https://ci.appveyor.com/api/projects/status/hxkr7yml7qhi9jo8?svg=true
[appveyor-url]: https://ci.appveyor.com/project/lpezet/etl-js-cli
[vulnerabilities-image]: https://snyk.io/test/github/lpezet/etl-js-cli/badge.svg
[vulnerabilities-url]: https://snyk.io/test/github/lpezet/etl-js-cli

# Publishing

To publish next version of `etl-js-cli`, run the following:

```bash
npm version patch
git push --tags origin master
npm run dist
npm publish dist/ --access public
```