npm package: @siteone/gitlab-ci-tools
Description: **NOTE:** **This is a work in progress, please note that it's far from stable version. It's in usable state, but still in development. There are a good chance the configuration and API will change in the future. Also the docs are still work in progress.**
Author: Jiří Cerhan jiri.cerhan@siteone.cz
Latest version: 1.0.7

# Deploy package

**NOTE:** **This is a work in progress, please note that it's far from stable version. It's in usable state, but still in development. There are a good chance the configuration and API will change in the future. Also the docs are still work in progress.**

## Table of contents

// TODO

## Introduction

This package main purpose is to provide basic tools to help you implement gitlab CI deployment process for projects. We use predefined (SiteOne standard) way to do this. If your project requires any changes to this standard process you can override or extend any part of the process.

It's created for deploying javascript frontend projects. In theory it can be used to deploy any project, but configuration is in javascript.

There are three main parts of this package:

1. **CI jobs task runner - Run any scripts in predefined CI stages**
2. **Deploy tools - Provide tools to upload, start, stop and delete your build from server during this stages**
3. **CLI - to help you create your gitlab-ci.yml config and create tags**

No previous knowledge of .gitlab-ci.yml is required, but it really helps to understand the concept. For the deploy part we are using PM2 process manager.

**NOTE:** You will need gitlab project with CI/CD enabled and configured runners (it's beyond scope of this package).

## SiteOne deploy process

SiteOne deploy process consists 2 stages (potentially 3 jobs)

1. Build
2. Deploy
3. Stop (deploy stage but different job (stop action), when `auto_stop_in` is configured)

You can setup unlimited environments, with different configuration. Also there is a special `isMultiEnv` option, which allows deploying multiple instances of the same environment. More on that later.

Deployment is initiated by creating new tag in gitlab matching certain regExp, don't worry there is a CLI command that helps you create thees tags.

## Understanding "CI jobs task runner" part

This part helps with dealing with the CI part of your deployment process. It's just a tool to enable you to run any shell commands during your CI jobs. Now it supports only this predefined three jobs mentioned above. You can run unlimited scripts and processes during these jobs, so it should fit most of your needs.

We use three predefined jobs in two stages

**1. Build**

In this job `build` (default) folder should be created. This build folder is saved as artifact to gitlab and used later in deploy stage. You can also run tests, and other code quality tools in this job, to make sure everything is ok and ready to deploy.

**2. Deploy**

This job should serve to upload previously created build artifact to your server start the build and make sure it's running. Potentially stop and delete old version of app if there is already one deployed.

**3. Stop**

This jobs is configured only for `isMultiEnv=true` environments. It can be triggered manually from gitlab, or you can set `auto_stop_in` option in you environment config. It's purpose is to stop and delete preview (review) of an app that is no longer needed.

**NOTE:** We are considering merging build and deploy job to one, to save time installing dependencies and setting up the runner.

### Process flow chart

```mermaid
graph TD
  subgraph create tag
  tag_created(Tag created) --> environment{environment}
  environment --> |"match"|build([build])
  environment --> |no match|E(end)
  build --> deployManual{deployManual}
  deployManual --> |false| deploy([environnementName_deploy job])
  deployManual --> |true| trigger_deploy([manual deploy job])

  end

  subgraph stop environment
  stop_environment("stop environment triggered <br> (manual in gitlab or autostop)") --> stop_job([environnementName_stop job])
  end

  subgraph trigger deploy
  trigger_deploy(Manually start <br> deploy production job in gitlab) --> manual_deploy([environnementName_deploy job])
  end



  deploy -.-> autostop
  manual_deploy -.-> autostop

  autostop{auto_stop_in}

  autostop --> |false| E
  autostop --> |true| stop_environment

  stop_job --> E

```

### Configuration (js)

#### 1. ci.config.js

Configuration contains 3 main sections corresponding to gitlab CI jobs (`build`, `deploy`, `stop`). Each job is divided to three phases: `before`, `main`, `after` executed in this order. It really doesn't meter. You can execute all the script in main section and the result will be the same. It's just for easier organization of your tasks and it corresponds to gitlab CI job settings.

Each section corresponds to CiScripts type. The config it self is deep partial of the CiConfig. It's later merged with defaults and validated, so you don't need to fill all values.

```typescript
type CiScripts {
  before: string[], // Array of shell commands
  main: string[], // Array of shell commands
  after: string[], // Array of shell commands
}

type CiConfig {
  build: CiScripts,
  deploy: CiScripts,
  stop: CiScripts,
}
```

So for example

```typescript
const config: PartialCiConfig = {
  build: {
    main: [
      "yarn test",
      "yarn build",
      "yarn start",
      "yarn healthcheck:local"
    ],
  },
  deploy: {
    main: ['yarn ci:deploy']
    after: ['yarn notification:deploy:done']
  }
  stop: {
      main: ['yarn ci:stop'],
  }
};
```

**NOTE:** Implementing this npm scripts is developer responsibility. It's pretty much standard supported by all popular javascript frontend frameworks. This package will help you with the `yarn ci:deploy` and `yarn ci:stop`. Also `yarn install` is executed at start of every job in Gitlab runner (this package wont work if it wasn't).

#### 2. setup gitlab ci/cd env vars

Next you will need to setup CI/CD environment variable in gitlab. First create `CI_TOOLS_IMAGE` variable. It's image gitlab runner should run your deploy scripts on. We are recommend alpine version and locking it to concrete version to ensure updates wont break your deploy process. _It has to contain node and yarn installed._ For example:

`node:12.18.3-alpine`

#### 3. Gitlab .ci-config.yml

First you need to create `.ci-config.yml` file in root of your repository (gitlab will recognize it and consume if CI/CI is enabled without any other setup). Than you can include your gitlab-ci config.

```yml
include: ./ci-tools-gl-config.yml
```

This `ci-tools-gl-config.yml` config file is generated with our CLI script, based on your `deploy.config.js`. More on this later.

### Triggering deploy process

To trigger deploy process just create new tag in gitlab. This tag must match certain regex to start the process. It differs by your environment settings:

// WIP
// TODO describe default env tag regexes.

The best way to create your tag is use this package CLI command `createEnv`. More on this in CLI section.

### Stop preview environment

If you configure auto_stop_in option your environment will auto stop in specified time. You can stop them manually in your gitlab repository environments (operations -> environments) by clicking the stop button. You can also prevent this environment from-auto stopping using button in gitlab environments page.

This is basically all you need to setup your deploy process with gitlab CI, at least the CI part.

## Understanding "Deploy tools" part

This part provide some tools to:

1. Create ssh connection to your severs and execute any command on this servers
2. Upload and delete files to and from you servers
3. Start, stop, delete processes from PM2
4. Perform basic health check on just deployed app

You can use all this tools separately, override or extend them.

### How it works

Deploy script is started using CLI command `gitlab-ci-tools deploy` (more about this in CLI section) or you can create your custom script importing `deploy` function from this package. Than add this script to your package json scripts.

There are two possible actions (--action parameter in CLI script on 3rd parameter on deploy function) `default` and `stop`

**Default**
Default action purpose is to copy your build folder to server, stop and remove old build (if present), start new build (in PM2) and make sure it's running. It's divided to 4 steps:

1. Deploy before
2. Deploy upload
3. Deploy start
4. Deploy after

**Stop**

Stop action is meant to stop and remove deployed build from server.

### Deploy function flow chart

```mermaid
graph TD
  cli_deploy[run deploy function] --> setup_dynamic_args["deployConfig.setupDynamicArgs || environmentConfig.setupDynamicArgs || nill"]
  setup_dynamic_args --> action{action}
  action --> |default| deploy_before["environmentConfig.deployBefore || deployConfig.deployBefore || nill"]
  deploy_before --> deploy_upload["environmentConfig.deployUpload || deployConfig.deployUpload"]
  deploy_upload --> deploy_start["environmentConfig.deployStart || deployConfig.deployStart"]
  deploy_start --> deploy_after["environmentConfig.deployAfter || deployConfig.deployAfter || nill"]
  action --> |stop| on_stop["environmentConfig.onStop || deployConfig.onStop"]
```

## Configuration

Work in progress. Meanwhile please check DeployConfig in @siteone/gitlab-ci-tools/dist/configuration/interface

// TODO in depth configuration (ci, preview, env vars)

## CLI

Work in progress. Meanwhile please check `gitlab-ci-tools --help` (node_modules/.bin/gitlab-ci-tools) for more information.

// WIP

## Api documentation

// TODO in depth api docs

## Debugging

If you need more debug information you can set (in gitlab ci/cd settings) `CI_TOOLS_DEBUG` environment variable for more verbose output.

Available levels are (using `loglevel` library):

- TRACE
- DEBUG
- INFO
- WARN
- ERROR
- SILENT

Any other non falsy value will set loglevel to `TRACE`.

## Examples

### Basic example (production deploy only)

We will start with minimum configuration example. Please note that this is simplest possible example, most of this can be changed or overridden.

### Pre requisites

- node/js project
  - `build` command (in package json scripts) that creates `./build` folder and all externalities are bundled in, so your project don't need `node_modules` folder to run.
  - entry point (`build/index.js`) that starts your project's server
- gitlab with enabled ci/ci and configured runners.
- configured server(s) with node, pm2 installed where you wish to deploy your project.

### Installation

`yarn add @siteone/gitlab-ci-tools --dev`

### Configuration

#### package.json

In your package.json add `ci` script:

```json
"scripts": {
    "build": "webpack --mode=production",
    "ci": "gitlab-ci-tools",
    "ci:createEnv": "gitlab-ci-tools createEnv",
    "ci:deploy": "gitlab-ci-tools deploy",
    "ci:stop": "gitlab-ci-tools deploy --action=stop",
    "ci:createCiConfig": "gitlab-ci-tools createCiConfig"
  }
```

#### deploy.config.js

Create file deploy.config.js and add following minimal configuration.

```typescript

const KEY_MASTER = process.env.KEY_MASTER || "UNSET OUTSIDE RUNNER"
module.exports = {
  environments: [
  {
    environmentName: "production",
    environmentTagCreate: createTag,
    environmentTagParseVersion: parseTag,
    environmentTagIsSemver: true,
    environmentTagIsVersioned: true,
    defaultEnvironmentBranch: 'master',
    isMultiEnv: false,
    deployManual: true,
    uploadConfig: {
      files: ["ecosystem.production.config.js"],
      folders: ["build"],
      uploadTo: "www",
    },
    servers: [
      {
        getPort: () => 8080,
        getAppName: () => "my-testapp",
        sshConfig: {
          host: "my.server.com",
          username: "deploy.user",
          privateKey: KEY_MASTER,
        },
      },
    ],
  },
};
```

#### ecosyste.production.config.js

Create pm2 ecosystem config file:

```typescript
// Extract arguments
var appName = process.argv[process.argv.length - 2];
var appPort = process.argv[process.argv.length - 1];

module.exports = {
  apps: [
    {
      name: appName,
      script: "./build/index.js",
      max_restarts: 50,
      instances: 6,
      exec_mode: "cluster",
      log_date_format: "YYYY-MM-DD HH:mm Z",
      source_map_support: true,
      // env related settings
      env: {
        NODE_ENV: "production",
        APP_PORT: appPort,
      },
    },
  ],
};
```

#### ci.config.js

Create your ci.config.js. In this example default values is enough, but file still needs to exist. So just export empty object.

```typescript
module.exports = {};
```

#### gitlab-ci.yml

Use CLI to generate your ci-tools-gl-config.yml.

Run `ci:createCiConfig` to generate your `ci-tools-gl-config.yml` file.

Create (or update if you already have one) gitlab-ci.yml with following content.

```yml
include: ./ci-tools-gl-config.yml
```

#### Environment variables

In your project add environment variables to your gitlab ci/cd settings. In this example we need `KEY_MASTER`, variable containing private key which gitlab runner can use to connect to your server over ssh.

You also need to setup `CI_TOOLS_IMAGE` which is image to create your gitlab CI runner from. Search docker hub to find image that suits your needs. Something like `lts-alpine` would be a good choice. It should be close to the server your project will run on.

## Development

This package is using microbundle to create production build.

`yarn dev` starts `microbundle watch` - complie files when they change
`yarn build` runs `microbundle` - create build (dist) from source

More in microbundle package documentation