pipeline.libsonnetを使ってSpinnakerのpipelineを作る

これは，NTTコミュニケーションズ Advent Calendar 2018 14日目の記事です

## はじめに

Cloud Native Days Tokyo 2018 で[発表](https://www.slideshare.net/td-nttcom/spinnakerkayenta)もさせていただいたのですが、会社で[Spinnaker](https://www.spinnaker.io/)を試しているチームのお手伝いをしております。そこの要望としてPipelineをWebのGUIからではなく、コードとして扱いgitなどでバージョン管理をしたいという話があったので、Pipeline-templateを試してみることにしました。


つまり今回の記事はSpinnakerのpipeline-templateで困っていると言うニッチな方に向けた記事となります。


Spinnakerは簡単に言うと「オープンソースのマルチクラウド対応な継続的デリバリのためのプラットフォーム」というものなのですが細かい説明は今回は割愛させていただきます。Spinnakerが気になる方は上記の公式ページのリンクや[Mercariさんのブログ](https://tech.mercari.com/entry/2017/08/21/092743)などが非常に参考になると思います。

## Spinnakerのpipline-template

実はあんまりpipelineをコードベースで管理する方法の情報はなく、 [githubのレポジトリ](https://github.com/spinnaker/dcd-spec)を見に行くと[roerというツールとyamlを使ってpipeline-templateを扱う方法](https://github.com/spinnaker/dcd-spec/blob/master/ALPHA-GETTING-STARTED.md)が紹介されています。

しかしながら、[roer のレポジトリ](https://github.com/spinnaker/roer)に行くと、roerを使ったpipeline-templateのやり方は古く今後更新されなさそうです。
> Note: A new, better CLI is on the way. This tool is in maintenance mode.

そこで次にspinを使ったpipeline管理を調べることになりました。

## ゴールは？そしてsponnetとは？

ゴールは sponnet + spin を使ってpipelineが作れるようになることです。

sponnetはPipelineやApplicationをspinやAPIからJSONを使ってSpinnakerに登録する際に、SpinnakerのPipelineやApplicationのフォーマットに従ったJSONを出力するために、JSONテンプレート言語であるJsonnetで書かれたpipelineやApplicationのテンプレートと言うのが私の理解です。

Jsonnetについては以下の記事が参考になります

- [Jsonnetの薦め](https://qiita.com/yugui/items/f4a5e0e9dbd8ddffa48e)
- [Kubernetesマニフェスト記述言語としてのJsonnet](https://qiita.com/ktateish/items/701a61e2be44ad012596)

## そんなわけでsponnetでpipelineの作成をやってみた

[sponnetのdemo](https://github.com/spinnaker/spinnaker/tree/master/sponnet/demo)を参考にやってみた。

### Spinのインストール

Spinnakerに[インストールドキュメント](https://www.spinnaker.io/guides/spin/cli/)があるのでそれに従います。

```
$ curl -LO https://storage.googleapis.com/spinnaker-artifacts/spin/$(curl -s https://storage.googleapis.com/spinnaker-artifacts/spin/latest)/linux/amd64/spin
$ chmod +x spin
$ sudo mv spin /usr/local/bin/spin
```

### libsonnetの取得

demoのページでは下記のように書かれているが最新版のデモは動かないので注意

```
$ curl -LO https://storage.googleapis.com/spinnaker-artifacts/sponnet/$(curl -s https://storage.googleapis.com/spinnaker-artifacts/sponnet/latest)/sponnet.tar.gz
$ tar -xzvf sponnet.tar.gz && rm sponnet.tar.gz
```

最新版を動かすときはmasterから取得

```
$ git clone https://github.com/spinnaker/spinnaker.git
$ cd spinnaker/sponnet/demo
```

### jsonnetの取得と設定

```shell
# this is the linux link -- substitute with darwin for macos
# MacOS の場合 下の `linux` を `darwin` に変更
$ curl -LO https://storage.googleapis.com/jsonnet/$(curl -s https://storage.googleapis.com/jsonnet/latest)/linux/amd64/jsonnet
$ chmod +x jsonnet
$ sudo mv jsonnet /usr/local/bin/jsonnet
```

### Applicationの作成

`demo-app.jsonnet` をjsonnetを使うと下記のようなJSONが出力されるのでこれをSpinを使ってSpinnaker上にApplicationを作成します。

```
$ jsonnet demo-app.jsonnet
{
   "attributes": {
      "description": "Demo sponnet application",
      "email": "youremail@example.com",
      "user": "youremail@example.com"
   },
   "cloudProviders": [
      "kubernetes"
   ],
   "email": "youremail@example.com",
   "name": "myapp"
}
```


```
$ jsonnet demo-app.jsonnet > demo-app.json && \
  spin application save --file demo-app.json
```

なお、私が最初に試したときはこのApplicationを作成するという手順がなく、なくてもPipelineの作成はでき、spinからPipelineの情報は取れるがGUIからは見つけられないという状態だったので、ちょっとずつ改善されてます・・・

### pipelineの作成

次に `demo-pipeline.jsonnet` を使ってJSONを出力します。
`demo-pipeline.jsonnet` には以下のようなPipelineの情報が含まれています。

- ExpectedArtifacts
    - docker/image
    - gitlab/file
- Stage
    - manualJudgment
    - checkPreconditions
    - wait
    - deployManifest x 3
    - findArtifactsFromResource
    - jenkins
- Notification
    - pipelineが始まった時にメールに通知
- Trigger
    - docker
    - git

jsonnetで`demo-pipeline.jsonnet`からJSONを生成すると下記のようなJSONが出力されるので、これをspinからSpinnakerに投入することで下図のようなPipelineが作成されます。

```
 jsonnet demo-pipeline.jsonnet
{
   "application": "myapp",
   "expectedArtifacts": [
      {
         "defaultArtifact": {
            "reference": "index.docker.io/yourorg/app",
            "type": "docker/image"
         },
         "id": "docker-name",
         "matchArtifact": {
            "name": "docker-name",
            "type": "docker/image"
         },
         "useDefaultArtifact": true,
         "usePriorArtifact": false
      },
      {
         "defaultArtifact": {
            "reference": "https://gitlab.com/api/v4/projects/your-org%2Fyour-project/repository/files/app%2Fmanifest%2Eyaml/raw",
            "type": "gitlab/file",
            "version": "master"
         },
         "id": "app/manifest.yaml",
         "matchArtifact": {
            "name": "app/manifest.yaml",
            "type": "gitlab/file"
         },
         "useDefaultArtifact": true,
         "usePriorArtifact": false
      }
   ],
   "id": "sponnet-demo-pipeline",
   "keepWaitingPipelines": false,
   "limitConcurrent": true,
   "name": "Demo pipeline",
   "notifications": [
      {
         "address": "someone@example.com",
         "cc": "test@example.com",
         "level": "pipeline",
         "type": "email",
         "when": [
            "pipeline.starting"
         ]
      }
   ],
   "stages": [
      {
         "instructions": "Do you want to go ahead?",
         "judgmentInputs": [
            {
               "value": "yes"
            },
            {
               "value": "no"
            }
         ],
         "name": "Manual Judgment",
         "refId": "Manual Judgment",
         "requisiteStageRefIds": [ ],
         "type": "manualJudgment"
      },
      {
         "name": "Confirm Judgment",
         "preconditions": [
            {
               "context": {
                  "expression": "${ #judgment('Manual Judgment') == 'yes' }"
               },
               "failPipeline": true,
               "type": "expression"
            }
         ],
         "refId": "Confirm Judgment",
         "requisiteStageRefIds": [
            "Manual Judgment"
         ],
         "type": "checkPreconditions"
      },
      {
         "name": "Wait",
         "refId": "Wait",
         "requisiteStageRefIds": [
            "Confirm Judgment"
         ],
         "skipWaitText": "Custom wait message",
         "type": "wait",
         "waitTime": 30
      },
      {
         "account": "staging-demo",
         "cloudProvider": "kubernetes",
         "manifests": [
            {
               "apiVersion": "apps/v1",
               "kind": "Deployment",
               "metadata": {
                  "labels": {
                     "app": "nginx"
                  },
                  "name": "nginx-deployment"
               },
               "spec": {
                  "replicas": 3,
                  "selector": {
                     "matchLabels": {
                        "app": "nginx"
                     }
                  },
                  "template": {
                     "metadata": {
                        "labels": {
                           "app": "nginx"
                        }
                     },
                     "spec": {
                        "containers": [
                           {
                              "image": "nginx:1.7.9",
                              "name": "nginx",
                              "ports": [
                                 {
                                    "containerPort": 80
                                 }
                              ]
                           }
                        ]
                     }
                  }
               }
            }
         ],
         "moniker": {
            "app": "myapp",
            "cluster": "some-cluster",
            "detail": "someDetail",
            "stack": "someStack"
         },
         "name": "Deploy a manifest",
         "refId": "Deploy a manifest",
         "requisiteStageRefIds": [
            "Wait"
         ],
         "source": "text",
         "type": "deployManifest"
      },
      {
         "account": "staging-demo",
         "cloudProvider": "kubernetes",
         "manifests": [
            {
               "apiVersion": "apps/v1",
               "kind": "Deployment",
               "metadata": {
                  "labels": {
                     "app": "nginx"
                  },
                  "name": "nginx-deployment-canary"
               },
               "spec": {
                  "replicas": 3,
                  "selector": {
                     "matchLabels": {
                        "app": "nginx",
                        "canary": "true"
                     }
                  },
                  "template": {
                     "metadata": {
                        "labels": {
                           "app": "nginx",
                           "canary": "true"
                        }
                     },
                     "spec": {
                        "containers": [
                           {
                              "image": "nginx:1.7.9",
                              "name": "nginx",
                              "ports": [
                                 {
                                    "containerPort": 80
                                 }
                              ]
                           }
                        ]
                     }
                  }
               }
            }
         ],
         "moniker": {
            "app": "myapp",
            "cluster": "some-cluster",
            "detail": "someDetail",
            "stack": "someStack"
         },
         "name": "Deploy a canary manifest",
         "refId": "Deploy a canary manifest",
         "requisiteStageRefIds": [
            "Wait"
         ],
         "source": "text",
         "type": "deployManifest"
      },
      {
         "account": "staging-demo",
         "cloudProvider": "kubernetes",
         "manifestArtifactAccount": "gitlab-account",
         "manifestArtifactId": "app/manifest.yaml",
         "moniker": {
            "app": "myapp",
            "cluster": "some-cluster",
            "detail": "someDetail",
            "stack": "someStack"
         },
         "name": "Deploy a manifest with artifact",
         "overrideTimeout": true,
         "refId": "Deploy a manifest with artifact",
         "requisiteStageRefIds": [
            "Wait"
         ],
         "restrictExecutionDuringTimeWindow": true,
         "restrictedExecutionWindow": {
            "days": [
               "1",
               "2",
               "3"
            ],
            "whitelist": [
               {
                  "endHour": 7,
                  "endMin": 0,
                  "startHour": 5,
                  "startMin": 0
               }
            ]
         },
         "source": "artifact",
         "stageTimeoutMs": "300000",
         "type": "deployManifest"
      },
      {
         "account": "staging-demo",
         "cloudProvider": "kubernetes",
         "location": "default",
         "manifestName": "Deployment nginx-deployment",
         "name": "Find nginx-deployment",
         "refId": "Find nginx-deployment",
         "requisiteStageRefIds": [
            "Deploy a manifest",
            "Deploy a canary manifest",
            "Deploy a manifest with artifact"
         ],
         "type": "findArtifactsFromResource"
      },
      {
         "job": "smoketest",
         "markUnstableAsSuccessful": "false",
         "master": "staging-jenkins",
         "name": "Run Jenkins Job",
         "notifications": [
            {
               "address": "development",
               "level": "stage",
               "message": {
                  "stage.complete": {
                     "text": "test"
                  },
                  "stage.failed": {
                     "text": "testf: one two three, $variable, %value, \"quoted\": https://example.com"
                  }
               },
               "type": "slack",
               "when": [
                  "stage.starting",
                  "stage.failed",
                  "stage.complete"
               ]
            }
         ],
         "overrideTimeout": true,
         "refId": "Run Jenkins Job",
         "requisiteStageRefIds": [
            "Find nginx-deployment"
         ],
         "sendNotifications": true,
         "stageTimeoutMs": "300000",
         "type": "jenkins",
         "waitForCompletion": "true"
      }
   ],
   "triggers": [
      {
         "account": "docker-account",
         "enabled": true,
         "name": "myDockerTrigger",
         "organization": "your-docker-org",
         "registry": "index.docker.io",
         "repository": "yourorg/app",
         "tag": "^git-.*$",
         "type": "docker"
      },
      {
         "branch": "master",
         "enabled": true,
         "name": "myGitTrigger",
         "project": "your-org",
         "slug": "your-project",
         "source": "gitlab",
         "type": "git"
      }
   ]
}
```

```
$ jsonnet demo-pipeline.jsonnet > demo-pipeline.json && \
  spin pipeline save --file demo-pipeline.json
```

<img width="908" alt="作成されたpipeline" src="https://qiita-image-store.s3.amazonaws.com/0/9622/9d5ab915-2070-7ce7-f610-79d649477c09.png">

※1:最初試してたときはpipelineのIDがpipeline.libsonnetになく動かなかったけどSlackで質問したら直してくれた
※2:今のバージョンだとExpected Artifactsがうまく設定できてないっぽい

## pipeline.libsponnetを読み解く

ひとまず目的は達成できたので次にpipeline.libsonnetをサラっと見てみたいと思います。

以下の `pipeline.libsponnet` の `pipeline` 部分を見る限り、pipelineに必要なフィールドは次のように見えます。

- id : String - pipelineのID
- limitConcurrent : boolean
- keepWaitingPipelines : boolean
- notifications: Array<notification> - 通知
- stages: Array<stage> - ステージ
- triggers: Array<trigger> - トリガー
- application: String - アプリケーション名
- expectedArtifacts: Array<expectedArtifact> - Artifact
- name: String - pipelineの名前

```jsonnet:pipeline.libsonnet
  pipeline():: {
    keepWaitingPipelines: false,
    limitConcurrent: true,
    notifications: [],
    stages: [],
    triggers: [],
    withApplication(application):: self + { application: application },
    withExpectedArtifacts(expectedArtifacts):: self + if std.type(expectedArtifacts) == 'array' then { expectedArtifacts: expectedArtifacts } else { expectedArtifacts: [expectedArtifacts] },
    withId(id):: self + { id: id },
    withKeepWaitingPipelines(keepWaitingPipelines):: self + { keepWaitingPipelines: keepWaitingPipelines },
    withLimitConcurrent(limitConcurrent):: self + { limitConcurrent: limitConcurrent },
    withName(name):: self + { name: name },
    withNotifications(notifications):: self + if std.type(notifications) == 'array' then { notifications: notifications } else { notifications: [notifications] },
    withStages(stages):: self + if std.type(stages) == 'array' then { stages: stages } else { stages: [stages] },
    withTriggers(triggers):: self + if std.type(triggers) == 'array' then { triggers: triggers } else { triggers: [triggers] },
  }
```

`demo.jsonnet` を見ると `sponnet.pipeline()` は以下のように使われています

```jsonnet:demo.jsonnet
local sponnet = import '../pipeline.libsonnet';
local deployment = import 'deployment.json';
local kubeutils = import 'kubeutils.libsonnet';

local canaryDeployment = kubeutils.canary(deployment);
local account = 'staging-demo';
local app = 'myapp';

(中略)

sponnet.pipeline()
.withApplication(app)
.withExpectedArtifacts([expectedDocker, expectedManifest])
.withId('sponnet-demo-pipeline')
.withName('Demo pipeline')
.withNotifications([emailPipelineNotification])
.withTriggers([dockerTrigger, gitTrigger])
.withStages([manualJudgment, checkPreconditions, wait, deployManifestTextBaseline, deployManifestTextCanary, deployManifestArtifact, findArtifactsFromResource, jenkinsJob])
```

上記 `demo.jsonnet` を見るにpipeline を作るには状況に応じて以下を作成＆設定し指定してやる必要があります。
- アプリケーション名
- PipelineのID
- Pipelineの名前
- Notification
- Trigger
- Stage
- ExpectedArtifacts

これらの作成方法については `pipeline.libsonnet` に定義が記載されているのでそちらを確認するのが良いです。

#### Notification

Notificationについてはこれだけです。

```jsonnet:pipeline.libsonnet
  // notifications

  notification:: {
    withAddress(address):: self + { address: address },
    withCC(cc):: self + { cc: cc },
    withLevel(level):: self + { level: level },
    // Custom notification messages are optional
    withWhen(when, message=false):: self + {
      when+: [when],
      [if std.isString(message) then 'message']+: {
        [when]: {
          text: message,
        },
      },
    },
    withType(type):: self + { type: type },
  }
```

#### Trigger

Triggerは以下のコードを継承したDockerやgitだけが用意されています。（詳細は [コード参照](https://github.com/spinnaker/spinnaker/blob/master/sponnet/pipeline.libsonnet#L104-L117)）

```jsonnet:pipeline.libsonnet
  // triggers

  trigger(name, type):: {
    enabled: true,
    name: name,
    type: type,
    withExpectedArtifacts(expectedArtifacts):: self + if std.type(expectedArtifacts) == 'array' then { expectedArtifactIds: std.map(function(expectedArtifact) expectedArtifact.id, expectedArtifacts) } else { expectedArtifactIds: [expectedArtifacts.id] },
  },
```

#### Stage

ステージのキホンは以下のコードになります。

```jsonnet:pipeline.libsonnet
  // stages

  stage(name, type):: {
    refId: name,
    name: name,
    type: type,
    requisiteStageRefIds: [],
    withNotifications(notifications):: self + { sendNotifications: true } + if std.type(notifications) == 'array' then { notifications: notifications } else { notifications: [notifications] },
    withRequisiteStages(stages):: self + if std.type(stages) == 'array' then { requisiteStageRefIds: std.map(function(stage) stage.refId, stages) } else { requisiteStageRefIds: [stages.refId] },
    // execution options
    // TODO (kskewes): Use a toggle or other mechanism to enforce single choice of `If stage fails`
    withCompleteOtherBranchesThenFail(completeOtherBranchesThenFail):: self + { completeOtherBranchesThenFail: completeOtherBranchesThenFail },
    withContinuePipeline(continuePipeline):: self + { continuePipeline: continuePipeline },
    withFailPipeline(failPipeline):: self + { failPipeline: failPipeline },
    withFailOnFailedExpressions(failOnFailedExpressions):: self + { failOnFailedExpressions: failOnFailedExpressions },
    withStageEnabled(expression):: self + { stageEnabled: { type: 'expression', expression: expression } },
    withRestrictedExecutionWindow(days, whitelist, jitter=null):: self + { restrictExecutionDuringTimeWindow: true } +
                                                                  (if std.type(days) == 'array' then { restrictedExecutionWindow+: { days: days } } else { restrictedExecutionWindow+: { days: [days] } }) +
                                                                  (if std.type(whitelist) == 'array' then { restrictedExecutionWindow+: { whitelist: whitelist } } else { restrictedExecutionWindow+: { whitelist: [whitelist] } }) +
                                                                  (std.prune({ restrictedExecutionWindow+: { jitter: jitter } })),
    withSkipWindowText(skipWindowText):: self + { skipWindowText: skipWindowText },
    withOverrideTimeout(timeoutMs):: self + { overrideTimeout: true, stageTimeoutMs: timeoutMs },
  }
```

これを継承した以下のステージが用意されています（詳細は [コード参照](https://github.com/spinnaker/spinnaker/blob/master/sponnet/pipeline.libsonnet#L147-L280)）
- wait
- checkPreconditions
- withClusterSize
- findArtifactFromExecution
- manualJudgement
- jenkins
- deployManifest
- deleteManifest
- findArtifactsFromResource
- patchManifest
- scaleManifest
- undoRolloutManifest
- pipeline
- wercker // This stage has only been written from spec and not tested


#### Artifact & ExpectedArtifact

```jsonnet:pipeline.libsonnet
  // artifacts

  artifact(type):: {
    type: type,
    withArtifactAccount(artifactAccount):: self + { artifactAccount: artifactAccount },
    withLocation(location):: self + { location: location },
    withName(name):: self + { name: name },
    withReference(reference):: self + { reference: reference },
    withVersion(version):: self + { version: version },
  }
```

Artifactで設定できるのは以下の通りですがArtifactにTypeを指定しているだけでほぼ素のArtifactと同じです。([詳細](https://github.com/spinnaker/spinnaker/blob/master/sponnet/pipeline.libsonnet#L39-L47))

- bitbucketFile
- dockerImage
- embeddedBase64
- gcsObject
- githubFile
- gitlabFile
- httpFile
- kubernetesObject


```jsonnet:pipeline.libsonnet
  // expected artifacts
  // TODO: This section may need splitting out by artifact type due to differing field requirements.

  expectedArtifact(id):: {
    id: id,
    withMatchArtifact(matchArtifact):: self + {
      matchArtifact+: {
        // TODO: For Docker, the name field should be registry and repository.
        name: matchArtifact.name,
        type: matchArtifact.type,
      },
    },
    withDefaultArtifact(defaultArtifact):: self + {
      defaultArtifact: {
        reference: defaultArtifact.reference,
        type: defaultArtifact.type,
        // TODO: Some Artifact types (docker) don't require version to be set. It may be better to do this differently.
        Handling connection for 8084ifact, 'version') then 'version']: defaultArtifact.version,
      },
    },
    withUsePriorArtifact(usePriorArtifact):: self + { usePriorArtifact: usePriorArtifact },
    withUseDefaultArtifact(useDefaultArtifact):: self + { useDefaultArtifact: useDefaultArtifact },
  }
```

## まとめ

spin と pipeline.libsonnetを使ってSpinnakerのPipelineが作成できること、ならびにpipeline.libsonnetの中身を少しだけ覗いてみました。

現状はsponnetに関してのドキュメントも揃っておらずコードを読まないと使うのは厳しいという状態ですが、本記事がspinとpipeline.libsonnetを使ってSpinnakerのpipelineを自由に作れるきっかけになれば幸いです。

そんなわけで本日1歳になった子供の寝顔を見ながら寝てこようと思います。
みなさん良いお年を！