Linkerd v2.0 自動Canaryリリース

# 自動Canaryリリース
Linkerdのトラフィック分割機能により、サービス間でトラフィックを動的にシフトできます。
これを使用して、ブルーグリーン展開やCanaryリリースなどを実装できます。

Linkerdでは、単にあるバージョンのサービスから次のバージョンにトラフィックを移行するだけでなく、トラフィックの分割をゴールデンメトリックテレメトリと組み合わせ、メトリックに基づいて自動的にトラフィックを決定できます。

例として、成功率を継続的に監視しながら、トラフィックを古い展開から新しい展開に徐々にシフトできます。
その際、いずれかの時点で成功率が低下した場合、トラフィックを元のdeploymentに戻すことができます。


# Flaggerのインストール (kubernetes v1.14移行)

Linkerdが実際のトラフィックルーティングを管理し、Flaggerが新しいKubernetesリソースの作成、メトリックの監視、およびユーザーを新しいバージョンに段階的に送信するプロセスを自動化します。
Flaggerをクラスターに追加し、Linkerdと連携させるために、以下を実行します。

```
# kubectl apply -k github.com/weaveworks/flagger/kustomize/linkerd
customresourcedefinition.apiextensions.k8s.io/canaries.flagger.app created
serviceaccount/flagger created
clusterrole.rbac.authorization.k8s.io/flagger created
clusterrolebinding.rbac.authorization.k8s.io/flagger created
deployment.apps/flagger created
```


# sample アプリケーションをデプロイ

```
# kubectl create ns test
namespace/test created

# kubectl apply -f https://run.linkerd.io/flagger.yml
deployment.apps/load created
configmap/frontend created
deployment.apps/frontend created
service/frontend created
deployment.apps/podinfo created
service/podinfo created


# k get all
NAME                            READY   STATUS    RESTARTS   AGE
pod/frontend-66db854c46-vwhbg   2/2     Running   0          57s
pod/load-7fbb5f6f84-8mb99       2/2     Running   0          57s
pod/podinfo-6664bccb6-88rgd     2/2     Running   0          57s


NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/frontend   ClusterIP   10.100.200.132   <none>        8080/TCP   57s
service/podinfo    ClusterIP   10.100.200.12    <none>        9898/TCP   57s


NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/frontend   1/1     1            1           57s
deployment.apps/load       1/1     1            1           57s
deployment.apps/podinfo    1/1     1            1           57s

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/frontend-66db854c46   1         1         1       57s
replicaset.apps/load-7fbb5f6f84       1         1         1       57s
replicaset.apps/podinfo-6664bccb6     1         1         1       57s
```

現在の状態としてはこんな感じになっています。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/416540/0611bef1-4915-0b2f-b295-0b5e114316b0.png)


それでは、port forwardingをして、podから情報を取得してみます。

```
# kubectl -n test port-forward svc/podinfo 9898
Forwarding from 127.0.0.1:9898 -> 9898


# curl 127.0.0.1:9898
{
  "hostname": "podinfo-6664bccb6-88rgd",
  "version": "1.7.0",
  "revision": "4fc593f42c7cd2e7319c83f6bfd3743c05523883",
  "color": "blue",
  "message": "greetings from podinfo v1.7.0",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.11.2",
  "num_goroutine": "7",
  "num_cpu": "2"
}
```

バージョンとしては、1.7.0として取得できました。


# Canary releaseのpolicyを定義

これから、自動でCanaryリリースをするためのPolicyを適用していきます。

```canary.yaml
apiVersion: flagger.app/v1alpha3
kind: Canary
metadata:
  name: podinfo
  namespace: test
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  service:
    port: 9898
  canaryAnalysis:
    interval: 10s
    threshold: 5
    stepWeight: 10
    metrics:
    - name: request-success-rate
      threshold: 99
      interval: 1m
```

これを適用し、動きを見ていきます。

```
# k apply -f canary.yaml
canary.flagger.app/podinfo created


kubectl -n test get ev --watch
<省略>
24s         Warning   Synced                         canary/podinfo                             Halt advancement podinfo-primary.test waiting for rollout to finish: observed deployment generation less then desired generation
4s          Warning   Synced                         canary/podinfo                             Halt advancement podinfo-primary.test waiting for rollout to finish: 0 of 1 updated replicas are available
0s          Warning   Synced                         canary/podinfo                             Halt advancement podinfo-primary.test waiting for rollout to finish: 0 of 1 updated replicas are available
0s          Normal    Pulled                         pod/podinfo-primary-8c85f78c4-99bpn        Successfully pulled image "quay.io/stefanprodan/podinfo:1.7.0"
0s          Normal    Created                        pod/podinfo-primary-8c85f78c4-99bpn        Created container podinfod
0s          Normal    Started                        pod/podinfo-primary-8c85f78c4-99bpn        Started container podinfod
0s          Normal    Pulled                         pod/podinfo-primary-8c85f78c4-99bpn        Container image "gcr.io/linkerd-io/proxy:stable-2.6.0" already present on machine
0s          Normal    Created                        pod/podinfo-primary-8c85f78c4-99bpn        Created container linkerd-proxy
0s          Normal    Started                        pod/podinfo-primary-8c85f78c4-99bpn        Started container linkerd-proxy
0s          Warning   Synced                         canary/podinfo                             Halt advancement podinfo-primary.test waiting for rollout to finish: 0 of 1 updated replicas are available
0s          Normal    ScalingReplicaSet              deployment/podinfo                         Scaled down replica set podinfo-6664bccb6 to 0
0s          Normal    SuccessfulDelete               replicaset/podinfo-6664bccb6               Deleted pod: podinfo-6664bccb6-88rgd
0s          Normal    Killing                        pod/podinfo-6664bccb6-88rgd                Stopping container linkerd-proxy
0s          Normal    Killing                        pod/podinfo-6664bccb6-88rgd                Stopping container podinfod
0s          Normal    Synced                         canary/podinfo                             Initialization done! podinfo.test
```

Initializeが終了しました。
リソースたちを見ていきます。

```
# k get all
NAME                                  READY   STATUS    RESTARTS   AGE
pod/frontend-66db854c46-vwhbg         2/2     Running   0          24m
pod/load-7fbb5f6f84-8mb99             2/2     Running   0          24m
pod/podinfo-primary-8c85f78c4-99bpn   2/2     Running   0          6m38s


NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/frontend          ClusterIP   10.100.200.132   <none>        8080/TCP   24m
service/podinfo           ClusterIP   10.100.200.12    <none>        9898/TCP   24m
service/podinfo-canary    ClusterIP   10.100.200.167   <none>        9898/TCP   6m38s
service/podinfo-primary   ClusterIP   10.100.200.23    <none>        9898/TCP   6m38s


NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/frontend          1/1     1            1           24m
deployment.apps/load              1/1     1            1           24m
deployment.apps/podinfo           0/0     0            0           24m
deployment.apps/podinfo-primary   1/1     1            1           6m38s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/frontend-66db854c46         1         1         1       24m
replicaset.apps/load-7fbb5f6f84             1         1         1       24m
replicaset.apps/podinfo-6664bccb6           0         0         0       24m
replicaset.apps/podinfo-primary-8c85f78c4   1         1         1       6m38s



NAME                         STATUS        WEIGHT   LASTTRANSITIONTIME
canary.flagger.app/podinfo   Initialized   0        2019-12-14T07:41:25Z
```

`primary`というものが増えました。
現在、このアプリケーションは以下の状態となっています。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/416540/70b2c1c8-712d-0c4a-318b-67105f7a4313.png)


Linkerdからも確認してみます。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/416540/fd28740a-50dd-4203-8197-2cdb824ba7ba.png)



# ロールアウトを開始

kubectl set image コマンドは、Deployment のポッドの nginx イメージを 1 つずつ更新します。
使用例：
 `kubectl set image deployment <deployment名> <container名> = <新image名(:tag)>`

これを用いて、新versionのデプロイを行います。

```
# kubectl -n test set image deployment/podinfo podinfod=quay.io/stefanprodan/podinfo:1.7.1
```

Canaryの動きを見ていきます。

```
# kubectl -n test get ev --watch
1s          Normal    Synced                    canary/podinfo                            New revision detected! Scaling up podinfo.test
1s          Normal    ScalingReplicaSet         deployment/podinfo                        Scaled up replica set podinfo-5b549f6b9c to 1
0s          Normal    Pulling                   pod/podinfo-5b549f6b9c-xmvcw              Pulling image "quay.io/stefanprodan/podinfo:1.7.1"
0s          Warning   Synced                    canary/podinfo                            Halt advancement podinfo.test waiting for rollout to finish: 0 of 1 updated replicas are available
0s          Warning   Synced                    canary/podinfo                            Halt advancement podinfo.test waiting for rollout to finish: 0 of 1 updated replicas are available
0s          Warning   Synced                    canary/podinfo                            Halt advancement podinfo.test waiting for rollout to finish: 0 of 1 updated replicas are available
0s          Normal    Pulled                    pod/podinfo-5b549f6b9c-xmvcw              Successfully pulled image "quay.io/stefanprodan/podinfo:1.7.1"
0s          Normal    Created                   pod/podinfo-5b549f6b9c-xmvcw              Created container podinfod
0s          Normal    Started                   pod/podinfo-5b549f6b9c-xmvcw              Started container podinfod
0s          Normal    Pulled                    pod/podinfo-5b549f6b9c-xmvcw              Container image "gcr.io/linkerd-io/proxy:stable-2.6.0" already present on machine
0s          Normal    Created                   pod/podinfo-5b549f6b9c-xmvcw              Created container linkerd-proxy
0s          Normal    Started                   pod/podinfo-5b549f6b9c-xmvcw              Started container linkerd-proxy
0s          Normal    Synced                    canary/podinfo                            Starting canary analysis for podinfo.test
0s          Normal    Synced                    canary/podinfo                            Advance podinfo.test canary weight 10
0s          Warning   Synced                    canary/podinfo                            Halt advancement no values found for metric request-success-rate probably podinfo.test is not receiving traffic
0s          Normal    Synced                    canary/podinfo                            Advance podinfo.test canary weight 20
0s          Normal    Synced                    canary/podinfo                            Advance podinfo.test canary weight 30
0s          Normal    Synced                    canary/podinfo                            Advance podinfo.test canary weight 40
0s          Normal    Synced                    canary/podinfo                            Advance podinfo.test canary weight 50
0s          Normal    Synced                    canary/podinfo                            Advance podinfo.test canary weight 60
0s          Normal    Synced                    canary/podinfo                            Advance podinfo.test canary weight 70
0s          Normal    Synced                    canary/podinfo                            (combined from similar events): Advance podinfo.test canary weight 80
0s          Normal    Synced                    canary/podinfo                            (combined from similar events): Advance podinfo.test canary weight 90
0s          Normal    Synced                    canary/podinfo                            (combined from similar events): Advance podinfo.test canary weight 100
0s          Normal    Synced                    canary/podinfo                            (combined from similar events): Copying podinfo.test template spec to podinfo-primary.test
0s          Normal    ScalingReplicaSet         deployment/podinfo-primary                Scaled up replica set podinfo-primary-694659d58d to 1
0s          Normal    Injected                  deployment/podinfo-primary                Linkerd sidecar proxy injected
0s          Normal    SuccessfulCreate          replicaset/podinfo-primary-694659d58d     Created pod: podinfo-primary-694659d58d-lzlkx
0s          Normal    Scheduled                 pod/podinfo-primary-694659d58d-lzlkx      Successfully assigned test/podinfo-primary-694659d58d-lzlkx to b5db1b15-ec4c-4ae6-a8da-4a2ffb3e4446
0s          Normal    Pulled                    pod/podinfo-primary-694659d58d-lzlkx      Container image "gcr.io/linkerd-io/proxy-init:v1.2.0" already present on machine
0s          Normal    Created                   pod/podinfo-primary-694659d58d-lzlkx      Created container linkerd-init
0s          Normal    Started                   pod/podinfo-primary-694659d58d-lzlkx      Started container linkerd-init
0s          Normal    Pulling                   pod/podinfo-primary-694659d58d-lzlkx      Pulling image "quay.io/stefanprodan/podinfo:1.7.1"
0s          Warning   Synced                    canary/podinfo                            Halt advancement podinfo-primary.test waiting for rollout to finish: 1 old replicas are pending termination
0s          Warning   Synced                    canary/podinfo                            Halt advancement podinfo-primary.test waiting for rollout to finish: 1 old replicas are pending termination
0s          Warning   Synced                    canary/podinfo                            Halt advancement podinfo-primary.test waiting for rollout to finish: 1 old replicas are pending termination
0s          Normal    Pulled                    pod/podinfo-primary-694659d58d-lzlkx      Successfully pulled image "quay.io/stefanprodan/podinfo:1.7.1"
0s          Normal    Created                   pod/podinfo-primary-694659d58d-lzlkx      Created container podinfod
0s          Normal    Started                   pod/podinfo-primary-694659d58d-lzlkx      Started container podinfod
0s          Normal    Pulled                    pod/podinfo-primary-694659d58d-lzlkx      Container image "gcr.io/linkerd-io/proxy:stable-2.6.0" already present on machine
0s          Normal    Created                   pod/podinfo-primary-694659d58d-lzlkx      Created container linkerd-proxy
0s          Normal    Started                   pod/podinfo-primary-694659d58d-lzlkx      Started container linkerd-proxy
0s          Warning   Synced                    canary/podinfo                            Halt advancement podinfo-primary.test waiting for rollout to finish: 1 old replicas are pending termination
0s          Normal    ScalingReplicaSet         deployment/podinfo-primary                Scaled down replica set podinfo-primary-8c85f78c4 to 0
0s          Normal    SuccessfulDelete          replicaset/podinfo-primary-8c85f78c4      Deleted pod: podinfo-primary-8c85f78c4-99bpn
0s          Normal    Killing                   pod/podinfo-primary-8c85f78c4-99bpn       Stopping container linkerd-proxy
0s          Normal    Killing                   pod/podinfo-primary-8c85f78c4-99bpn       Stopping container podinfod
0s          Normal    Synced                    canary/podinfo                            (combined from similar events): Routing all traffic to primary
0s          Normal    ScalingReplicaSet         deployment/podinfo                        Scaled down replica set podinfo-5b549f6b9c to 0
0s          Normal    Synced                    canary/podinfo                            (combined from similar events): Promotion completed! Scaling down podinfo.test
0s          Normal    SuccessfulDelete          replicaset/podinfo-5b549f6b9c             Deleted pod: podinfo-5b549f6b9c-xmvcw
0s          Normal    Killing                   pod/podinfo-5b549f6b9c-xmvcw              Stopping container linkerd-proxy
0s          Normal    Killing                   pod/podinfo-5b549f6b9c-xmvcw              Stopping container podinfod
```

Linkerdを使って、動きを監視していきます。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/416540/10265800-4177-f39f-83f3-44b438c8572b.png)


![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/416540/9bc4d53f-f4bd-d423-7e4b-ef5df6cc3200.png)


![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/416540/97a4ce31-cc2b-1a40-f439-0cbf99bf78b3.png)



1. 更新がかかった、deploy/podinfoにもトラフィックが流れるようになる
2. トラフィックが新versionの方だけになる。
3. 新versionの方がprimaryのpodとして再デプロイされ、再度deploy/podinfoは落ちる。

という流れになっていることがわかりました。


replica数が多い場合などは、以下の動きになるようです。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/416540/bd6747f2-ec84-f11a-1848-81ff45bda889.png)



Canary リリースが成功したことが以下のコマンドからも確認できます。

```
# watch kubectl -n test get canary
Every 2.0s: kubectl -n test get canary                                                                                                  Sat Dec 14 17:28:16 2019

NAME      STATUS      WEIGHT   LASTTRANSITIONTIME
podinfo   Succeeded   0        2019-12-14T08:20:55Z
```

それでは、先ほどと同様にport forwarding をして確認していきます。

```
# kubectl -n test port-forward svc/podinfo 9898
Forwarding from 127.0.0.1:9898 -> 9898

# curl 127.0.0.1:9898
{
  "hostname": "podinfo-primary-694659d58d-lzlkx",
  "version": "1.7.1",
  "revision": "c9dc78f29c5087e7c181e58a56667a75072e6196",
  "color": "blue",
  "message": "greetings from podinfo v1.7.1",
  "goos": "linux",
  "goarch": "amd64",
  "runtime": "go1.11.12",
  "num_goroutine": "7",
  "num_cpu": "2"
}
```

しっかり、versionが1.7.1にアップグレードされています。