etcd on docker でクラスタ構成を試してみる

# 目的(確認したいこと)
- etcdをクラスタ構成で動作させるるときのdocker run のパラメータなどの書き方と起動方法。
- 3台がクラスタ化されているかの確認。
- リーダー(マスタと)なる1台を停止したときに、他の2台のどちらかにリーダーが引き継がれるか。
- 停止した1台を再開したときに、クラスタの仲間に戻るか。

# 環境と構成
- 動作環境は、GCPのVMを1台。
- etcd001, etcd002, etcd003と3台のetcd(on docker)をクラスタ構成でVM上に立てる。
- etcdのコンテナはこちらを利用。(gcr.io/etcd-development/etcd:v3.4.7)

# Dockerネットワークの作成

コンテナ間で通信させるための土俵として、
「base-network」という名称で、docker(etcd)が所属するネットワークを作成。

```
docker network create \
    --driver bridge \
    --subnet=172.19.0.0/24 \
    --gateway=172.19.0.1 \
    base-network
```

「gateway=172.19.0.1」は、VMのIPアドレス。subnetは、上記に合わせて適宜設定する。

# 1号機(etcd001)の起動

**IPアドレス** : 172.19.0.10
**ポート番号** : docker内で、etcdがサービスとして使うポートは「2379と2380」の二つ。VM側から参照させるためにそのまま外だしするすると、3台同じ番号になり被るので「1」号機なので「1」をつけてVM側に公開しています。(1号機:2379->23791,2380->23801)(2号機:2379->23792,2380->23802)
**データ** : データを永続化(dockerを落としてもデータが消えないように)させるため、事前にVM側にディレクトリを作ってマウント **/tmp/etcd-data-001.tmp/** しています。

分かりやすく標準出力にログを出しているので、サービス起動時はターミナルを占拠します。

```
docker run \
  --network base-network \
  --ip 172.19.0.10 \
  -p 23791:2379 \
  -p 23801:2380 \
  --mount type=bind,source=/tmp/etcd-data-001.tmp,destination=/etcd-data \
  --name etcd001 \
  gcr.io/etcd-development/etcd:v3.4.7 \
  /usr/local/bin/etcd \
  --name etcd001 \
  --data-dir /etcd-data \
  --listen-client-urls http://172.19.0.10:23791,http://127.0.0.1:2379 \
  --advertise-client-urls http://172.19.0.10:23791 \
  --listen-peer-urls http://172.19.0.10:23801 \
  --initial-advertise-peer-urls http://172.19.0.10:23801 \
  --initial-cluster etcd001=http://172.19.0.10:23801,etcd002=http://172.19.0.20:23802,etcd003=http://172.19.0.30:23803 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --log-level info \
  --logger zap \
  --log-outputs stderr
```

# 2号機(etcd002)の起動

**IPアドレス**: 172.19.0.20
**ポート番号**: 2379->23792,2380->23802
**データ**: /tmp/etcd-data-002.tmp/

```
docker run \
  --network base-network \
  --ip 172.19.0.20 \
  -p 23792:2379 \
  -p 23802:2380 \
  --mount type=bind,source=/tmp/etcd-data-002.tmp,destination=/etcd-data \
  --name etcd002 \
  gcr.io/etcd-development/etcd:v3.4.7 \
  /usr/local/bin/etcd \
  --name etcd002 \
  --data-dir /etcd-data \
  --listen-client-urls http://172.19.0.20:23792,http://127.0.0.1:2379 \
  --advertise-client-urls http://172.19.0.20:23792 \
  --listen-peer-urls http://172.19.0.20:23802 \
  --initial-advertise-peer-urls http://172.19.0.20:23802 \
  --initial-cluster etcd001=http://172.19.0.10:23801,etcd002=http://172.19.0.20:23802,etcd003=http://172.19.0.30:23803 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --log-level info \
  --logger zap \
  --log-outputs stderr
```

# 3号機(etcd003)の起動

**IPアドレス**: 172.19.0.30
**ポート番号**: 2379->23793,2380->23803
**データ**: /tmp/etcd-data-003.tmp/

```
docker run \
  --network base-network \
  --ip 172.19.0.30 \
  -p 23793:2379 \
  -p 23803:2380 \
  --mount type=bind,source=/tmp/etcd-data-003.tmp,destination=/etcd-data \
  --name etcd003 \
  gcr.io/etcd-development/etcd:v3.4.7 \
  /usr/local/bin/etcd \
  --name etcd003 \
  --data-dir /etcd-data \
  --listen-client-urls http://172.19.0.30:23793,http://127.0.0.1:2379 \
  --advertise-client-urls http://172.19.0.30:23793 \
  --listen-peer-urls http://172.19.0.30:23803 \
  --initial-advertise-peer-urls http://172.19.0.30:23803 \
  --initial-cluster etcd001=http://172.19.0.10:23801,etcd002=http://172.19.0.20:23802,etcd003=http://172.19.0.30:23803 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --log-level info \
  --logger zap \
  --log-outputs stderr
```

# 3台がクラスタ化されてるかなの確認

お。できてるぽい。
そして1号機がリーダー(マスター)のようだ。(endpoint statusのIS LEADERで確認)

・member list

```
[fujita1203@instance-1 ~]$ docker exec etcd001 /bin/sh -c "ETCDCTL_API=3 && /usr/local/bin/etcdctl -w table member list"
+------------------+---------+---------+--------------------------+--------------------------+------------+
|        ID        | STATUS  |  NAME   |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+---------+--------------------------+--------------------------+------------+
| a5a9dff0272bf242 | started | etcd003 | http://172.19.0.30:23803 | http://172.19.0.30:23793 |      false |
| a9f542562b5fed3d | started | etcd001 | http://172.19.0.10:23801 | http://172.19.0.10:23791 |      false |
| db0290dd0457c29f | started | etcd002 | http://172.19.0.20:23802 | http://172.19.0.20:23792 |      false |
+------------------+---------+---------+--------------------------+--------------------------+------------+
```

・endpoint health

```
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint health"
+--------------------------+--------+--------------+-------+
|         ENDPOINT         | HEALTH |     TOOK     | ERROR |
+--------------------------+--------+--------------+-------+
| http://172.19.0.30:23793 |   true | 310.139671ms |       |
| http://172.19.0.20:23792 |   true |  310.08538ms |       |
| http://172.19.0.10:23791 |   true | 310.206383ms |       |
+--------------------------+--------+--------------+-------+
```

・endpoint status

```
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint status"
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://172.19.0.10:23791 | a9f542562b5fed3d |   3.4.7 |   20 kB |      true |      false |        15 |          9 |                  9 |        |
| http://172.19.0.20:23792 | db0290dd0457c29f |   3.4.7 |   20 kB |     false |      false |        15 |          9 |                  9 |        |
| http://172.19.0.30:23793 | a5a9dff0272bf242 |   3.4.7 |   20 kB |     false |      false |        15 |          9 |                  9 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
```

- 最初は値が入ってないことを確認

```
[fujita1203@instance-1 ~]$ curl -L http://172.19.0.10:23801/v3/kv/range -X POST -d '{"key": "Zm9v"}'
404 page not found
[fujita1203@instance-1 ~]$ curl -L http://172.19.0.20:23802/v3/kv/range -X POST -d '{"key": "Zm9v"}'
404 page not found
[fujita1203@instance-1 ~]$ curl -L http://172.19.0.30:23803/v3/kv/range -X POST -d '{"key": "Zm9v"}'
404 page not found
```

- 3号機に入れたものが、1,2,3号機から取得できるか確認

コマンド

```
curl -L http://172.19.0.30:23793/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}' | python -mjson.tool
curl -L http://172.19.0.10:23791/v3/kv/range -X POST -d '{"key": "Zm9v"}' | python -mjson.tool
curl -L http://172.19.0.20:23792/v3/kv/range -X POST -d '{"key": "Zm9v"}' | python -mjson.tool
curl -L http://172.19.0.30:23793/v3/kv/range -X POST -d '{"key": "Zm9v"}' | python -mjson.tool
```

とれたとれた。(長いので1号機だけ) 

```
[fujita1203@instance-1 ~]$ curl -L http://172.19.0.10:23791/v3/kv/range -X POST -d '{"key": "Zm9v"}' | python -mjson.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   234  100   219  100    15  19527   1337 --:--:-- --:--:-- --:--:-- 21900
{
    "count": "1",
    "header": {
        "cluster_id": "3282096315814289506",
        "member_id": "12246767699566259517",
        "raft_term": "16",
        "revision": "2"
    },
    "kvs": [
        {
            "create_revision": "2",
            "key": "Zm9v",
            "mod_revision": "2",
            "value": "YmFy",
            "version": "1"
        }
    ]
}
```

# 1号機(リーダー)を停止したときの挙動と、復帰したときの挙動の確認

- リーダーを強制終了(ctr+c)してみる (リーダーがetcd002になってる)

```
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint status"
{"level":"warn","ts":"2020-04-12T23:19:41.068Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///http://172.19.0.10:23791","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Failed to get the status of endpoint http://172.19.0.10:23791 (context deadline exceeded)
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://172.19.0.20:23792 | db0290dd0457c29f |   3.4.7 |   20 kB |      true |      false |        16 |         11 |                 11 |        |
| http://172.19.0.30:23793 | a5a9dff0272bf242 |   3.4.7 |   20 kB |     false |      false |        16 |         11 |                 11 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table member list"
+------------------+---------+---------+--------------------------+--------------------------+------------+
|        ID        | STATUS  |  NAME   |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+---------+--------------------------+--------------------------+------------+
| a5a9dff0272bf242 | started | etcd003 | http://172.19.0.30:23803 | http://172.19.0.30:23793 |      false |
| a9f542562b5fed3d | started | etcd001 | http://172.19.0.10:23801 | http://172.19.0.10:23791 |      false |
| db0290dd0457c29f | started | etcd002 | http://172.19.0.20:23802 | http://172.19.0.20:23792 |      false |
+------------------+---------+---------+--------------------------+--------------------------+------------+
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint health"
+--------------------------+--------+--------------+---------------------------+
|         ENDPOINT         | HEALTH |     TOOK     |           ERROR           |
+--------------------------+--------+--------------+---------------------------+
| http://172.19.0.30:23793 |   true | 270.115572ms |                           |
| http://172.19.0.20:23792 |   true |   269.9296ms |                           |
| http://172.19.0.10:23791 |  false | 5.000427627s | context deadline exceeded |
+--------------------------+--------+--------------+---------------------------+
{"level":"warn","ts":"2020-04-12T23:22:39.902Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"endpoint://client-b4e8b2e6-5e62-4210-b353-dd518d9051a3/172.19.0.10:23791","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.19.0.10:23791: connect: no route to host\""}
Error: unhealthy cluster
[fujita1203@instance-1 ~]$ 
```

- 1号機を起動させる (クラスタに仲間入りした。2号機がリーダーのまま。1号機から値がとれる。)

```
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint status"
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://172.19.0.10:23791 | a9f542562b5fed3d |   3.4.7 |   20 kB |     false |      false |        16 |         12 |                 12 |        |
| http://172.19.0.20:23792 | db0290dd0457c29f |   3.4.7 |   20 kB |      true |      false |        16 |         12 |                 12 |        |
| http://172.19.0.30:23793 | a5a9dff0272bf242 |   3.4.7 |   20 kB |     false |      false |        16 |         12 |                 12 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table member list"
+------------------+---------+---------+--------------------------+--------------------------+------------+
|        ID        | STATUS  |  NAME   |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+---------+--------------------------+--------------------------+------------+
| a5a9dff0272bf242 | started | etcd003 | http://172.19.0.30:23803 | http://172.19.0.30:23793 |      false |
| a9f542562b5fed3d | started | etcd001 | http://172.19.0.10:23801 | http://172.19.0.10:23791 |      false |
| db0290dd0457c29f | started | etcd002 | http://172.19.0.20:23802 | http://172.19.0.20:23792 |      false |
+------------------+---------+---------+--------------------------+--------------------------+------------+
[fujita1203@instance-1 ~]$ docker exec etcd003 /bin/sh -c "ETCDCTL_API=3 /usr/local/bin/etcdctl -w table --endpoints http://172.19.0.10:23791,http://172.19.0.20:23792,http://172.19.0.30:23793 endpoint health"
+--------------------------+--------+--------------+-------+
|         ENDPOINT         | HEALTH |     TOOK     | ERROR |
+--------------------------+--------+--------------+-------+
| http://172.19.0.30:23793 |   true | 145.849727ms |       |
| http://172.19.0.10:23791 |   true |  145.79547ms |       |
| http://172.19.0.20:23792 |   true | 145.736273ms |       |
+--------------------------+--------+--------------+-------+
[fujita1203@instance-1 ~]$ curl -L http://172.19.0.10:23791/v3/kv/range -X POST -d '{"key": "Zm9v"}' | python -mjson.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   234  100   219  100    15  19527   1337 --:--:-- --:--:-- --:--:-- 21900
{
    "count": "1",
    "header": {
        "cluster_id": "3282096315814289506",
        "member_id": "12246767699566259517",
        "raft_term": "16",
        "revision": "2"
    },
    "kvs": [
        {
            "create_revision": "2",
            "key": "Zm9v",
            "mod_revision": "2",
            "value": "YmFy",
            "version": "1"
        }
    ]
}
```

一通り今回確認したかったことは確認できたかな。

# 参考

https://etcd.io/docs/v3.4.0/op-guide/container/

https://tech-lab.sios.jp/archives/13221
http://pj-doaa.hatenablog.com/entry/2019/09/04/144520
https://qiita.com/pocket8137/items/ef44ca68ffc0f4e70995
https://qiita.com/hana_shin/items/602f98bd9b153d22e50c

そもそもetcdって何状態だったので、とても参考になりました。
ありがとうございました。