【Docker】etcd+docker-machineでdocker-swarmを使う

etcdを管理情報を扱うBackendにして、docker-machineを使ってSwarm Clusterを構築する。

#　環境

* docker(version 1.12.3)
* docker-machine(version 0.7.0)
* etcd(version 2.3.7)

# Vagrantfile

```
# -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # If true, then any SSH connections made will enable agent forwarding.
  # Default value: false
  config.ssh.forward_agent = true

  config.vm.define "manager" do |d|
    d.vm.box = "ubuntu/trusty64"
    d.vm.network :private_network, ip: "192.168.33.10", virtualbox__intnet: "intnet"
  end

  config.vm.define "node1" do |d|
    d.vm.box = "ubuntu/trusty64"
    d.vm.network :private_network, ip: "192.168.33.20", virtualbox__intnet: "intnet"
  end

  config.vm.define "node2" do |d|
    d.vm.box = "ubuntu/trusty64"
    d.vm.network :private_network, ip: "192.168.33.30", virtualbox__intnet: "intnet"
  end

  config.vm.define "kvstore" do |d|
    d.vm.box = "ubuntu/trusty64"
    d.vm.network :private_network, ip: "192.168.33.40", virtualbox__intnet: "intnet"
  end

  config.vm.define "host" do |d|
    d.vm.box = "ubuntu/trusty64"
    d.vm.network :private_network, ip: "192.168.33.50", virtualbox__intnet: "intnet"
  end
end
```

|        |IP             | 用途  |
|--------|---------------|---|
|manager |192.168.33.10  |Swarmクラスターのmaster         |
|node1   |192.168.33.20  |Swarmクラスターのnode1          |
|node2   |192.168.33.30  |Swarmクラスターのmaster         |
|kvstore |192.168.33.40  |SwarmクラスターBackendのetcd用  |
|host    |192.168.33.50  |作業場所                      |

hostからmanager, node1, node2, kvstoreにSSHで入れるようにしておく。
最初にetcd作ってからSwarmを構築していくような形。
作業は全てhostVM上でやる。

# etcd

`kvstore`にetcdコンテナを立てる。

```
$ docker-machine create \
   --driver generic --generic-ip-address=192.168.33.40 \
   --generic-ssh-user vagrant --generic-ssh-key ~/.ssh/id_rsa \
   kvstore
$ eval $(docker-machine env kvstore)
$ docker pull quay.io/coreos/etcd:v2.3.7
$ docker run -d --name etcd \
   -p 2379:2379 -p 4001:4001 \
   quay.io/coreos/etcd:v2.3.7 \
   --data-dir=/tmp/default.etcd \
   --advertise-client-urls 'http://192.168.33.40:2379,http://192.168.33.40:4001' \
   --listen-client-urls 'http://0.0.0.0:2379,http://0.0.0.0:4001'
$ ETCDCTL_ENDPOINT="http://192.168.33.40:2379" ./etcdctl --no-sync  get /
```

## はまったポイント

`--advertise-client-urls`は`'http://0.0.0.0:2379,http://0.0.0.0:4001'`ではなく`http://192.168.33.40:2379,http://192.168.33.40:4001`にする必要がある。


# Swarm Clusterの構築

## swarm-masterを作る

**manager**

```
$ docker-machine create \
   --driver generic --generic-ip-address=192.168.33.10 \
   --generic-ssh-user vagrant --generic-ssh-key ~/.ssh/id_rsa \
   --swarm --swarm-master \
   --swarm-discovery="etcd://$(docker-machine ip kvstore):2379/swarm" \
   --engine-opt="cluster-store=etcd://$(docker-machine ip kvstore):2379" \
   --engine-opt="cluster-advertise=eth1:2376" \
   manager
```

## swarm-nodeを作る

**node1**

```
$ docker-machine create \
   --driver generic --generic-ip-address=192.168.33.20 \
   --generic-ssh-user vagrant --generic-ssh-key ~/.ssh/id_rsa \
   --swarm \
   --swarm-discovery="etcd://$(docker-machine ip kvstore):2379/swarm" \
   node1
```

**node2**

```
$ docker-machine create \
   --driver generic --generic-ip-address=192.168.33.30 \
   --generic-ssh-user vagrant --generic-ssh-key ~/.ssh/id_rsa \
   --swarm \
   --swarm-discovery="etcd://$(docker-machine ip kvstore):2379/swarm" \
   node2
```

# 動作確認

**etcd**

etcdのAPIで状態を確認する。
manager, node1, node2が登録されている。

```
{
  "node": {
    "nodes": [
      {
        "createdIndex": 4,
        "modifiedIndex": 4,
        "nodes": [
          {
            "createdIndex": 4,
            "modifiedIndex": 4,
            "nodes": [
              {
                "createdIndex": 4,
                "modifiedIndex": 4,
                "nodes": [
                  {
                    "createdIndex": 4,
                    "modifiedIndex": 4,
                    "nodes": [
                      {
                        "createdIndex": 100,
                        "modifiedIndex": 100,
                        "ttl": 134,
                        "expiration": "2016-11-23T11:45:51.259602538Z",
                        "value": "192.168.33.10:2376",
                        "key": "/swarm/docker/swarm/nodes/192.168.33.10:2376"
                      },
                      {
                        "createdIndex": 102,
                        "modifiedIndex": 102,
                        "ttl": 168,
                        "expiration": "2016-11-23T11:46:25.014658247Z",
                        "value": "192.168.33.20:2376",
                        "key": "/swarm/docker/swarm/nodes/192.168.33.20:2376"
                      },
                      {
                        "createdIndex": 101,
                        "modifiedIndex": 101,
                        "ttl": 137,
                        "expiration": "2016-11-23T11:45:53.997167245Z",
                        "value": "192.168.33.30:2376",
                        "key": "/swarm/docker/swarm/nodes/192.168.33.30:2376"
                      }
                    ],
                    "dir": true,
                    "key": "/swarm/docker/swarm/nodes"
                  }
                ],
                "dir": true,
                "key": "/swarm/docker/swarm"
              }
            ],
            "dir": true,
            "key": "/swarm/docker"
          }
        ],
        "dir": true,
        "key": "/swarm"
      }
    ],
    "dir": true
  },
  "action": "get"
}
```

**cluster**

docker-machineで状態確認。
`192.168.33.10`をmasterにしてRunningになっている。

```
$ docker-machine ls
NAME      ACTIVE      DRIVER    STATE     URL                        SWARM              DOCKER    ERRORS
kvstore   -           generic   Running   tcp://192.168.33.40:2376                      v1.12.3
manager   * (swarm)   generic   Running   tcp://192.168.33.10:2376   manager (master)   v1.12.3
node1     -           generic   Running   tcp://192.168.33.20:2376   manager            v1.12.3
node2     -           generic   Running   tcp://192.168.33.30:2376   manager            v1.12.3
```

swarm-managerをdockerホストにしてdockerコマンドを起動。
クラスタの情報が確認できる。

```
$ eval $(docker-machine env --swarm manager)
$ docker info
...
Nodes: 3
 manager: 192.168.33.10:2376
  └ ID: 7MOY:Q7KV:KRXU:ORDJ:7LNB:LGSM:GT5H:N4J3:BMC7:WC7C:RMEA:UPA6
  └ Status: Healthy
  └ Containers: 2 (2 Running, 0 Paused, 0 Stopped)
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 514.5 MiB
  └ Labels: kernelversion=3.13.0-95-generic, operatingsystem=Ubuntu 14.04.5 LTS, provider=generic, storagedriver=aufs
  └ UpdatedAt: 2016-11-23T11:49:04Z
  └ ServerVersion: 1.12.3
 node1: 192.168.33.20:2376
  └ ID: C6GI:OKAT:H2LH:VCBU:OF65:YWST:QVQ7:X6RP:6JY5:2CQK:4PVM:LVCZ
  └ Status: Healthy
  └ Containers: 1 (1 Running, 0 Paused, 0 Stopped)
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 514.5 MiB
  └ Labels: kernelversion=3.13.0-95-generic, operatingsystem=Ubuntu 14.04.5 LTS, provider=generic, storagedriver=aufs
  └ UpdatedAt: 2016-11-23T11:49:16Z
  └ ServerVersion: 1.12.3
 node2: 192.168.33.30:2376
  └ ID: IJAI:XSRL:3T5S:5XEF:AUVZ:TLYM:RVAH:MYAW:Q43J:CVCF:J5JH:BGZA
  └ Status: Healthy
  └ Containers: 1 (1 Running, 0 Paused, 0 Stopped)
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 514.5 MiB
  └ Labels: kernelversion=3.13.0-95-generic, operatingsystem=Ubuntu 14.04.5 LTS, provider=generic, storagedriver=aufs
  └ UpdatedAt: 2016-11-23T11:49:16Z
  └ ServerVersion: 1.12.3
...
```

# 適当なコンテナを立ててみる

ubuntuをpullして10個ぐらい立てる。

```
$ docker pull ubuntu
$ docker run -d ubuntu tail -f /dev/nul
.....
$ docker run -d ubuntu tail -f /dev/nul
$ docker ps
CONTAINER ID        IMAGE               COMMAND               CREATED             STATUS              PORTS               NAMES
adb3b7d5e716        ubuntu              "tail -f /dev/null"   4 seconds ago       Up 3 seconds                            node1/cocky_stallman
dcd7f4f99864        ubuntu              "tail -f /dev/null"   22 seconds ago      Up 21 seconds                           node2/furious_payne
734878f86e59        ubuntu              "tail -f /dev/null"   23 seconds ago      Up 22 seconds                           node1/small_ardinghelli
0d6099e7cc05        ubuntu              "tail -f /dev/null"   25 seconds ago      Up 24 seconds                           node2/nauseous_hoover
f06a02774421        ubuntu              "tail -f /dev/null"   26 seconds ago      Up 25 seconds                           node1/big_kare
b24d42d433f4        ubuntu              "tail -f /dev/null"   29 seconds ago      Up 28 seconds                           node2/hungry_engelbart
7daa0be7d630        ubuntu              "tail -f /dev/null"   54 seconds ago      Up 52 seconds                           node1/stupefied_joliot
9ddaf2e88b93        ubuntu              "tail -f /dev/null"   2 seconds ago       Up 1 seconds                            manager/elated_northcutt
66b4da48e09d        ubuntu              "tail -f /dev/null"   24 seconds ago      Up 23 seconds                           manager/elated_stonebraker
bb97b9a39644        ubuntu              "tail -f /dev/null"   27 seconds ago      Up 26 seconds                           manager/determined_lichterman
```

デフォルトのstrategyが`spread`なのでmanager, node1, node2にそれぞれ均等にコンテナが立っている。

# 参考

https://www.linux.com/learn/how-use-docker-machine-create-swarm-cluster

https://docs.docker.com/swarm/provision-with-machine/

http://qiita.com/ohhara_shiojiri/items/47e4111a81431c7cb3a7

https://docs.docker.com/swarm/install-w-machine/

http://blog.scottlowe.org/2015/04/19/running-etcd-backed-docker-swarm-cluster/

https://docs.docker.com/machine/reference/create/#/specifying-docker-swarm-options-for-the-created-machine

http://trainingdevops.com/insights-and-tutorials/building-cloud-with-etcd-docker-swarm-and-overlay-network

https://docs.docker.com/engine/userguide/networking/get-started-overlay/#/create-a-swarm-cluster

http://stackoverflow.com/questions/34892377/the-cluster-store-and-cluster-advertise-dont-work

https://github.com/docker/swarm/issues/1710

https://github.com/kubernetes/kubernetes/issues/19235

https://docs.docker.com/swarm/discovery/