マルチホスト上での、dockerコンテナ間ネットワーク環境整備メモ（Overlay Networking w/ Etcd）

マルチホスト上での、dockerコンテナ間ネットワーク環境整備メモっておきます。

- dockerコンテナ間ネットワーク情報は、事前に、KeyValueストア"etcd"で共有される
- dockerコンテナ間でのデータ通信は、VXLANでカプセル化される

という仕組みになっている点が、特徴になります。
ちなみに、今回の環境整備にあたり、こちらの[Web記事"Multi-Host Overlay Networking with Etcd"](http://docker-k8s-lab.readthedocs.io/en/latest/docker/docker-etcd.html)を参考にしています。

## ■ Ubuntu環境準備
今回、構築したdockerコンテナ間ネットワークは、次のようなUbuntu構成（３台）としました。

![docker network.001.jpeg](https://qiita-image-store.s3.amazonaws.com/0/103595/4f1fbf68-d4a9-1dad-62bd-b44a4ca31990.jpeg)

### (1) Ubuntuサーバ環境
```
tsubo@docker-node1:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.5 LTS"
```

### (2) Docker for ubuntu インストール
- [Docker for ubuntu](https://docs.docker.com/engine/installation/linux/ubuntu/) をインストールする

```
tsubo@docker-node1:~$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:56 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:21 2018
  OS/Arch:          linux/amd64
  Experimental:     false
```
- Ubuntuサーバ側ネットワーク構成を確認しておく

```
tsubo@docker-node1:~$ ifconfig
docker0   Link encap:Ethernet  HWaddr 02:42:4b:53:0c:44  
          inet addr:172.17.0.1  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ens2      Link encap:Ethernet  HWaddr 52:54:00:2a:d7:b1  
          inet addr:192.168.100.201  Bcast:192.168.100.255  Mask:255.255.255.0
          inet6 addr: 240b:11:53a0:400:5054:ff:fe2a:d7b1/64 Scope:Global
          inet6 addr: fe80::5054:ff:fe2a:d7b1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3009 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3012 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:248266 (248.2 KB)  TX bytes:265342 (265.3 KB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:285 errors:0 dropped:0 overruns:0 frame:0
          TX packets:285 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:18952 (18.9 KB)  TX bytes:18952 (18.9 KB)
```

## ■ etcd環境構築
dockerコンテナ間ネットワーク情報は、次のようなUbuntu構成（３台）に、KeyValueストア"etcd"を配備して共有する形態となります。

![docker network.002.jpeg](https://qiita-image-store.s3.amazonaws.com/0/103595/a67f9e30-e4ba-2c24-de18-4d3af583b19c.jpeg)

### (1) etcd環境整備
[etcd3 multi-node cluster](https://github.com/coreos/etcd/tree/master/contrib/systemd/etcd3-multinode)の構築手順を参考に、etcd環境をセットアップする。
以下、 **docker-node1** でのetcd環境整備の作業履歴メモ

- etcdデータ保管ディレクトリの作成

```
tsubo@docker-node1:~$ sudo mkdir -p /var/lib/etcd 
tsubo@docker-node1:~$ sudo chown -R root:$(whoami) /var/lib/etcd
tsubo@docker-node1:~$ sudo chmod -R a+rwx /var/lib/etcd
```
- etcdソフトウェアのダウンロード＆配備

```
tsubo@docker-node1:~$ wget https://github.com/coreos/etcd/releases/download/v3.3.9/etcd-v3.3.9-linux-amd64.tar.gz
tsubo@docker-node1:~$ gzip -d etcd-v3.3.9-linux-amd64.tar.gz 
tsubo@docker-node1:~$ tar xvf etcd-v3.3.9-linux-amd64.tar
tsubo@docker-node1:~$ cd etcd-v3.3.9-linux-amd64/
tsubo@docker-node1:~/etcd-v3.3.9-linux-amd64$ ls -l
total 33992
drwxr-xr-x 11 tsubo tsubo     4096 Jul 25 02:13 Documentation
-rw-r--r--  1 tsubo tsubo    38864 Jul 25 02:13 README-etcdctl.md
-rw-r--r--  1 tsubo tsubo     7262 Jul 25 02:13 README.md
-rw-r--r--  1 tsubo tsubo     7855 Jul 25 02:13 READMEv2-etcdctl.md
-rwxr-xr-x  1 tsubo tsubo 18934016 Jul 25 02:13 etcd
-rwxr-xr-x  1 tsubo tsubo 15809280 Jul 25 02:13 etcdctl

tsubo@docker-node1:~/etcd-v3.3.9-linux-amd64$ sudo cp etcd /usr/local/sbin/
tsubo@docker-node1:~/etcd-v3.3.9-linux-amd64$ sudo cp etcdctl /usr/local/sbin/
```
- etcd起動用systemdファイルの準備

```
tsubo@docker-node1:~$ IP_1="192.168.100.201"
tsubo@docker-node1:~$ IP_2="192.168.100.202"
tsubo@docker-node1:~$ IP_3="192.168.100.203"
```
```
tsubo@docker-node1:~$ cat > /tmp/my-etcd-1.service <<EOF
> [Unit]
> Description=etcd
> Documentation=https://github.com/coreos/etcd
> Conflicts=etcd.service
> Conflicts=etcd2.service
> 
> [Service]
> Type=notify
> Restart=always
> RestartSec=5s
> LimitNOFILE=40000
> TimeoutStartSec=0
> 
> ExecStart=/usr/local/sbin/etcd --name my-etcd-1 \
>     --data-dir /var/lib/etcd \
>     --listen-client-urls http://${IP_1}:2379 \
>     --advertise-client-urls http://${IP_1}:2379 \
>     --listen-peer-urls http://${IP_1}:2380 \
>     --initial-advertise-peer-urls http://${IP_1}:2380 \
>     --initial-cluster my-etcd-1=http://${IP_1}:2380,my-etcd-2=http://${IP_2}:2380,my-etcd-3=http://${IP_3}:2380 \
>     --initial-cluster-token my-etcd-token \
>     --initial-cluster-state new
> 
> [Install]
> WantedBy=multi-user.target
> EOF
```
```
tsubo@docker-node1:~$ sudo mv /tmp/my-etcd-1.service /etc/systemd/system/my-etcd-1.service
```
- etcd起動用systemdファイルの適用

```
tsubo@docker-node1:~$ sudo systemctl daemon-reload
tsubo@docker-node1:~$ sudo systemctl enable my-etcd-1.service
tsubo@docker-node1:~$ sudo systemctl start my-etcd-1.service
```
### (2) etcdクラスタ構成確認
**docker-node1**, **docker-node2**, **docker-node3**全てにおいて、etcd環境整備を行ったのちに、etcdクラスタ構成を確認しておく。

```
tsubo@docker-node1:~$ etcdctl -C http://192.168.100.201:2379 cluster-health
member 39b9111a3bc023ca is healthy: got healthy result from http://192.168.100.201:2379
member 98c8df2619e159f6 is healthy: got healthy result from http://192.168.100.202:2379
member e322c953b6b14c07 is healthy: got healthy result from http://192.168.100.203:2379
cluster is healthy
```
## ■ dockerコンテナ間ネットワーク（Overlay Networking）環境整備
dockerコンテナ間ネットワーク（Overlay Networking）が有効になるように、Dockerエンジンの起動オプションを変更します。

![docker network.003.jpeg](https://qiita-image-store.s3.amazonaws.com/0/103595/46d20bce-57bd-1d8c-8c0b-dd6f534d1b3d.jpeg)

### (1) **docker-node1**での、docker起動オプション設定
- docker-engine起動ファイル場所の確認

```
tsubo@docker-node1:~$ sudo service docker status
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-09-27 08:44:20 JST; 1h 10min ago
     Docs: https://docs.docker.com
 Main PID: 640 (dockerd)
    Tasks: 32
   Memory: 179.5M
      CPU: 20.955s
   CGroup: /system.slice/docker.service
           ├─640 /usr/bin/dockerd -H fd://
           └─752 docker-containerd --config /var/run/docker/containerd/containerd.toml

Sep 27 08:44:19 docker-node1 dockerd[640]: time="2018-09-27T08:44:19.315284175+09:00" lev
Sep 27 08:44:19 docker-node1 dockerd[640]: time="2018-09-27T08:44:19.315331295+09:00" lev
Sep 27 08:44:19 docker-node1 dockerd[640]: time="2018-09-27T08:44:19.315661102+09:00" lev
Sep 27 08:44:19 docker-node1 dockerd[640]: time="2018-09-27T08:44:19.315687104+09:00" lev
Sep 27 08:44:19 docker-node1 dockerd[640]: time="2018-09-27T08:44:19.837864761+09:00" lev
Sep 27 08:44:19 docker-node1 dockerd[640]: time="2018-09-27T08:44:19.979038913+09:00" lev
Sep 27 08:44:20 docker-node1 dockerd[640]: time="2018-09-27T08:44:20.016485828+09:00" lev
Sep 27 08:44:20 docker-node1 dockerd[640]: time="2018-09-27T08:44:20.016885790+09:00" lev
Sep 27 08:44:20 docker-node1 dockerd[640]: time="2018-09-27T08:44:20.070150577+09:00" lev
Sep 27 08:44:20 docker-node1 systemd[1]: Started Docker Application Container Engine.
```

- docker-engine起動ファイルの編集

```
tsubo@docker-node1:~$ sudo vi /lib/systemd/system/docker.service
```
```text:/lib/systemd/system/docker.service
...(snip)
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --cluster-store=etcd://192.168.100.201:2379 --cluster-advertise=192.168.100.201:2376
...(snip)
```
- docker-engine起動ファイルの適用

```
tsubo@docker-node1:~$ sudo systemctl daemon-reload
```
- docker-engine起動

```
tsubo@docker-node1:~$ sudo service docker start
```
- docker-engine起動結果の確認

```
tsubo@docker-node1:~$ sudo service docker status
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-09-27 09:56:25 JST; 8s ago
     Docs: https://docs.docker.com
 Main PID: 1668 (dockerd)
    Tasks: 27
   Memory: 174.4M
      CPU: 948ms
   CGroup: /system.slice/docker.service
           ├─1668 /usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock -
           └─1678 docker-containerd --config /var/run/docker/containerd/containerd.toml

Sep 27 09:56:23 docker-node1 dockerd[1668]: time="2018-09-27T09:56:23.887721158+09:00" le
Sep 27 09:56:23 docker-node1 dockerd[1668]: time="2018-09-27T09:56:23.887777496+09:00" le
Sep 27 09:56:24 docker-node1 dockerd[1668]: time="2018-09-27T09:56:24.439637282+09:00" le
Sep 27 09:56:25 docker-node1 dockerd[1668]: time="2018-09-27T09:56:25.563648097+09:00" le
Sep 27 09:56:25 docker-node1 dockerd[1668]: time="2018-09-27T09:56:25.716963378+09:00" le
Sep 27 09:56:25 docker-node1 dockerd[1668]: time="2018-09-27T09:56:25.819701288+09:00" le
Sep 27 09:56:25 docker-node1 dockerd[1668]: time="2018-09-27T09:56:25.819793818+09:00" le
Sep 27 09:56:25 docker-node1 dockerd[1668]: time="2018-09-27T09:56:25.852983239+09:00" le
Sep 27 09:56:25 docker-node1 dockerd[1668]: time="2018-09-27T09:56:25.852998881+09:00" le
Sep 27 09:56:25 docker-node1 systemd[1]: Started Docker Application Container Engine.
```
### (2) **docker-node2**での、docker起動オプション設定
- docker engine停止

```
tsubo@docker-node2:~$ sudo service docker stop
```
- docker-engine起動ファイルの編集

```
tsubo@docker-node2:~$ sudo vi /lib/systemd/system/docker.service
```
```text:/lib/systemd/system/docker.service
...(snip)
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --cluster-store=etcd://192.168.100.202:2379 --cluster-advertise=192.168.100.202:2376
...(snip)
```
- docker-engine起動ファイルの適用

```
tsubo@docker-node2:~$ sudo systemctl daemon-reload
```
- docker-engine起動

```
tsubo@docker-node2:~$ sudo service docker start
```

### (3) **docker-node3**での、docker起動オプション設定
- docker engine停止

```
tsubo@docker-node3:~$ sudo service docker stop
```
- docker-engine起動ファイルの編集

```
tsubo@docker-node3:~$ sudo vi /lib/systemd/system/docker.service
```
```text:/lib/systemd/system/docker.service
...(snip)
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --cluster-store=etcd://192.168.100.203:2379 --cluster-advertise=192.168.100.203:2376
...(snip)
```
- docker-engine起動ファイルの適用

```
tsubo@docker-node3:~$ sudo systemctl daemon-reload
```
- docker-engine起動

```
tsubo@docker-node3:~$ sudo service docker start
```

### (4) 初期ネットワーク構成の確認
- 初期ネットワーク構成の確認

```
tsubo@docker-node1:~$ docker network ls
NETWORK ID          NAME                               DRIVER              SCOPE
722a87d53691        bridge                             bridge              local
f13e8a2111c1        docker_gwbridge                    bridge              local
4399a0c0eabd        host                               host                local
4a12154fa3db        none                               null                local
```
- docker_gwbridgeの構成確認

```
tsubo@docker-node1:~$ docker inspect docker_gwbridge
[
    {
        "Name": "docker_gwbridge",
        "Id": "f13e8a2111c137ae7d08ddc501ce12e5747281329ef5ee6d6413bb3a37df1f34",
        "Created": "2017-12-08T10:00:23.303426747+09:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.enable_icc": "false",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.name": "docker_gwbridge"
        },
        "Labels": {}
    }
]
```

## ■ dockerコンテナ間ネットワークを作成してみる
dockerコンテナ間ネットワーク（Overlay Networking）上に、Dockerコンテナを配備します。

![docker network.004.jpeg](https://qiita-image-store.s3.amazonaws.com/0/103595/ba24a397-7b15-0904-eef4-506850066d3f.jpeg)

### (1) Overlay Networking"192.168.0.0/24"を作成する
- **docker-node1**で、dockerコンテナ間ネットワーク" **demo** "を作成

```
tsubo@docker-node1:~$ docker network create -d overlay --subnet 192.168.0.0/24 demo
ac084ea33128eea0d625514a1242c9d0739c9e5e9ad37b6303dd410c0be361b0
```
- **demo** の構成確認

```
tsubo@docker-node1:~$ docker inspect demo
[
    {
        "Name": "demo",
        "Id": "ac084ea33128eea0d625514a1242c9d0739c9e5e9ad37b6303dd410c0be361b0",
        "Created": "2018-09-27T10:18:32.358732946+09:00",
        "Scope": "global",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.0.0/24"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {}
    }
]
```
### (2) **docker-node1**で、dockerコンテナを作成する
- dockerコンテナを作成する

```
tsubo@docker-node1:~$ docker run -d --name ubuntu01 -h ubuntu01 --net demo -itd ubuntu:14.04 bash
c2bffc1ba23aebd87bc35fb70868316eccc4cbfd9e26ef3595630bce62b169d6
```
- dockerコンテナ上で、ネットワーク構成を確認する

```
tsubo@docker-node1:~$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
c2bffc1ba23a        ubuntu:14.04        "bash"              26 seconds ago      Up 23 seconds                           ubuntu01
```
```
tsubo@docker-node1:~$ docker exec -it ubuntu01 bash
root@ubuntu01:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:c0:a8:00:02  
          inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth1      Link encap:Ethernet  HWaddr 02:42:ac:13:00:02  
          inet addr:172.18.0.2  Bcast:172.19.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1296 (1.2 KB)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@ubuntu01:/# exit
```

### (3) **docker-node2**で、dockerコンテナを作成する
- dockerコンテナを作成する

```
tsubo@docker-node2:~$ docker run -d --name ubuntu02 -h ubuntu02 --net demo -itd ubuntu:14.04 bash
dd78e85953aa836fd851652ee5d738df295f2cc5388f19338e8f28f1cb746cca
```
- dockerコンテナ上で、ネットワーク構成を確認する

```
tsubo@docker-node2:~$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
dd78e85953aa        ubuntu:14.04        "bash"              18 seconds ago      Up 15 seconds                           ubuntu02
```
```
tsubo@docker-node2:~$ docker exec -it ubuntu02 bash
root@ubuntu02:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:c0:a8:00:03  
          inet addr:192.168.0.3  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth1      Link encap:Ethernet  HWaddr 02:42:ac:13:00:02  
          inet addr:172.18.0.2  Bcast:172.19.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1296 (1.2 KB)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@ubuntu02:/# exit
```
### (4) **docker-node3**で、dockerコンテナを作成する
- dockerコンテナを作成する

```
tsubo@docker-node3:~$ docker run -d --name ubuntu03 -h ubuntu03 --net demo -itd ubuntu:14.04 bash
85b28e5d18446e9ac65d6af3b939fa1e7e940e03173982febb0a976b5b830389
```
- dockerコンテナ上で、ネットワーク構成を確認する

```
tsubo@docker-node3:~$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
85b28e5d1844        ubuntu:14.04        "bash"              17 seconds ago      Up 13 seconds                           ubuntu03
```
```
tsubo@docker-node3:~$ docker exec -it ubuntu03 bash
root@ubuntu03:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 02:42:c0:a8:00:04  
          inet addr:192.168.0.4  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth1      Link encap:Ethernet  HWaddr 02:42:ac:12:00:02  
          inet addr:172.18.0.2  Bcast:172.18.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1296 (1.2 KB)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

root@ubuntu03:/# exit
```
## ■ dockerコンテナ間で通信してみる
### (1) **docker-node1上の"ubuntu01"**から、**docker-node3上の"ubuntu03"**へ、pingを打ってみる

```
tsubo@docker-node1:~$ docker exec -it ubuntu01 bash
root@ubuntu01:/# ping 192.168.0.4
PING 192.168.0.4 (192.168.0.4) 56(84) bytes of data.
64 bytes from 192.168.0.4: icmp_seq=1 ttl=64 time=2.19 ms
64 bytes from 192.168.0.4: icmp_seq=2 ttl=64 time=1.51 ms
64 bytes from 192.168.0.4: icmp_seq=3 ttl=64 time=1.21 ms
64 bytes from 192.168.0.4: icmp_seq=4 ttl=64 time=1.14 ms
^C
--- 192.168.0.4 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 1.141/1.515/2.195/0.416 ms
```
### (2) dockerコンテナ間のデータ通信をキャプチャしてみる

```
root@ubuntu01:/# arp -a
ubuntu03.demo (192.168.0.4) at 02:42:c0:a8:00:04 [ether] on eth0
```
![wireshark.png](https://qiita-image-store.s3.amazonaws.com/0/103595/df29b415-5e77-c583-5b00-8322a51a3f8d.png)


### (3)dockerコンテナ作成後の"**demo**"の構成状態を確認しておく
```
tsubo@docker-node1:~$ docker inspect demo
[
    {
        "Name": "demo",
        "Id": "ac084ea33128eea0d625514a1242c9d0739c9e5e9ad37b6303dd410c0be361b0",
        "Created": "2018-09-27T10:18:32.358732946+09:00",
        "Scope": "global",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.0.0/24"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "c2bffc1ba23aebd87bc35fb70868316eccc4cbfd9e26ef3595630bce62b169d6": {
                "Name": "ubuntu01",
                "EndpointID": "1ae0eb7096dd6e679af24ec2cdfd36b6f3d58f552d7079c8da3999c791315b8d",
                "MacAddress": "02:42:c0:a8:00:02",
                "IPv4Address": "192.168.0.2/24",
                "IPv6Address": ""
            },
            "ep-5afb3034fc3ea5d72283ad66af5eab2894768903d1d1b4e32de12cb9b4c5dcba": {
                "Name": "ubuntu02",
                "EndpointID": "5afb3034fc3ea5d72283ad66af5eab2894768903d1d1b4e32de12cb9b4c5dcba",
                "MacAddress": "02:42:c0:a8:00:03",
                "IPv4Address": "192.168.0.3/24",
                "IPv6Address": ""
            },
            "ep-a0b44a7c9bbd93758a5471ada9425afdadd4e2e327a8d7be48ccf4d6df6e7be9": {
                "Name": "ubuntu03",
                "EndpointID": "a0b44a7c9bbd93758a5471ada9425afdadd4e2e327a8d7be48ccf4d6df6e7be9",
                "MacAddress": "02:42:c0:a8:00:04",
                "IPv4Address": "192.168.0.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]
```

ネットワーク"**demo**"に、dockerコンテナ３台が収容されている様子が確認できました。