Docker の ネットワークを開発用と確認用にいい感じに分ける

# Docker の ネットワークを開発用と確認用にいい感じに分ける

## やりたいこと

開発は、VirtualBox のホストオンリーアダプタのネットワーク帯を使用し、
ブリッジアダプタ ネットワーク 内の他の人からも確認出来るようにしたい。

## ホストOS環境

### OS

- CentOS Linux release 7.2.1511 (Core)

### ネットワーク

- enp0s3 … VirtualBoxブリッジアダプタ（10.0.17.70/8）
- enp0s8 … VirtualBoxホストオンリーアダプタ（192.168.17.70/24）

## Docker設定

### Docker を最新版にあげる。

バージョンが低いとブリッジ接続が上手くいかなかったのでバージョンを上げました。

```
root@localhost:~# yum remove docker docker-common docker-selinux -y
root@localhost:~# touch /etc/yum.repos.d/docker.repo
root@localhost:~# vim /etc/yum.repos.d/docker.repo
root@localhost:~# cat /etc/yum.repos.d/docker.repo
[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/$releasever/
enabled=0
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg
root@localhost:~# yum install docker-engine --enablerepo=dockerrepo -y
root@localhost:~# docker -v
Docker version 1.12.1, build 23cf638
root@localhost:~# systemctl start docker
root@localhost:~# systemctl enable docker
```

### ネットワーク確認

```
root@localhost:~# ip addr show                                                                                         
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:2a:10:99 brd ff:ff:ff:ff:ff:ff
    inet 10.0.17.70/8 brd 10.0.17.70 scope global enp0s3
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:41:6c:7b brd ff:ff:ff:ff:ff:ff
    inet 192.168.17.70/24 brd 192.168.17.255 scope global enp0s8
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:f2:44:a7:c2 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
```

### bridge-utils インストール

```
root@localhost:~# yum install -y bridge-utils
```

### enp0s8 (192.168.17.0/24) を削除

```
root@localhost:~# ip addr flush dev enp0s8
root@localhost:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:2a:10:99 brd ff:ff:ff:ff:ff:ff
    inet 10.0.17.70/8 brd 10.0.17.70 scope global enp0s3
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:41:6c:7b brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:f2:44:a7:c2 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
```

### Dockerネットワーク host_only と、ホストPCのDocker用ネットワークデバイス docker1 を作成

```
root@localhost:~# docker network create --driver bridge --subnet=192.168.17.0/24 --gateway=192.168.17.70 --opt "com.docker.network.bridge.name"="docker1" host_only
ed7f072fc81b180d64824a6981059dee3c668cf52db984fc37f59a4b758363a4
root@localhost:~# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
3613dc157cec        bridge              bridge              local               
9537f837ef9a        host                host                local               
ed7f072fc81b        host_only           bridge              local               
ed010e616095        none                null                local 
root@localhost:~# docker network inspect host_only
[
    {
        "Name": "host_only",
        "Id": "ed7f072fc81b180d64824a6981059dee3c668cf52db984fc37f59a4b758363a4",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.17.0/24",
                    "Gateway": "192.168.17.70"
                }
            ]
        },
        "Internal": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.name": "docker1"
        },
        "Labels": {}
    }
]
root@localhost:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:2a:10:99 brd ff:ff:ff:ff:ff:ff
    inet 10.0.17.70/8 brd 10.0.17.70 scope global enp0s3
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:41:6c:7b brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:f2:44:a7:c2 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
5: docker1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:f9:b2:25:c6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.17.70/24 scope global docker1
       valid_lft forever preferred_lft forever
```

### docker1 を enp0s8 に紐付ける

```
root@localhost:~# brctl addif docker1 enp0s8
root@localhost:~# brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.02427d9620c8       no             
docker1         8000.0242a4d75f14       no              enp0s8
root@localhost:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:2a:10:99 brd ff:ff:ff:ff:ff:ff
    inet 10.0.17.70/8 brd 10.0.17.70 scope global enp0s3
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master docker1 state UP qlen 1000
    link/ether 08:00:27:41:6c:7b brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:7d:96:20:c8 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
5: docker1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:a4:d7:5f:14 brd ff:ff:ff:ff:ff:ff
    inet 192.168.17.70/24 scope global docker1
       valid_lft forever preferred_lft forever
7: vethc9a28f5@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker1 state UP
    link/ether d2:bb:5b:3f:ae:2f brd ff:ff:ff:ff:ff:ff link-netnsid 0
```

### ブリッジ接続の恒久化

```
root@localhost:~# mkdir /etc/systemd/system/docker.service.d
root@localhost:~# vim /etc/systemd/system/docker.service.d/brctl.conf
root@localhost:~# cat /etc/systemd/system/docker.service.d/brctl.conf
[Service]
ExecStartPost=/bin/sh -c "/usr/sbin/ip addr flush dev enp0s8; /usr/sbin/brctl addif docker1 enp0s8"
ExecStopPost=/bin/sh -c "/usr/sbin/brctl delif docker1 enp0s8"

root@localhost:~# systemctl daemon-reload
```

### Dockerコンテナを作ってみる

```
root@localhost:~# docker run --name sample --net=host_only --ip=192.168.17.71 -d nginx
root@localhost:~# docker inspect sample
[
    {
        "Id": "c474d567469bd13b14b1f1bc563e755bbaa7ff6d167114730f83b61252678002",
        "Created": "2016-09-20T06:46:22.150351555Z",
        "Path": "nginx",
        "Args": [
            "-g",
            "daemon off;"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 4106,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2016-09-20T06:46:24.066430192Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:4a88d06e26f40b14f702c65c5915bd8e18600520352fa49ece8b5052db0e76fd",
        "ResolvConfPath": "/var/lib/docker/containers/c474d567469bd13b14b1f1bc563e755bbaa7ff6d167114730f83b61252678002/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/c474d567469bd13b14b1f1bc563e755bbaa7ff6d167114730f83b61252678002/hostname",
        "HostsPath": "/var/lib/docker/containers/c474d567469bd13b14b1f1bc563e755bbaa7ff6d167114730f83b61252678002/hosts",
        "LogPath": "/var/lib/docker/containers/c474d567469bd13b14b1f1bc563e755bbaa7ff6d167114730f83b61252678002/c474d567469bd13b14b1f1bc563e755bbaa7ff6d167114730f83b61252678002-json.log",
        "Name": "/sample",
        "RestartCount": 0,
        "Driver": "devicemapper",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "host_only",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "devicemapper",
            "Data": {
                "DeviceId": "114",
                "DeviceName": "docker-253:0-300537-57578930fbc2939efc566ca90b5c275b3029af16746c92c796236fb403b2c3fa",
                "DeviceSize": "10737418240"
            }
        },
        "Mounts": [],
        "Config": {
            "Hostname": "c474d567469b",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "443/tcp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.11.4-1~jessie"
            ],
            "Cmd": [
                "nginx",
                "-g",
                "daemon off;"
            ],
            "Image": "nginx",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "cf385e3ad23f7874d3eadafd7df79bc75cde278596ef6af96b2633bce938f5ad",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "443/tcp": null,
                "80/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/cf385e3ad23f",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host_only": {
                    "IPAMConfig": {
                        "IPv4Address": "192.168.17.71"
                    },
                    "Links": null,
                    "Aliases": [
                        "c474d567469b"
                    ],
                    "NetworkID": "ed7f072fc81b180d64824a6981059dee3c668cf52db984fc37f59a4b758363a4",
                    "EndpointID": "085441d37c54830358b47ca6800ed4390421fe61bae6cde591aa1a1b7db21297",
                    "Gateway": "192.168.17.70",
                    "IPAddress": "192.168.17.71",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:c0:a8:11:48"
                }
            }
        }
    }
]
```

これで、開発環境からは固定IP（192.168.17.71）でアクセス出来るようになりました。

## ネットワーク設定

### IPエイリアス追加

```
root@localhost:~# nmcli connection mod enp0s3 +ipv4.addresses 10.0.17.71/8
root@localhost:~# nmcli connection down enp0s3 ; nmcli connection up enp0s3
root@localhost:~# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:2a:10:99 brd ff:ff:ff:ff:ff:ff
    inet 10.0.17.70/8 brd 10.255.255.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet 10.0.17.71/8 brd 10.255.255.255 scope global secondary enp0s3
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master docker1 state UP qlen 1000
    link/ether 08:00:27:41:6c:7b brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:fb:72:45:62 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
5: docker1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:06:63:91:c9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.17.70/24 scope global docker1
       valid_lft forever preferred_lft forever
```

### Firewalld を切る

Docker のネットワークとコンフリクトを起こすので、
Firewalld は切る。

```
root@localhost:~# systemctl stop firewalld
root@localhost:~# systemctl disable firewalld
```

### iptables 設定

10.0.17.71 にアクセスがあった場合、
192.168.17.71 にルーティングするようにする。

#### 現状確認

```
root@localhost:~# iptables -t nat -L -n --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
2    MASQUERADE  all  --  192.168.17.0/24      0.0.0.0/0           

Chain DOCKER (2 references)
num  target     prot opt source               destination         
1    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
2    RETURN     all  --  0.0.0.0/0            0.0.0.0/0
```

#### 10.0.17.71 への全てのアクセスを 192.168.17.71 に転送する場合

```
root@localhost:~# iptables -t nat -A PREROUTING -d 10.0.17.72/32 -p tcp -m tcp -j DNAT --to-destination 192.168.17.71
root@localhost:~# iptables -t nat -L -n --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
2    DNAT       tcp  --  0.0.0.0/0            10.0.17.72           tcp to:192.168.17.71

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
2    MASQUERADE  all  --  192.168.17.0/24      0.0.0.0/0           

Chain DOCKER (2 references)
num  target     prot opt source               destination         
1    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
2    RETURN     all  --  0.0.0.0/0            0.0.0.0/0
```

#### 10.0.17.71 への80番へのアクセスを 192.168.17.71 の8080番に転送する場合

```
root@localhost:~# iptables -t nat -A PREROUTING -d 10.0.17.71/32 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.17.71:8080
root@localhost:~# iptables -t nat -L -n --line-numbers
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
2    DNAT       tcp  --  0.0.0.0/0            10.0.17.71           tcp dpt:80 to:192.168.17.71:8080

Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
num  target     prot opt source               destination         
1    MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
2    MASQUERADE  all  --  192.168.17.0/24      0.0.0.0/0           

Chain DOCKER (2 references)
num  target     prot opt source               destination         
1    RETURN     all  --  0.0.0.0/0            0.0.0.0/0           
2    RETURN     all  --  0.0.0.0/0            0.0.0.0/0
```

#### iptables の設定を保存

```
root@localhost:~# iptables-save > /etc/sysconfig/iptables
```

### iptables, docker の再起動（おまじない）

```
root@localhost:~# systemctl restart iptables docker
```

これで、ホストオンリー配下に立てた dockerコンテナ に、
ブリッジアダプタ ネットワーク 内の他の人からもアクセスすることが可能です。

あとは、コレをコマンドとか（pipeworks みたいに）で 簡単に出来ればいいなぁ...（　= =）ﾄｵｲﾒ