Play-with-Docker環境をAWSで自前構築

# 前置き
https://labs.play-with-docker.com
↑これのサーバ周りを自前で用意しようという趣旨。
[AWSでやるメリット]
・dockerログインが不要
・負荷対策ができる

ハンズオン実施の際、地味に「dockerコマンドが打てる」までにハードルがあり冒頭でグダつきがちだが、これならブラウザで動く！勝利！

※以下を参考にしたら上手くいかず、「差し当たって動く」を目標に弄っていったものです。機能十全ではない可能性がかなりあります。
https://github.com/play-with-docker/play-with-docker/wiki/Deploying-PWD-in-your-own-infrastructure
# ・AWS環境
```
Route53
 |
ALB
 |
EC2
```
## Route53

|Name|Type|Value|Comment|
|:--:|:--:|:--:|:--:|
|example.com|A|{ALB}||
|*.direct.example.com|A|{ALB}|playground内で動くvhost用|

## ALB
- TargetGroup: ポート80でEC2へ
- SecurityGroup: ブラウザアクセス元IPからのポート80と443を許可

## EC2
AMI: [ECS-optimized]
  ・devicemapper storageの設定が済んでいるのが主な理由。
今回は `amzn-ami-2018.03.h-amazon-ecs-optimized (ami-0edf19001c48838c7)` を使用。
Type: 最低でもt2.medium。t2.smallだと一人で使うにも重い。
SecurityGroup:
  ・ローカルからのssh(22)
  ・ALBからの80
  いろんな通信はhaproxyがよしなに振り分けてくれている様子。

# ・EC2プロビジョニング
ec2-userで操作。
###　カーネルモジュールのロード
所定の場所にシェルスクリプトを放り込む。

```/etc/sysconfig/modules/play-with-docker.modules
#!/bin/bash
# play-with-dockerに必要なカーネルモジュールの起動時ロード
/sbin/modprobe overlay
/sbin/modprobe xt_ipvs
```
```
sudo chmod 755 /etc/sysconfig/modules/play-with-docker.modules
```
##　ipv4確認
https://qiita.com/masahixixi/items/14630b501d8fb8aee477
起動時に`/etc/sysctl.conf`から設定ロードするとのことで、書き換える。

```
sudo cat /etc/sysctl.conf | grep net.ipv4.ip_forward

net.ipv4.ip_forward = 1
```

`sysctl -p` での反映は(あとでサーバ再起動するので)省略
`sysctl -w` 何某は暫定変更なので不採用。

## docker-compose導入
AMIとの兼ね合いで、dockerは最初から入っている。

```
sudo curl -L "https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
```
`1.22.0`の部分はバージョン指定。
## golang関連
```
# go導入
sudo yum install go
# パッケージ管理dep導入
go get -u github.com/golang/dep/cmd/dep
```

・gopath
環境変数+ec2-userとして諸々実行予定なので、
`/home/ec2-user/.bash_profile`
ここに以下を追記。

```
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin
```

## play-with-docker本体のgit clone
gitがgoの依存物として一緒に入るので引き続き以下を実施

```
mkdir play-with-docker;cd $_
git clone https://github.com/play-with-docker/play-with-docker.git
```
---
以降はレポジトリ内での作業。

```
cd play-with-docker
pwd

/home/ec2-user/go/src/github.com/play-with-docker/play-with-docker/
```

## Gopkg.toml の修正
https://dev.classmethod.jp/go/dep/
depの実行前に数カ所を変更。

・docker/distribution
  -> constraintからoverrideに
  -> versionからbranch masterに

・googollee/go-socket.io
・zbindenren/negroni-prometheus
  -> constraintからoverrideに

```
$ diff -u Gopkg.toml Gopkg.toml.bak
--- Gopkg.toml	2018-11-06 07:33:52.198165366 +0000
+++ Gopkg.toml.bak	2018-11-06 07:30:35.864569383 +0000
@@ -21,10 +21,9 @@
 #  version = "2.4.0"


-[[override]]
+[[constraint]]
   name = "github.com/docker/distribution"
-  branch = "master"
-#  version = "2.6.2"
+  version = "2.6.2"

 [[constraint]]
   name = "github.com/docker/go-connections"
@@ -34,7 +33,7 @@
   name = "github.com/docker/go-units"
   version = "0.3.2"

-[[override]]
+[[constraint]]
   branch = "master"
   name = "github.com/googollee/go-socket.io"

@@ -86,6 +85,6 @@
   branch = "master"
   name = "github.com/satori/go.uuid"

-[[override]]
+[[constraint]]
   name = "github.com/zbindenren/negroni-prometheus"
   version = "0.1.1"
```
```
# 書き換え後に実際の取得
dep ensure
```
割と処理に時間がかかる+エラーがなければ特に出力なく完了する。

## Dockerのswarmモード有効化
```
$ docker swarm init
Swarm initialized: current node (30va69p3j32vewjtltpoa43qx) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-2n4p2an0i6m87xtg8kbtvn4bu7x0vib9ovbctnvx3fnbhwfub3-7os0uwuzixoetablq0mfcpjfz 172.31.18.236:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
```
joinの案内あるけどスルーで。

## dockerのDNS設定追加
```
# docker_gwbridgeのネットワークを調べて、
$ docker network inspect docker_gwbridge | grep Gateway
                    "Gateway": "xxx.xx.xx.xx"

# 設定ファイルに入れ込む
$ cat /etc/docker/daemon.json
{
    "dns": [
        "xxx.xx.xx.xx", #ここに"Gateway"のIPを入れる
        "8.8.8.8",
        "10.0.0.2"
    ]
}
```

## docker-compose書き換え
[pwd]コンテナのみ。
commandでの仕込みをやめ、dockerhubの作者提供イメージを拾ってくる形に変更。

```
$ diff -u docker-compose.yml docker-compose.yml.bak
--- docker-compose.yml	2018-11-06 08:54:22.872359931 +0000
+++ docker-compose.yml.bak	2018-11-06 08:51:32.894490772 +0000
@@ -12,10 +12,9 @@
         # pwd daemon container always needs to be named this way
         container_name: pwd
         # use the latest golang image
-        # image: golang
-        image: franela/play-with-docker:latest
-	# go to the right place and starts the app
-        # command: /bin/sh -c 'ssh-keygen -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key >/dev/null; cd /go/src/github.com/play-with-docker/play-with-docker; go run api.go -save /pwd/sessions -name l2'
+        image: golang
+        # go to the right place and starts the app
+        command: /bin/sh -c 'ssh-keygen -N "" -t rsa -f /etc/ssh/ssh_host_rsa_key >/dev/null; cd /go/src/github.com/play-with-docker/play-with-docker; go run api.go -save /pwd/sessions -name l2'
         volumes:
             # since this app creates networks and launches containers, we need to talk to docker daemon
             - /var/run/docker.sock:/var/run/docker.sock
```

## playground場で利用するdockerイメージをあらかじめpull
```
# docker pull franela/dind
```

---
# 最終確認と実働

諸々設定おわったらサーバ再起動。
その後、再度sshの上ちゃんと設定できてるか確認

```
# GOPATH
echo $PATH
# カーネルモジュール
lsmod
# ipv4
sysctl -a | grep net.ipv4.ip_forward
```
概ね問題なさそうであれば起動。

```
cd /home/ec2-user/go/src/github.com/play-with-docker/play-with-docker
docker-compose up
```
起動順の指定みたいなのは無いので、ちょいちょいhaproxyがこけたりする。
遊び場提供というゆるい用途なので、とりあえずリトライで対処。

うまくいけばpwdがブラウザ側のレスポンスを返し、
l2が「Saved networks」を継続して吐き続ける。

```
$ docker-compose -f /home/ec2-user/go/src/github.com/play-with-docker/play-with-docker/docker-compose.yml up
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating network "play-with-docker_default" with the default driver
Creating haproxy ... done
Creating l2      ... done
Creating pwd     ... done
Attaching to l2, pwd, haproxy
pwd        | 2018/11/07 07:29:08 Listening on port 3000
haproxy    | [WARNING] 310/072915 (1) : config : missing timeouts for frontend 'http-in'.
haproxy    |    | While not properly invalid, you will certainly encounter various problems
haproxy    |    | with such a configuration. To fix this, please ensure that all following
haproxy    |    | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
haproxy    | [WARNING] 310/072915 (1) : config : missing timeouts for backend 'pwd'.
haproxy    |    | While not properly invalid, you will certainly encounter various problems
haproxy    |    | with such a configuration. To fix this, please ensure that all following
haproxy    |    | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
haproxy    | [WARNING] 310/072915 (1) : config : missing timeouts for backend 'l2'.
haproxy    |    | While not properly invalid, you will certainly encounter various problems
haproxy    |    | with such a configuration. To fix this, please ensure that all following
haproxy    |    | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
pwd        | [negroni] Started GET /
pwd        | [negroni] Completed 200 OK in 89.312201ms
pwd        | [negroni] Started GET /
pwd        | [negroni] Completed 200 OK in 63.363607ms
l2         | 2018/11/07 07:30:05 Connected to network [bfglrth2894g00f34e6g] with ip [10.0.1.7]
l2         | 2018/11/07 07:30:06 Connected to network [bfgm4nub6c2g008ktoj0] with ip [10.0.3.7]
pwd        | [negroni] Started GET /
pwd        | [negroni] Completed 200 OK in 687.007µs
l2         | 2018/11/07 07:30:08 Connected to network [bfgm8ej05bv00096tqeg] with ip [10.0.4.7]
l2         | 2018/11/07 07:30:09 Connected to network [bfh5n0t3562g00fe6pog] with ip [10.0.6.8]
l2         | 2018/11/07 07:30:10 Connected to network [bfh7daelhh7g008nq1dg] with ip [10.0.8.3]
l2         | 2018/11/07 07:30:11 Connected to network [bfgljsp2894g00f34e50] with ip [10.0.0.7]
l2         | 2018/11/07 07:30:12 Connected to network [bfglvrcbb1n000bodkb0] with ip [10.0.2.7]
l2         | 2018/11/07 07:30:14 Connected to network [bfgmohdnejv000cq81og] with ip [10.0.5.7]
l2         | 2018/11/07 07:30:15 Connected to network [bfh76dkt695g00abn6pg] with ip [10.0.7.7]
pwd        | [negroni] Started GET /
pwd        | [negroni] Completed 200 OK in 186.359µs
l2         | 2018/11/07 07:30:17 Connected to network [play-with-docker_default] with ip [172.23.0.3]
l2         | 2018/11/07 07:30:18 Saved networks
l2         | 2018/11/07 07:30:21 Saved networks
.
.
.
```
・起動中の様子

```
$ docker-compose ps
 Name                Command               State                                Ports
-------------------------------------------------------------------------------------------------------------------
haproxy   /docker-entrypoint.sh hapr ...   Up      0.0.0.0:80->8080/tcp
l2        /bin/sh -c ssh-keygen -N " ...   Up      0.0.0.0:8022->22/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8053->53/tcp
pwd       ./play-with-docker               Up      3000/tcp
```

あとはよしなにURL接続してplay-with-docker!

---
# 付録
ec2-userのホームディレクトリに置いて使う、path潜りを省くシェルスクリプト

```PwD-compose.sh
#! /bin/bash

source ./.bash_profile
YAMLPATH="${GOPATH}/src/github.com/play-with-docker/play-with-docker/docker-compose.yml"

function help() {
cat <<_EOT_

### HELP ###

# Overview
「cd ${GOPATH}/src/github.com/play-with-docker/play-with-docker/」を省いてdocker-composeコマンドを実行するためだけのスクリプトです。
# Usage
$0 {up|up -d|down}

_EOT_
}

case $1 in
  up )
    case $2 in
      -d )
      echo "Execute:"
      echo "docker-compose -f ${YAMLPATH} up -d"
      docker-compose -f ${YAMLPATH} up -d
        ;;
      "" )
      echo "Execute:"
      echo "docker-compose -f ${YAMLPATH} up"  
      docker-compose -f ${YAMLPATH} up
        ;;
      * )
      help
        ;;
    esac
    ;;

  down )
    if [[ $2 != "" ]]; then
      help
      exit 0
    fi
    echo "Execute:"
    echo "docker-compose -f ${YAMLPATH} down"
    docker-compose -f ${YAMLPATH} down
    ;;

  * )
  help
    ;;      
esac
```