Docker Enterprise Edition 確認


# Docker Enterprise Edition セットアップ
https://docs.docker.com/install/linux/docker-ee/centos/

## 準備
* Docker アカウント
* Docker Store にて Trial を登録
* サブスクリプション・ライセンスファイルの取得
* URLの取得
* CentOS 7.4 x86_64 以上用意
* 必要ポートの開放（参考：https://docs.docker.com/datacenter/ucp/2.2/guides/admin/install/system-requirements/）

## DockerEE Install

```shell-session
# export DOCKERURL="https://storebits.docker.com/ee/centos/sub-XXXX"
# echo $DOCKERURL/centos > /etc/yum/vars/dockerurl
# yum install -y yum-utils device-mapper-persistent-data lvm2
# yum-config-manager --add-repo "$DOCKERURL/centos/docker-ee.repo"
# yum install docker-ee
# yum list docker-ee --showduplicates | sort -r
# systemctl start docker
# docker run hello-world
# yum install bash-completion

# groupadd docker
# useradd -m hoge
# usermod -aG docker hoge
# su - hoge
$
```

## UCP（Universal Control Plane）Install

```shell-session
$ docker image pull docker/ucp:2.2.10

$ docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:2.2.10 install --host-address AAA.AAA.AAA.AAA --interactive
```

## DTR（Docker Trusted Registry） Install

* DTR は UCPにより管理されたSwarm上で動くコンテナApp
* なのでSwarmが必要・・・！？
* Workerノード上であること
* 固定のホスト名をもつ
* 80/443 used
 
https://docs.docker.com/datacenter/dtr/2.4/guides/architecture/#under-the-hood
https://docs.docker.com/datacenter/dtr/2.4/guides/admin/install/system-requirements/
https://docs.docker.com/datacenter/dtr/2.4/guides/admin/install/

* Admin Settings -> Docker Trusted Registry

```shell-session
$ docker pull docker/dtr:2.4.4
2.4.4: Pulling from docker/dtr
605ce1bd3f31: Already exists
d91aa78802c7: Pull complete
2309fc511a17: Pull complete
d25a2b267e92: Pull complete
Digest: sha256:4c6a428021bd5a1f69969502c5579a517b9639b97cca2c806178b17ad4389394
Status: Downloaded newer image for docker/dtr:2.4.4

$ docker run -it --rm docker/dtr install  --ucp-node worker02  --ucp-username admin  --ucp-url https://XXX.XXX.XXX.XXX  --ucp-ca "-----BEGIN CERTIFICATE-----
> MIIBgTCCASegAwIBAgIUTGt5qZPo9lxIQHULKQ0l9KAQbCEwCgYIKoZIzj0EAwIw
> HTEbMBkGA1UEAxMSVUNQIENsaWVudCBSb290IENBMB4XDTE4MDYxMjIzNTMwMFoX
> DTIzMDYxMTIzNTMwMFowHTEbMBkGA1UEAxMSVUNQIENsaWVudCBSb290IENBMFkw
> EwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEb5iHVmmZ8h4xmYBPp1KmjkRiZXFIFJK4
> Q8Los7BjhyEpSN3zm+RrvRf/fcaLrOTrY96/RrnMbO2Uuiy9fgElz6NFMEMwDgYD
> VR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQIwHQYDVR0OBBYEFJRoHop/
> C+thDl9igNC1Xu/AZHXaMAoGCCqGSM49BAMCA0gAMEUCIGUuXC0hM9iFHob4dULC
> N5HYClbFNag4ZyKk74Zsy1H0AiEA4KfQseEHlzryUW6zOteWjbk5BWtKPZscw5gU
> fVo6/kE=
> -----END CERTIFICATE-----
> "
INFO[0000] Beginning Docker Trusted Registry installation
ucp-password:
INFO[0002] Validating UCP cert
INFO[0002] Connecting to UCP
INFO[0002] health checking ucp
INFO[0002] The UCP cluster contains the following nodes without port conflicts: worker01, worker02
INFO[0002] Searching containers in UCP for DTR replicas
INFO[0002] Searching containers in UCP for DTR replicas
INFO[0003] verifying [80 443] ports on worker02
INFO[0019] Waiting for running dtr-phase2 container to finish
INFO[0019] starting phase 2
INFO[0000] Validating UCP cert
INFO[0000] Connecting to UCP
INFO[0000] health checking ucp
INFO[0000] Verifying your system is compatible with DTR
INFO[0000] Checking if the node is okay to install on
INFO[0000] Creating network: dtr-ol
INFO[0000] Connecting to network: dtr-ol
INFO[0000] Waiting for phase2 container to be known to the Docker daemon
INFO[0001] Setting up replica volumes...
INFO[0002] Creating initial CA certificates
INFO[0002] Bootstrapping rethink...
INFO[0002] Creating dtr-rethinkdb-0893da3497a7...
INFO[0019] Establishing connection with Rethinkdb
INFO[0020] Waiting for database dtr2 to exist
INFO[0021] Establishing connection with Rethinkdb
INFO[0021] Generated TLS certificate.                    dnsNames=[*.com *.*.com example.com *.dtr *.*.dtr] domains=[*.com *.*.com 172.17.0.1 example.com *.dtr *.*.dtr] ipAddresses=[172.17.0.1]
INFO[0021] License config copied from UCP.
INFO[0021] Migrating db...
INFO[0000] Establishing connection with Rethinkdb
INFO[0000] Migrating database schema                     fromVersion=0 toVersion=9
INFO[0007] Waiting for database notaryserver to exist
INFO[0008] Waiting for database notarysigner to exist
INFO[0008] Waiting for database jobrunner to exist
INFO[0010] Migrated database from version 0 to 9
INFO[0032] Starting all containers...
INFO[0032] Getting container configuration and starting containers...
INFO[0033] Recreating dtr-rethinkdb-0893da3497a7...
INFO[0038] Creating dtr-registry-0893da3497a7...
INFO[0047] Creating dtr-garant-0893da3497a7...
INFO[0056] Creating dtr-api-0893da3497a7...
INFO[0090] Creating dtr-notary-server-0893da3497a7...
INFO[0100] Recreating dtr-nginx-0893da3497a7...
INFO[0109] Creating dtr-jobrunner-0893da3497a7...
INFO[0149] Creating dtr-notary-signer-0893da3497a7...
INFO[0158] Creating dtr-scanningstore-0893da3497a7...
INFO[0170] Trying to get the kv store connection back after reconfigure
INFO[0170] Establishing connection with Rethinkdb
INFO[0171] Verifying auth settings...
INFO[0175] Successfully registered dtr with UCP
INFO[0175] Establishing connection with Rethinkdb
INFO[0176] Background tag migration started
INFO[0176] Installation is complete
INFO[0176] Replica ID is set to: 0893da3497a7

INFO[0176] You can use flag '--existing-replica-id 0893da3497a7' when joining other replicas to your Docker Trusted Registry Cluster
$
```

* フラグを付けて、ほかのDTRクラスタにレプリカをジョインできるらしい
* UCP -> Shared Resources -> Stacks で**Basic Containers**としてリストされているのを確認
* DTR Web UI -> ```https://<host addr where DTR is working>/```

* ログイン後、設定（System -> ）
  * CA for TLS Coommunication（GENERAL -> Domain & proxies -> LOAD BALANCER / PUBLIC ADDRESSにDTRが動いているホストのアドレスを入れてSave、これをしないと login できなかった）
  * Backend Storage
  * Security
  * Garbage collection（Remove Untagged Images , upgradeは試験的）

* Users -> 
  * ユーザ（hoge）を作ってみる

* System -> Repositories -> "CREATE REPOSITORY ON PUSH" をオンにしないと「denied: requested access to the resource is denied」でおこられる

# DTRにPushしてみる

* Docker Hubから落としてタグ付け（ex: ZZZ.ZZZ.ZZZ.ZZZ/hoge/irohaboard
* docker login ZZZ.ZZZ.ZZZ.ZZZ（作ったユーザにて）
* docker push ZZZ.ZZZ.ZZZ.ZZZ/hoge/irohaboard


```shell-session
$ docker login ZZZ.ZZZ.ZZZ.ZZZ
$ docker push ZZZ.ZZZ.ZZZ.ZZZ/hoge/hello-world
The push refers to a repository [ZZZ.ZZZ.ZZZ.ZZZ/hoge/hello-world]
2b8cbd0846c5: Pushed
latest: digest: sha256:d5c74e6f8efc7bdf42a5e22bd764400692cf82360d86b8c587a7584b03f51520 size: 524
```

DTR Web UIからも確認

# DTRからPullしてみる

```shell-session
$ docker login ZZZ.ZZZ.ZZZ.ZZZ
Username: hoge
Password:
Login Succeeded

$ docker pull ZZZ.ZZZ.ZZZ.ZZZ/hoge/irohaboard
Using default tag: latest
latest: Pulling from hoge/irohaboard
d9aaf4d82f24: Pull complete
4ae61ec9aa1f: Pull complete
Digest: sha256:d206780920d470eafa0d192a04588a52f0a48429b00b4f269bfea126efa558dc
Status: Downloaded newer image for ZZZ.ZZZ.ZZZ.ZZZ/hoge/irohaboard:latest
$
```

* 比較的直観的にリポジトリを管理できそう

# デフォルトオーケストレーションタイプの変更

* デフォルト mixed ？
* Settings -> Scheduler page にて変更可能

# Workerノード追加（Swarm）

* Worker対象ノードにもDockerEEインストール、デーモン起動
* Shared Resources -> Nodes -> Add Node でコマンドコピペ
* 対象ノード上で実行

```shell-session
# docker swarm join --token SWMTKN-1-270gpjyei5f77czsszmzsgaf07lm6h0jea6tqa7uj5wwkh9794-f5itn0ncolqfjehv0p4xoiupx XXX.XXX.XXX.XXX:2377
This node joined a swarm as a worker.
```
 

# Workerノード追加（k8s）

* Worker対象ノードにもDockerEEインストール、デーモン起動
* Shared Resources -> Nodes -> Add Node でコマンドコピペ
* 対象ノード上で実行
* k8sをワーカーとするとき、必要ポートもチェック
  * 参考：https://docs.docker.com/ee/ucp/admin/install/system-requirements/
  * 参考：https://kubernetes.io/docs/tasks/tools/install-kubeadm/

# kubernetes CLI インストール

https://docs.docker.com/ee/ucp/user-access/kubectl/

```shell-session
# curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
# chmod + ./kubectl
# mv ./kubectl /usr/local/bin/kubectl
```
UCP Client Bundleを入れないとエラーになる

https://docs.docker.com/ee/ucp/user-access/cli/#use-client-certificates


* My Profile -> Client Bundles -> New Client Bundle -> Generate Client Bundle
  * Certificate Bundleのダウンロードが始まる 
* unzip して、以下実行

> ダウンロードしたファイルは、マネージャが名前解決したものになって、このはIPアドレスによりクラスタができてるので、IPアドレスに関係するファイルを直した。

```shell-session
$ cd client-bundle/
$ eval "$(<env.sh)"
Cluster "XXX.XXX.XXX.XXX:6443_admin" set.
User "ucp_XXX.XXX.XXX.XXX:6443_admin" set.
Context "ucp_XXX.XXX.XXX.XXX:6443_admin" created.

$ kubectl config current-context
ucp_sun-t-XXX.XXX.XXX.XXX_admin

$ kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   1h

$ kubectl get ns
NAME          STATUS    AGE
default       Active    1h
kube-public   Active    1h
kube-system   Active    1h
```

試し、コマンドからNamespace作って、UIから確認

```shell_session
$ cat namespace.yml
apiVersion: v1
kind: Namespace
metadata:
  name: devel

$ kubectl apply -f namespace.yml
namespace "devel" created

$ kubectl get ns devel
NAME      STATUS    AGE
devel     Active    1m
```

UCPでも確認


# UCP Uninstall

```shell-session
$ docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:2.2.10 uninstall-ucp --interactive
INFO[0000] Your engine version 17.06.2-ee-13, build ac44d73 (3.10.0-862.2.3.el7.x86_64) is compatible
INFO[0000] We're about to uninstall from this swarm cluster. UCP ID: x6zb1r849byqgcme942trrnqw
Do you want to proceed with the uninstall? (y/n): y
INFO[0000] Uninstalling UCP on each node...
INFO[0010] UCP has been removed from this cluster successfully.
INFO[0012] Removing UCP Services
```

# マネージャノード、IPアドレス変えたら（テスト用）

* Uninstall UCP（docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:2.2.10 uninstall-ucp）
* docker swarm leave --force
* docker swarm init --advertise-addr XXX.XXX.XXX.XXX
* Reinstall UCP