kubernetes(1.6)のインストール手順


#1 環境

-　クラスタは2台構成(マスタ1台、ノード1台)
-　2台ともVMware Workstation Playerのゲストマシン
-　OSの版数は以下のとおり

```
[root@master ~]# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)

[root@master ~]# uname -r
3.10.0-514.el7.x86_64
```

#2 マスタのインストール手順
今現在、centosリポジトリにはkubernetes1.6のパッケージが存在しないので、googleのリポジトリより
パッケージをダウンロードします。

```console:hostsファイルの編集
hostsファイルを編集する。masterとnodeを登録する。
[root@master ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.10 master
192.168.0.20 node
```


```console:リポジトリの追加
kubernetesリポジトリを追加する。
[root@master ~]# vi /etc/yum.repos.d/kubernetes.repo
[root@master ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
```

```console:暫定対処（SELinuxの無効化）
SELinuを無効にする。現在はSELinxuを有効にできない模様(kubernetes公式ページより)
[root@master ~]# setenforce 0
```

```console:パッケージのインストール
[root@master ~]# yum install -y docker kubelet kubeadm kubernetes-cni
```

```console:dockerの起動
[root@master ~]# systemctl enable docker && systemctl start docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
```

```console:kubeletの起動
[root@master ~]# systemctl enable kubelet && systemctl start kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
```

```console:ポート番号の開放
kube-apiserver(6443)とetcd(6666,6667)が使うポート番号をアクセス可にする。
[root@master ~]# firewall-cmd --add-port=6443/tcp
[root@master ~]# firewall-cmd --add-port=6666/tcp
[root@master ~]# firewall-cmd --add-port=6667/tcp
[root@master ~]# firewall-cmd --runtime-to-permanent
```

```console:マスタの初期化
マスタを初期化する。私の環境では、「Created API client,...」のところで(★印)、5,6分ほど待たされた。
[root@master ~]# kubeadm init
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.4
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] WARNING: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.10]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[apiclient] ★Created API client, waiting for the control plane to become ready

[apiclient] All control plane components are healthy after 413.882066 seconds
[apiclient] Waiting for at least one node to register
[apiclient] First node has registered after 3.938993 seconds
[token] Using token: aa4830.cc1a037a87d63f1e
[apiconfig] Created RBAC rules
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

● sudo cp /etc/kubernetes/admin.conf $HOME/
● sudo chown $(id -u):$(id -g) $HOME/admin.conf
● export KUBECONFIG=$HOME/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  ★クラスタにノードを追加するさい、下記コマンドをそのまま入力する。
  kubeadm join --token aa4830.cc1a037a87d63f1e 192.168.0.10:6443


kubeadm initコマンドの実行結果に表示される指示(●印)にしたがって、下記コマンドを実行する。
[root@master ~]# cp /etc/kubernetes/admin.conf $HOME/
[root@master ~]# chown $(id -u):$(id -g) $HOME/admin.conf
[root@master ~]# export KUBECONFIG=$HOME/admin.conf

バグ回避策適用(https://github.com/kubernetes/kubernetes/issues/43815)
[root@master ~]# vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
  ↓以下に変更
#Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

calicoのyamlファイルをダウンロードする。
[root@master ~]# wget http://docs.projectcalico.org/v2.3/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml

ファイルを確認する。中身は編集せず、そのまま使う。
[root@master ~]# ls calico.yaml
calico.yaml

calicoのネットワークを作成する。
[root@master ~]# kubectl apply -f calico.yaml
configmap "calico-config" created
daemonset "calico-etcd" created
service "calico-etcd" created
daemonset "calico-node" created
deployment "calico-policy-controller" created
clusterrolebinding "calico-cni-plugin" created
clusterrole "calico-cni-plugin" created
serviceaccount "calico-cni-plugin" created
clusterrolebinding "calico-policy-controller" created
clusterrole "calico-policy-controller" created
serviceaccount "calico-policy-controller" created

[root@master ~]# systemctl daemon-reload
[root@master ~]# systemctl restart kubelet

```



#3 ノードのインストール手順

```
hostsファイルを編集する。masterとnodeを登録する。
[root@node ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.10 master
192.168.0.20 node

kubernetesリポジトリを追加する。
[root@node ~]# vi /etc/yum.repos.d/kubernetes.repo
[root@node ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

[root@node ~]# setenforce 0
[root@node ~]# yum install -y docker kubelet kubeadm kubernetes-cni

[root@node ~]#  systemctl enable docker && systemctl start docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

[root@node ~]# systemctl enable kubelet && systemctl start kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.


バグ回避策適用(https://github.com/kubernetes/kubernetes/issues/43815)
[root@master ~]# vi /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
  ↓以下に変更
#Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

[root@node ~]# kubeadm join --token aa4830.cc1a037a87d63f1e 192.168.0.10:6443
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "192.168.0.10:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.10:6443"
[discovery] Cluster info signature and contents are valid, will use API Server "https://192.168.0.10:6443"
[discovery] Successfully established connection with API Server "192.168.0.10:6443"
[bootstrap] Detected server version: v1.6.4
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)
[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request
[csr] Received signed certificate from the API server, generating KubeConfig...
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.
[root@node ~]#

```


#4 インストール後の状態確認
##4.1 kubectlコマンドの補完機能の有効化
kubectlコマンドを実行するさい、補完機能が使えると便利です。
下記コマンドを実行して、補完機能を有効化しておきます。

```
[root@master ~]# yum -y install bash-completion
[root@master ~]# source <(kubectl completion bash)

```


##4.2 kubernetes版数の確認
```
[root@master ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

```

##4.3 ノードの確認

```
[root@master ~]# kubectl get node -o wide
NAME      STATUS    AGE       VERSION   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION
master    Ready     52m       v1.6.4    <none>        CentOS Linux 7 (Core)   3.10.0-514.el7.x86_64
node      Ready     50m       v1.6.4    <none>        CentOS Linux 7 (Core)   3.10.0-514.el7.x86_64

```


##4.4 Podの確認

```
Podの状態を確認する。全てRunningになるまで40分ほどかかった。自宅のインテーネット回線速度は数百kbs程度です。
[root@master ~]# kubectl get pod --all-namespaces -o wide
NAMESPACE     NAME                                        READY     STATUS    RESTARTS   AGE       IP             NODE
kube-system   calico-etcd-hx2n1                           1/1       Running   0          35m       192.168.0.10   master
kube-system   calico-node-4kgrp                           2/2       Running   1          34m       192.168.0.20   node
kube-system   calico-node-pnc1j                           2/2       Running   0          35m       192.168.0.10   master
kube-system   calico-policy-controller-1324707180-h8pxc   1/1       Running   3          35m       192.168.0.10   master
kube-system   etcd-master                                 1/1       Running   2          36m       192.168.0.10   master
kube-system   kube-apiserver-master                       1/1       Running   9          35m       192.168.0.10   master
kube-system   kube-controller-manager-master              1/1       Running   6          36m       192.168.0.10   master
kube-system   kube-dns-3913472980-42bnq                   3/3       Running   9          36m       172.17.0.2     master
kube-system   kube-proxy-979tn                            1/1       Running   0          36m       192.168.0.10   master
kube-system   kube-proxy-z2rcr                            1/1       Running   0          34m       192.168.0.20   node
kube-system   kube-scheduler-master                       1/1       Running   2          36m       192.168.0.10   master

```

##4.5 サービスの確認

```
[root@master ~]# kubectl get svc --all-namespaces
NAMESPACE     NAME          CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       kubernetes    10.96.0.1       <none>        443/TCP         56m
kube-system   calico-etcd   10.96.232.136   <none>        6666/TCP        37m
kube-system   kube-dns      10.96.0.10      <none>        53/UDP,53/TCP   38m

===================================================================================================
digコマンドのパラメータの意味について説明する。
(1)@に指定するIPアドレス。kube-dnsのCLUSTER-IPを指定する。ここの例では10.96.0.10。
(2)名前の書式は以下のとおり。NAMEとNAMESPACEは、kubectl get svcより得たものを指定する。
   <NAME>.<NAMESPACE>.svc.cluster.local
(3)ドメイン名
    KUBELET_DNS_ARGS(/etc/systemd/system/kubelet.service.d/10-kubeadm.conf)に設定したものを指定する。
    ここの例では、ドメイン名はcluster.localとなる。
===================================================================================================

サービス名(kubernetes)に対するIPアドレスを求める。正しくIPアドレスが引けていることがわかる。
[root@master ~]# dig @10.96.0.10 kubernetes.default.svc.cluster.local +short
10.96.0.1

サービス名(calico-etcd)に対するIPアドレスを求める。正しくIPアドレスが引けていることがわかる。
[root@master ~]# dig @10.96.0.10 calico-etcd.kube-system.svc.cluster.local +short
10.96.232.136

サービス名(kube-dns)に対するIPアドレスを求める。正しくIPアドレスが引けていることがわかる。
[root@master ~]# dig @10.96.0.10 kube-dns.kube-system.svc.cluster.local +short
10.96.0.10

```


##4.6 ポート番号の確認

```
kube-apiserverが使うポート番号を確認する。
[root@master ~]# lsof -i:6443 -P
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
kube-prox  3029 root    3u  IPv4 106012      0t0  TCP master:44318->master:6443 (ESTABLISHED)
kube-cont  8890 root    3u  IPv4 105054      0t0  TCP master:44296->master:6443 (ESTABLISHED)
kube-cont  8890 root    7u  IPv4 105159      0t0  TCP master:44352->master:6443 (ESTABLISHED)
kube-sche  9422 root    9u  IPv4 106002      0t0  TCP master:44288->master:6443 (ESTABLISHED)
kubelet   10630 root   24u  IPv4 105053      0t0  TCP master:44294->master:6443 (ESTABLISHED)
kube-apis 10817 root   47u  IPv6 105025      0t0  TCP *:6443 (LISTEN) ★6443番ポートでListenしていることがわかる。
kube-apis 10817 root   48u  IPv6 105160      0t0  TCP master:6443->master:44352 (ESTABLISHED)
kube-apis 10817 root   49u  IPv6 112510      0t0  TCP master:6443->master:59824 (ESTABLISHED)
kube-apis 10817 root   50u  IPv4 105989      0t0  TCP localhost:37636->localhost:6443 (ESTABLISHED)
kube-apis 10817 root   51u  IPv6 112512      0t0  TCP master:6443->master:59826 (ESTABLISHED)
kube-apis 10817 root   52u  IPv6 112966      0t0  TCP master:6443->master:59828 (ESTABLISHED)
kube-apis 10817 root   53u  IPv6 105990      0t0  TCP localhost:6443->localhost:37636 (ESTABLISHED)
kube-apis 10817 root   56u  IPv6 105987      0t0  TCP master:6443->172.17.0.2:42284 (ESTABLISHED)
kube-apis 10817 root   59u  IPv6 105997      0t0  TCP master:6443->node:34990 (ESTABLISHED)
kube-apis 10817 root   64u  IPv6 105048      0t0  TCP master:6443->master:44288 (ESTABLISHED)
kube-apis 10817 root   67u  IPv6 105057      0t0  TCP master:6443->master:44294 (ESTABLISHED)
kube-apis 10817 root   68u  IPv6 105058      0t0  TCP master:6443->master:44296 (ESTABLISHED)
kube-apis 10817 root   86u  IPv6 105071      0t0  TCP master:6443->master:44318 (ESTABLISHED)
kube-apis 10817 root   88u  IPv6 105073      0t0  TCP master:6443->node:34994 (ESTABLISHED)


etcdが使うポート番号を確認する。etcdは6666番と6667番を使っていることがわかる。
6666番がクライアント用、6667番がetcdピア用に使うと推測しています。従来の2379番と2380番と同じ使い方と思われる。
[root@master ~]# lsof -i:6666 -P
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
etcd       4484 root    5u  IPv6  48375      0t0  TCP *:6666 (LISTEN) ★6666番ポートでListenしていることがわかる。
etcd       4484 root    6u  IPv6 107284      0t0  TCP master:6666->master:39854 (ESTABLISHED)
etcd       4484 root    9u  IPv6 108237      0t0  TCP master:6666->master:39878 (ESTABLISHED)
etcd       4484 root   10u  IPv6 108242      0t0  TCP master:6666->master:39884 (ESTABLISHED)
etcd       4484 root   11u  IPv6  56255      0t0  TCP master:6666->node:51660 (ESTABLISHED)
etcd       4484 root   12u  IPv6  60360      0t0  TCP master:6666->master:57912 (ESTABLISHED)
etcd       4484 root   13u  IPv6  56257      0t0  TCP master:6666->node:51664 (ESTABLISHED)
etcd       4484 root   14u  IPv6  56964      0t0  TCP master:6666->node:51682 (ESTABLISHED)
etcd       4484 root   15u  IPv6  56259      0t0  TCP master:6666->node:51668 (ESTABLISHED)
etcd       4484 root   16u  IPv6  56260      0t0  TCP master:6666->node:51670 (ESTABLISHED)
etcd       4484 root   17u  IPv6  56261      0t0  TCP master:6666->node:51672 (ESTABLISHED)
etcd       4484 root   18u  IPv6  56968      0t0  TCP master:6666->node:51688 (ESTABLISHED)
etcd       4484 root   19u  IPv6  56965      0t0  TCP master:6666->node:51684 (ESTABLISHED)
etcd       4484 root   20u  IPv6  56966      0t0  TCP master:6666->node:51686 (ESTABLISHED)
etcd       4484 root   21u  IPv6  58304      0t0  TCP master:6666->node:51696 (ESTABLISHED)
etcd       4484 root   22u  IPv6  56969      0t0  TCP master:6666->node:51690 (ESTABLISHED)
etcd       4484 root   23u  IPv6  56282      0t0  TCP master:6666->node:51692 (ESTABLISHED)
etcd       4484 root   25u  IPv6  58957      0t0  TCP master:6666->master:57970 (ESTABLISHED)
etcd       4484 root   26u  IPv6  58306      0t0  TCP master:6666->node:51698 (ESTABLISHED)
etcd       4484 root   27u  IPv6  58911      0t0  TCP master:6666->master:57930 (ESTABLISHED)
etcd       4484 root   28u  IPv6  58961      0t0  TCP master:6666->master:57974 (ESTABLISHED)
etcd       4484 root   29u  IPv6  58922      0t0  TCP master:6666->master:57934 (ESTABLISHED)
etcd       4484 root   30u  IPv6  60409      0t0  TCP master:6666->master:57962 (ESTABLISHED)
etcd       4484 root   31u  IPv6  60410      0t0  TCP master:6666->master:57964 (ESTABLISHED)
etcd       4484 root   32u  IPv6  60407      0t0  TCP master:6666->master:57960 (ESTABLISHED)
etcd       4484 root   33u  IPv6  60411      0t0  TCP master:6666->master:57966 (ESTABLISHED)
etcd       4484 root   34u  IPv6  60412      0t0  TCP master:6666->master:57968 (ESTABLISHED)
etcd       4484 root   35u  IPv6  58964      0t0  TCP master:6666->master:57976 (ESTABLISHED)
etcd       4484 root   36u  IPv6  60399      0t0  TCP master:6666->master:57948 (ESTABLISHED)
etcd       4484 root   39u  IPv6  60435      0t0  TCP master:6666->master:57984 (ESTABLISHED)
etcd       4484 root   41u  IPv6  87842      0t0  TCP master:6666->master:60654 (ESTABLISHED)
etcd       4484 root   42u  IPv6  87667      0t0  TCP master:6666->node:51706 (ESTABLISHED)
calico-fe  5525 root    3u  IPv4  58879      0t0  TCP master:57912->10.96.232.136:6666 (ESTABLISHED)
calico-fe  5525 root    5u  IPv4  58992      0t0  TCP master:57984->10.96.232.136:6666 (ESTABLISHED)
calico-fe  5525 root   12u  IPv4  88399      0t0  TCP master:60654->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root    3u  IPv4  58952      0t0  TCP master:57964->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root    5u  IPv4  60387      0t0  TCP master:57930->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root    6u  IPv4  58953      0t0  TCP master:57966->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root    7u  IPv4  60389      0t0  TCP master:57934->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root    8u  IPv4  58954      0t0  TCP master:57968->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root    9u  IPv4  58960      0t0  TCP master:57974->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root   10u  IPv4  58963      0t0  TCP master:57976->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root   11u  IPv4  58950      0t0  TCP master:57960->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root   12u  IPv4  58956      0t0  TCP master:57970->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root   13u  IPv4  58951      0t0  TCP master:57962->10.96.232.136:6666 (ESTABLISHED)
confd      5527 root   14u  IPv4  60395      0t0  TCP master:57948->10.96.232.136:6666 (ESTABLISHED)
controlle 11147 root    5u  IPv4 108168      0t0  TCP master:39854->10.96.232.136:6666 (ESTABLISHED)
controlle 11147 root    9u  IPv4 108233      0t0  TCP master:39878->10.96.232.136:6666 (ESTABLISHED)
controlle 11147 root   13u  IPv4 108241      0t0  TCP master:39884->10.96.232.136:6666 (ESTABLISHED)
[root@master ~]# 

[root@master ~]# lsof -i:6667 -P
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
etcd    4484 root    3u  IPv6  48374      0t0  TCP *:6667 (LISTEN) ★6667番ポートでListenしていることがわかる。
[root@master ~]#

```


##4.7 Podの起動・終了ができるかどうかの確認

```
Podの状態を確認する。何も起動していないことがわかる。
[root@master ~]# kubectl get pod
No resources found.

Podを起動する。
[root@master ~]# kubectl run nginx --image=nginx
deployment "nginx" created

Podの状態を確認する。Pod(Nginx)が起動できたことがわかる。
[root@master ~]# kubectl get pod -o wide
NAME                     READY     STATUS    RESTARTS   AGE       IP           NODE
nginx-2371676037-z8rcr   1/1       Running   0          15m       172.18.0.2   node

Podを終了する。
[root@master ~]# kubectl delete deployment nginx
deployment "nginx" deleted

Podの状態を確認する。Pod(Nginx)が終了したことがわかる。
[root@master ~]# kubectl get pod -o wide
No resources found.

```


#5 コマンド一覧
##5.1 kubectlコマンド
```
[root@master ~]# kubectl
annotate        autoscale       convert         describe        expose          patch           rollout         top
api-versions    certificate     cordon          drain           get             port-forward    run             uncordon
apply           cluster-info    cp              edit            label           proxy           scale           version
attach          completion      create          exec            logs            replace         set
auth            config          delete          explain         options         rolling-update  taint

```


##5.2 kubeadmコマンド
kubeadmコマンドはベータ版のようです。まだ商用製品に使わないほうがよさそうです。

```
[root@master ~]# kubeadm --help
kubeadm: easily bootstrap a secure Kubernetes cluster.

    lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
    x KUBEADM IS BETA, DO NOT USE IT FOR PRODUCTION CLUSTERS!  x
    x                                                          x
    x But, please try it out! Give us feedback at:             x
    x https://github.com/kubernetes/kubeadm/issues             x
    x and at-mention @kubernetes/sig-cluster-lifecycle-misc    x
    mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj

-中略-

Available Commands:
  alpha       Experimental sub-commands not yet fully functional.
  completion  Output shell completion code for the specified shell (bash or zsh)
  init        Run this in order to set up the Kubernetes master
  join        Run this on any machine you wish to join an existing cluster
  reset       Run this to revert any changes made to this host by 'kubeadm init' or 'kubeadm join'.
  token       Manage bootstrap tokens.
  version     Print the version of kubeadm

```


#6 参考情報
[kubernetes公式ページのインストール説明（その1）](https://kubernetes.io/docs/setup/independent/install-kubeadm/)
[kubernetes公式ページのインストール説明（その2）](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/)
[Calico公式ページ(calico.yamlのダウンロード方法を説明したページ)](http://docs.projectcalico.org/v2.3/getting-started/kubernetes/installation/hosted/kubeadm/)
[kubeadm initがハングする件の対処方法の説明](https://github.com/kubernetes/kubeadm/issues/228)