kubernetes1.9 インストール on centos7

# はじめに
CentOS7にKubernetes(1.9)を導入していきます
下記の構成でKubernetesクラスタを構成していきます

sugi-kubernetes19-master01 : Master
sugi-kubernetes19-node01 : Node
sugi-kubernetes19-node02 : Node


# 事前設定 全てのサーバで実施

kubeadmを実行する際に、swapが有効だとinstallにFailしてしますので、
swapの無効化を実施します

```
swapoff -a
```

編集

```
vim /etc/fstab
```

# kubeadmインストール 全てのサーバで実施
Dockerをinstall

```
yum install -y docker
systemctl enable docker && systemctl start docker
```

依存関係メモ

```
============================================================================================================================================================
Package                                    Arch                      Version                                               Repository                 Size
 ============================================================================================================================================================
Installing:
docker                                     x86_64                    2:1.13.1-53.git774336d.el7.centos                     extras                     16 M
Installing for dependencies:
audit-libs-python                          x86_64                    2.7.6-3.el7                                           base                       73 k
checkpolicy                                x86_64                    2.5-4.el7                                             base                      290 k
container-selinux                          noarch                    2:2.42-1.gitad8f0f7.el7                               extras                     32 k
container-storage-setup                    noarch                    0.8.0-3.git1d27ecf.el7                                extras                     33 k
docker-client                              x86_64                    2:1.13.1-53.git774336d.el7.centos                     extras                    3.7 M
docker-common                              x86_64                    2:1.13.1-53.git774336d.el7.centos                     extras                     86 k
libcgroup                                  x86_64                    0.41-13.el7                                           base                       65 k
libsemanage-python                         x86_64                    2.5-8.el7                                             base                      104 k
oci-register-machine                       x86_64                    1:0-6.git2b44233.el7                                  extras                    1.1 M
oci-systemd-hook                           x86_64                    1:0.1.15-2.gitc04483d.el7                             extras                     33 k
oci-umount                                 x86_64                    2:2.3.3-3.gite3c9055.el7                              extras                     32 k
policycoreutils-python                     x86_64                    2.5-17.1.el7                                          base                      446 k
python-IPy                                 noarch                    0.75-6.el7                                            base                       32 k
setools-libs                               x86_64                    3.3.8-1.1.el7                                         base                      612 k
skopeo-containers                          x86_64                    1:0.1.28-1.git0270e56.el7                             extras                     13 k
yajl                                       x86_64                    2.0.4-4.el7                                           base                       39 k

Transaction Summary
 ============================================================================================================================================================
Install  1 Package (+16 Dependent packages)
```

repository設定

```
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
```

kubelet, kubeadm, kubectlをinstall

```
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
```

依存関係メモ

```
============================================================================================================================================================
 Package                                 Arch                            Version                                  Repository                           Size
============================================================================================================================================================
Installing:
 kubeadm                                 x86_64                          1.9.4-0                                  kubernetes                           17 M
 kubectl                                 x86_64                          1.9.4-0                                  kubernetes                          8.9 M
 kubelet                                 x86_64                          1.9.4-0                                  kubernetes                           17 M
Installing for dependencies:
 kubernetes-cni                          x86_64                          0.6.0-0                                  kubernetes                          8.6 M
 socat                                   x86_64                          1.7.3.2-2.el7                            base                                290 k

Transaction Summary
============================================================================================================================================================
Install  3 Packages (+2 Dependent packages)
```

iptablesがバイパスしているため、正しくルーティングされない問題が、いくつか報告されているらしい。
以下のsysctlで回避

```
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
```

実行例

```
[root@sugi-kubernetes19-node02 ~]# sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /usr/lib/sysctl.d/99-docker.conf ...
fs.may_detach_mounts = 1
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
* Applying /etc/sysctl.conf ...
```

dockerが使用している cgourp driver と kubeletが認識している cgoup driver が一致しているか確認する  
Dockerは、 systemd を使用している

```
[root@sugi-kubernetes19-master01 ~]# docker info | grep -i cgroup
  WARNING: You're not using the default seccomp profile
Cgroup Driver: systemd
```

kubeletのdriver指定もsystemdとなっているので、問題なし

```
[root@sugi-kubernetes19-master01 ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | grep cgroup
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
```

# Master を kubeadm で構成
- --pod-network-cidr の指定  
Kubernetesクラスタで使用するNetworkPluginに依存しますが、FlannelをはじめとしたOverlayNWで使用するNWを指定します  
kubeadm を使用して Flannelをインストールする場合、「10.244.0.0/16」固定となる

実行時間は約2分くらい

```
kubeadm init --pod-network-cidr '10.244.0.0/16'
```

実行例

```
[root@sugi-kubernetes19-master01 ~]# kubeadm init --pod-network-cidr '10.244.0.0/16'
[init] Using Kubernetes version: v1.9.4
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING Hostname]: hostname "sugi-kubernetes19-master01.localdomain" could not be reached
        [WARNING Hostname]: hostname "sugi-kubernetes19-master01.localdomain" lookup sugi-kubernetes19-master01.localdomain on 8.8.8.8:53: no such host
        [WARNING FileExisting-crictl]: crictl not found in system path
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [sugi-kubernetes19-master01.localdomain kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.120.220]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 114.503352 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node sugi-kubernetes19-master01.localdomain as master by adding a label and a taint
[markmaster] Master sugi-kubernetes19-master01.localdomain tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 99bcf2.e89be75362d8794b
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token 99bcf2.e89be75362d8794b 192.168.120.220:6443 --discovery-token-ca-cert-hash sha256:c6094ae953604710d9f10fd4f248e35d7f8c4f0829eff777d677c9462fb83ce1
```

rootユーザでkubectlを実行するために環境変数を指定する必要がある

```
export KUBECONFIG=/etc/kubernetes/admin.conf
```

コマンド実行確認

```
[root@sugi-kubernetes19-master01 kubernetes]# kubectl get nodes
NAME                                     STATUS     ROLES     AGE       VERSION
sugi-kubernetes19-master01.localdomain   NotReady   master    4m        v1.9.4
```

.bash_profileに環境変数を指定する

```
cat <<'EOF' > /root/.bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH

# add for kubernetes
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF
```

master に join するための kubeadm の トークン確認
Expire が 24時間制限となっている。
次回スケールアウトするときには、tokenのcreateが必要なのかも

```
[root@sugi-kubernetes19-master01 ~]# kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION                                                EXTRA GROUPS
99bcf2.e89be75362d8794b   23h       2018-03-18T22:00:39+09:00   authentication,signing   The default bootstrap token generated by 'kubeadm init'.   system:bootstrappers:kubeadm:default-node-token
```

# Pod Network の install
Flannelをinstallしてみる

```
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
```

実行例

```
[root@sugi-kubernetes19-master01 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
clusterrole "flannel" created
clusterrolebinding "flannel" created
serviceaccount "flannel" created
configmap "kube-flannel-cfg" created
daemonset "kube-flannel-ds" created
```

正常に実行されたかを確認するための、すべてのnamespaceでpodsを確認する
以下のように複数のpodが作成され、kube-dnsがRunningであることを確認する

```
[root@sugi-kubernetes19-master01 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                                             READY     STATUS    RESTARTS   AGE
kube-system   etcd-sugi-kubernetes19-master01.localdomain                      1/1       Running   0          6m
kube-system   kube-apiserver-sugi-kubernetes19-master01.localdomain            1/1       Running   0          7m
kube-system   kube-controller-manager-sugi-kubernetes19-master01.localdomain   1/1       Running   0          7m
kube-system   kube-dns-6f4fd4bdf-5tvpn                                         3/3       Running   0          7m
kube-system   kube-flannel-ds-z8btx                                            1/1       Running   0          2m
kube-system   kube-proxy-24mm5                                                 1/1       Running   0          7m
kube-system   kube-scheduler-sugi-kubernetes19-master01.localdomain            1/1       Running   0          6m
```

flannel.1が作成されている

```
[root@sugi-kubernetes19-master01 ~]# ip -d a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:98:17:ee brd ff:ff:ff:ff:ff:ff promiscuity 0
    inet 192.168.120.220/24 brd 192.168.120.255 scope global ens192
       valid_lft forever preferred_lft forever
    inet6 fe80::98a0:413d:6b71:8fbd/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:f6:06:77:86 brd ff:ff:ff:ff:ff:ff promiscuity 0
    bridge forward_delay 1500 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.2:42:f6:6:77:86 designated_root 8000.2:42:f6:6:77:86 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer   89.73 vlan_default_pvid 1 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 4 mcast_hash_max 512 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
    link/ether 8a:35:24:38:60:de brd ff:ff:ff:ff:ff:ff promiscuity 0
    vxlan id 1 local 192.168.120.220 dev ens192 srcport 0 0 dstport 8472 nolearning ageing 300
    inet 10.244.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::8835:24ff:fe38:60de/64 scope link
       valid_lft forever preferred_lft forever
5: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP qlen 1000
    link/ether 0a:58:0a:f4:00:01 brd ff:ff:ff:ff:ff:ff promiscuity 0
    bridge forward_delay 1500 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.a:58:a:f4:0:1 designated_root 8000.a:58:a:f4:0:1 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  255.62 vlan_default_pvid 1 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 4 mcast_hash_max 512 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125
    inet 10.244.0.1/24 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::ac47:5ff:fe51:b4b2/64 scope link
       valid_lft forever preferred_lft forever
6: vethe30d042d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP
    link/ether de:8f:ad:8b:9a:bd brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 1
    veth
    bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.a:58:a:f4:0:1 designated_root 8000.a:58:a:f4:0:1 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on
    inet6 fe80::dc8f:adff:fe8b:9abd/64 scope link
       valid_lft forever preferred_lft forever
```

# node サーバで実行
Masterに参加するように、kubeadmコマンドを実行する
Master側でkubeadmを実行した際に出力された最後の文字列から引用

```
kubeadm join --token 99bcf2.e89be75362d8794b 192.168.120.220:6443 --discovery-token-ca-cert-hash sha256:c6094ae953604710d9f10fd4f248e35d7f8c4f0829eff777d677c9462fb83ce1
```

実行例
2秒ほどで終了する

```
[root@sugi-kubernetes19-node01 ~]# kubeadm join --token 99bcf2.e89be75362d8794b 192.168.120.220:6443 --discovery-token-ca-cert-hash sha256:c6094ae953604710d9f10fd4f248e35d7f8c4f0829eff777d677c9462fb83ce1
[preflight] Running pre-flight checks.
        [WARNING Hostname]: hostname "sugi-kubernetes19-node01.localdomain" could not be reached
        [WARNING Hostname]: hostname "sugi-kubernetes19-node01.localdomain" lookup sugi-kubernetes19-node01.localdomain on 8.8.8.8:53: no such host
        [WARNING FileExisting-crictl]: crictl not found in system path
[discovery] Trying to connect to API Server "192.168.120.220:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.120.220:6443"
[discovery] Requesting info from "https://192.168.120.220:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.120.220:6443"
[discovery] Successfully established connection with API Server "192.168.120.220:6443"

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.
```

nodeサーバでもコンテナが実行されている

```
[root@sugi-kubernetes19-node01 ~]# docker ps
CONTAINER ID        IMAGE                                                                                                               COMMAND                  CREATED              STATUS              PORTS               NAMES
d47fa6aa99b3        quay.io/coreos/flannel@sha256:056cf57fd3bbe7264c0be1a3b34ec2e289b33e51c70f332f4e88aa83970ad891                      "/opt/bin/flanneld..."   46 seconds ago       Up 45 seconds                           k8s_kube-flannel_kube-flannel-ds-w9dz9_kube-system_2b9cd429-29e5-11e8-9843-0050569817ee_1
789d0beaac8e        gcr.io/google_containers/kube-proxy-amd64@sha256:424a9dfc295f26f9d1e8070836d6fa08c83f22d86e592dfccddc847a85b1ef20   "/usr/local/bin/ku..."   50 seconds ago       Up 49 seconds                           k8s_kube-proxy_kube-proxy-ddtfk_kube-system_2b9cbb8a-29e5-11e8-9843-0050569817ee_0
33cc95e86646        gcr.io/google_containers/pause-amd64:3.0                                                                            "/pause"                 About a minute ago   Up About a minute                       k8s_POD_kube-proxy-ddtfk_kube-system_2b9cbb8a-29e5-11e8-9843-0050569817ee_0
513e60c0149b        gcr.io/google_containers/pause-amd64:3.0                                                                            "/pause"                 About a minute ago   Up About a minute                       k8s_POD_kube-flannel-ds-w9dz9_kube-system_2b9cd429-29e5-11e8-9843-0050569817ee_0
```

ステータス確認

```
[root@sugi-kubernetes19-master01 ~]# kubectl get node
NAME                                     STATUS    ROLES     AGE       VERSION
sugi-kubernetes19-master01.localdomain   Ready     master    18m       v1.9.4
sugi-kubernetes19-node01.localdomain     Ready     <none>    4m        v1.9.4
sugi-kubernetes19-node02.localdomain     Ready     <none>    3m        v1.9.4
```

pods確認

```
[root@sugi-kubernetes19-master01 ~]# kubectl get pods --all-namespaces
NAMESPACE     NAME                                                             READY     STATUS    RESTARTS   AGE
kube-system   etcd-sugi-kubernetes19-master01.localdomain                      1/1       Running   0          28m
kube-system   kube-apiserver-sugi-kubernetes19-master01.localdomain            1/1       Running   0          29m
kube-system   kube-controller-manager-sugi-kubernetes19-master01.localdomain   1/1       Running   0          29m
kube-system   kube-dns-6f4fd4bdf-5tvpn                                         3/3       Running   0          29m
kube-system   kube-flannel-ds-tvvbj                                            1/1       Running   1          14m
kube-system   kube-flannel-ds-w9dz9                                            1/1       Running   1          15m
kube-system   kube-flannel-ds-z8btx                                            1/1       Running   0          24m
kube-system   kube-proxy-24mm5                                                 1/1       Running   0          29m
kube-system   kube-proxy-ddtfk                                                 1/1       Running   0          15m
kube-system   kube-proxy-gnnvw                                                 1/1       Running   0          14m
kube-system   kube-scheduler-sugi-kubernetes19-master01.localdomain            1/1       Running   0          28m
```

上記で構築完了です

# 参考URL
https://kubernetes.io/docs/setup/independent/install-kubeadm/