kubeadmでCoreOS Container LinuxにKubernetes 1.17

[kubeadmでCoreOS Container LinuxにKubernetes 1.16を入れる場合はこちら](https://qiita.com/rururu_kenken/items/ce8a5cbe3be5855c06b9)


# CoreOS Container LinuxのEOLが発表されました(2020/02/20追記)

情報の一部

2020/05/26に最終アップデートの公開が開始されます。
その日以降に発見されたバグやセキュリティ脆弱性は修正されません。

2020/09/01以降、CoreOS Container Linuxに関連する公開リソースは削除されるか、読み取り専用になります。 OSダウンロードは削除され、CoreUpdateサーバーはシャットダウンされ、OSイメージはAWS、Azure、Google Compute Engineから削除されます。

全ての内容は以下のサイトをご参照ください
[End-of-life announcement for CoreOS Container Linux](https://coreos.com/os/eol/)

# Master Node

|    EC2　　 |    OS         | Kubernetes | Docker    |
|:----------:|:-------------:|:----------:|:---------:|
| t3.small   | CoreOS Container Linux 2303.3.0| 1.17.0|18.06.3-ce |

CoreOS Container Linux AMI のユーザー名は core です

Master Nodeは、2CPU、メモリ2GB以下の場合、kubeadm init実行時にエラーになります。
ディスクサイズはデフォルトの8G

Master NodeとWorker Nodeは同じVPC
Master NodeのIPアドレスは172.31.21.25

## SELinux設定確認

```
$ getenforce

# SELinuxは有効だが、アクセス制限は行わず警告を出力
Permissive
```

## swap確認

```
$ free

# ないので何もしない。ある場合はswapoff -a
              total        used        free      shared  buff/cache   available
Mem:        2002464       80540     1502068      204416      419856     1575644
Swap:             0           0           0
```

## Docker設定

```
$ docker -v

Docker version 18.06.3-ce, build d7080c1

# Dockerの自動起動有効
$ sudo systemctl enable docker

# daemon.json

$ df -T

Filesystem      Type     1K-blocks   Used Available Use% Mounted on
devtmpfs        devtmpfs    979456      0    979456   0% /dev
tmpfs           tmpfs      1001232      0   1001232   0% /dev/shm
tmpfs           tmpfs      1001232    304   1000928   1% /run
tmpfs           tmpfs      1001232      0   1001232   0% /sys/fs/cgroup
/dev/nvme0n1p9  ext4       5706380  27144   5402908   1% /
/dev/mapper/usr ext4       1007760 874208     81536  92% /usr
none            tmpfs      1001232 204112    797120  21% /run/torcx/unpack
tmpfs           tmpfs      1001232      0   1001232   0% /media
tmpfs           tmpfs      1001232      0   1001232   0% /tmp
/dev/nvme0n1p6  ext4        110576    112    101292   1% /usr/share/oem
/dev/nvme0n1p1  vfat        129039  55019     74020  43% /boot
tmpfs           tmpfs       200244      0    200244   0% /run/user/500

$ sudo mkdir -p /etc/docker
$ sudo tee /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

$ sudo mkdir -p /etc/systemd/system/docker.service.d
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
```

## kubelet、kubeadm、kubectlインストール

CNI plugins (required for most pod network)

[plugins](https://github.com/containernetworking/plugins)

```
$ CNI_VERSION="v0.8.4"
$ sudo mkdir -p /opt/cni/bin
$ curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | sudo tar -C /opt/cni/bin -xz

$ ls /opt/cni/bin

bandwidth  bridge  dhcp  firewall  flannel  host-device  host-local  ipvlan  loopback  macvlan  portmap  ptp  sbr  static  tuning  vlan
```

crictl (required for kubeadm / Kubelet Container Runtime Interface (CRI))

[cri-tools](https://github.com/kubernetes-sigs/cri-tools)

```
$ CRICTL_VERSION="v1.17.0"
$ sudo mkdir -p /opt/bin
$ curl -L "https://github.com/kubernetes-incubator/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-amd64.tar.gz" | sudo tar -C /opt/bin -xz

$ ls /opt/bin

crictl
```

kubeadm, kubelet, kubectl

```
$ RELEASE="v1.17.0"
$ sudo mkdir -p /opt/bin
$ cd /opt/bin
$ sudo curl -L --remote-name-all https://storage.googleapis.com/kubernetes-release/release/${RELEASE}/bin/linux/amd64/{kubeadm,kubelet,kubectl}
$ sudo chmod +x {kubeadm,kubelet,kubectl}

$ ls

crictl  kubeadm  kubectl  kubelet

$ cd
$ curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/kubelet.service" | sed "s:/usr/bin:/opt/bin:g" > ./kubelet.service
$ sudo mv ./kubelet.service /etc/systemd/system/

$ sudo mkdir -p /etc/systemd/system/kubelet.service.d
$ curl -sSL "https://raw.githubusercontent.com/kubernetes/kubernetes/${RELEASE}/build/debs/10-kubeadm.conf" | sed "s:/usr/bin:/opt/bin:g" > ./10-kubeadm.conf
$ sudo mv 10-kubeadm.conf /etc/systemd/system/kubelet.service.d
```

## kubelet 起動

```
$ sudo systemctl enable --now kubelet
```

## Masterノードのセットアップ

[kubernetes 1.17.0 - kubeadm init - kube-controller-manager status is ContainerCreating #86961](https://github.com/kubernetes/kubernetes/issues/86961)

```
$ cat <<EOF > ./kubeadm-custom.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.17.0
controllerManager:
  extraArgs:
    flex-volume-plugin-dir: "/etc/kubernetes/kubelet-plugins/volume/exec"
networking:
  podSubnet: 10.244.0.0/16
EOF

$ sudo kubeadm init --config kubeadm-custom.yaml
```

## kubectl 接続設定

```
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

## 確認

```
$ kubectl get pod --all-namespaces

NAMESPACE     NAME                                                                      READY   STATUS    RESTARTS   AGE
kube-system   coredns-6955765f44-57nmn                                                  0/1     Pending   0          6m54s
kube-system   coredns-6955765f44-j6g2f                                                  0/1     Pending   0          6m54s
kube-system   etcd-ip-172-31-21-25.ap-northeast-1.compute.internal                      1/1     Running   0          7m8s
kube-system   kube-apiserver-ip-172-31-21-25.ap-northeast-1.compute.internal            1/1     Running   0          7m8s
kube-system   kube-controller-manager-ip-172-31-21-25.ap-northeast-1.compute.internal   1/1     Running   0          7m9s
kube-system   kube-proxy-5kjt7                                                          1/1     Running   0          6m54s
kube-system   kube-scheduler-ip-172-31-21-25.ap-northeast-1.compute.internal            1/1     Running   0          7m9s
```

## Flannelデプロイ

[Super slow access to service IP from host (& host-networked pods) with Flannel CNI #1245](https://github.com/coreos/flannel/issues/1245)

```
$ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
```

```
$ kubectl get pod --all-namespaces

NAMESPACE     NAME                                                                      READY   STATUS    RESTARTS   AGE
kube-system   coredns-6955765f44-57nmn                                                  1/1     Running   0          8m15s
kube-system   coredns-6955765f44-j6g2f                                                  1/1     Running   0          8m15s
kube-system   etcd-ip-172-31-21-25.ap-northeast-1.compute.internal                      1/1     Running   0          8m29s
kube-system   kube-apiserver-ip-172-31-21-25.ap-northeast-1.compute.internal            1/1     Running   0          8m29s
kube-system   kube-controller-manager-ip-172-31-21-25.ap-northeast-1.compute.internal   1/1     Running   0          8m30s
kube-system   kube-flannel-ds-amd64-5r8ck                                               1/1     Running   0          20s
kube-system   kube-proxy-5kjt7                                                          1/1     Running   0          8m15s
kube-system   kube-scheduler-ip-172-31-21-25.ap-northeast-1.compute.internal            1/1     Running   0          8m30s

$ kubectl get node

NAME                                              STATUS   ROLES    AGE     VERSION
ip-172-31-21-25.ap-northeast-1.compute.internal   Ready    master   9m21s   v1.17.0

$ kubectl get cs  

NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
```

# 参考URL

[kubeadmのインストール](https://kubernetes.io/ja/docs/setup/independent/install-kubeadm/)
[kubeadmのトラブルシューティング](https://kubernetes.io/ja/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/)