kubeadmでk8sクラスタを構築

ついにここまでやってきました。ここではオレオレk8s環境を、minikubeなどを使わずにkubeadmを使ってサーバにセットアップしていきます。
公式の手順を参考にしながら進めましょう。
https://kubernetes.io/ja/docs/setup/independent/install-kubeadm/

kubernetes v1.19.1版で書き直しました。

CRI-Oを利用したv1.25のインストール方法はこちら。 https://qiita.com/murata-tomohide/items/cd408dbed0211fedf5dc

```
murata:/etc/sysctl.d $ uname -r
5.8.12-200.fc32.x86_64
murata:~ $ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:26:42Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-09T11:18:22Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
```

# TL;DR

オレオレk8sサーバでローカルPCから`kubectl apply`できるようにする。
k8sとDockerをローカルPCからいじくれるようにしてます。
master＆worker nodeとして動くようにしています。


# Install Docker

k8sのキモとなるDockerはどの場合でもインストールしておく必要があるよ。
公式ドキュメントを参考にいれましょう。今回はDocker CEを使っています。(CEはcommunity editionのこと)
https://docs.docker.com/

2020/12/03追記
kubernetes 1.20からDockerが非推奨となるようです。代わりにCRI-OもしくはCoitainerdを使う必要があります。やり方は公式に。
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#container-runtimes

Kubernetesさんが作ったCRIっていうAPI仕様をDockerがサポートしていないから＆Dockerは管理デーモンが存在していて障害点になりえるってことでDeprecateだそうです。

# OSの準備

公式の手順を参考にしながら進めましょう。
https://kubernetes.io/ja/docs/setup/independent/install-kubeadm/

## swapをoff 

[始める前に](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin)の項にswapをOFFにしておけよってあるのでします。
**Swapがオフであること。kubeletが正常に動作するためにはswapは必ずオフでなければなりません。**

```
murata:~ $ sudo swapoff -a
murata:~ $ free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       269Mi        30Gi       0.0Ki       540Mi        30Gi
Swap:            0B          0B          0B
```

`sudo swapoff -a`してswap領域がなくなっていることを確認すればよろしいかと。

## sysctl 調整

sysctlでiptablesの調整をします。マニュアル通りにコピペします。（root権限が必要）

```
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
```

結果に下記の内容が含まれていればよしです。

```
~~snip;
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
~~snip;
```

## その他

そのほかにMACアドレスやホスト名がユニークであること、product_uuidがほかサーバと重複していないか確認すべし！とあります。
firewallによって利用するポートを許可してください。とりあえず今回はお試しなので `systemctl stop firewalld` しました。（外に出ているサーバの場合には絶対にしないように注意）

fedoraなどでiptablesを利用していない場合は、使うように設定します。（デフォルトはnftablesを使っていると思います。）
ちなみにルーティングにiptablesを使っているようなのでOFFにしてはいけません。

```
update-alternatives --set iptables /usr/sbin/iptables-legacy
```

確認します。

```
[root@k8s-tmp ~]# alternatives --display iptables
iptables - status is auto.
 link currently points to /usr/sbin/iptables-legacy
/usr/sbin/iptables-legacy - priority 10
 slave ip6tables: /usr/sbin/ip6tables-legacy
 slave ip6tables-restore: /usr/sbin/ip6tables-legacy-restore
 slave ip6tables-save: /usr/sbin/ip6tables-legacy-save
 slave iptables-restore: /usr/sbin/iptables-legacy-restore
 slave iptables-save: /usr/sbin/iptables-legacy-save
Current `best' version is /usr/sbin/iptables-legacy.
```

# インストール

今回はFedora32を使っています。

[kubeadm、kubelet、kubectlのインストール](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl)を参考にyumリポジトリを追加します。

公式のままですね。

```
[root@k8s-tmp ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
> enabled=1
> gpgcheck=1
> repo_gpgcheck=1
> gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
> exclude=kube*
> EOF
[root@k8s-tmp ~]# ll /etc/yum.repos.d/kubernetes.repo
-rw-r--r--. 1 root root 277 Aug  7  2019 /etc/yum.repos.d/kubernetes.repo
```

selinuxを調整して、インストールします。公式ではyumコマンドを利用していますが、fedoraさんらしくdnfでインストールします。

```
[root@murata murata]# sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
[root@k8s-tmp ~]# dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
google-chrome                                                                                                                                                                                          5.6 kB/s | 1.3 kB     00:00
Dependencies resolved.
=======================================================================================================================================================================================================================================
 Package                                                             Architecture                                   Version                                                   Repository                                          Size
=======================================================================================================================================================================================================================================
Installing:
 kubeadm                                                             x86_64                                         1.19.2-0                                                  kubernetes                                         8.3 M
 kubectl                                                             x86_64                                         1.19.2-0                                                  kubernetes                                         9.0 M
 kubelet                                                             x86_64                                         1.19.2-0                                                  kubernetes                                          19 M
Installing dependencies:
 conntrack-tools                                                     x86_64                                         1.4.5-5.fc32                                              fedora                                             205 k
 containernetworking-plugins                                         x86_64                                         0.8.7-1.fc32                                              updates                                             11 M
 cri-tools                                                           x86_64                                         1.13.0-0                                                  kubernetes                                         5.1 M
 ethtool                                                             x86_64                                         2:5.8-1.fc32                                              updates                                            208 k
 libnetfilter_cthelper                                               x86_64                                         1.0.0-17.fc32                                             fedora                                              23 k
 libnetfilter_cttimeout                                              x86_64                                         1.0.0-15.fc32                                             fedora                                              23 k
 libnetfilter_queue                                                  x86_64                                         1.0.2-15.fc32                                             fedora                                              27 k
 socat                                                               x86_64                                         1.7.3.4-2.fc32                                            fedora                                             304 k

Transaction Summary
=======================================================================================================================================================================================================================================
Install  11 Packages

Total download size: 54 M
Installed size: 275 M
Downloading Packages:
(1/11): ethtool-5.8-1.fc32.x86_64.rpm                                                                                                                                                                  2.8 MB/s | 208 kB     00:00
(2/11): conntrack-tools-1.4.5-5.fc32.x86_64.rpm                                                                                                                                                        2.5 MB/s | 205 kB     00:00
(3/11): libnetfilter_cttimeout-1.0.0-15.fc32.x86_64.rpm                                                                                                                                                1.1 MB/s |  23 kB     00:00
(4/11): libnetfilter_cthelper-1.0.0-17.fc32.x86_64.rpm                                                                                                                                                 672 kB/s |  23 kB     00:00
(5/11): libnetfilter_queue-1.0.2-15.fc32.x86_64.rpm                                                                                                                                                    1.8 MB/s |  27 kB     00:00
(6/11): socat-1.7.3.4-2.fc32.x86_64.rpm                                                                                                                                                                6.9 MB/s | 304 kB     00:00
(7/11): 14bfe6e75a9efc8eca3f638eb22c7e2ce759c67f95b43b16fae4ebabde1549f3-cri-tools-1.13.0-0.x86_64.rpm                                                                                                  13 MB/s | 5.1 MB     00:00
(8/11): containernetworking-plugins-0.8.7-1.fc32.x86_64.rpm                                                                                                                                             11 MB/s |  11 MB     00:00
(9/11): d0ba40edfc0fdf3aeec3dd8e56c01ff0d3a511cc0012aabce55d9a83d9bf2b69-kubeadm-1.19.2-0.x86_64.rpm                                                                                                   9.4 MB/s | 8.3 MB     00:00
(10/11): d9d997cdbfd6562824eb7786abbc7f4c6a6825662d0f451793aa5ab8c4a85c96-kubelet-1.19.2-0.x86_64.rpm                                                                                                   12 MB/s |  19 MB     00:01
(11/11): b1b077555664655ba01b2c68d13239eaf9db1025287d0d9ccaeb4a8850c7a9b7-kubectl-1.19.2-0.x86_64.rpm                                                                                                  4.1 MB/s | 9.0 MB     00:02
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                   15 MB/s |  54 MB     00:03
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                               1/1
  Installing       : containernetworking-plugins-0.8.7-1.fc32.x86_64                                                                                                                                                              1/11
  Installing       : kubectl-1.19.2-0.x86_64                                                                                                                                                                                      2/11
  Installing       : cri-tools-1.13.0-0.x86_64                                                                                                                                                                                    3/11
  Installing       : socat-1.7.3.4-2.fc32.x86_64                                                                                                                                                                                  4/11
  Installing       : libnetfilter_queue-1.0.2-15.fc32.x86_64                                                                                                                                                                      5/11
  Installing       : libnetfilter_cttimeout-1.0.0-15.fc32.x86_64                                                                                                                                                                  6/11
  Installing       : libnetfilter_cthelper-1.0.0-17.fc32.x86_64                                                                                                                                                                   7/11
  Installing       : conntrack-tools-1.4.5-5.fc32.x86_64                                                                                                                                                                          8/11
  Running scriptlet: conntrack-tools-1.4.5-5.fc32.x86_64                                                                                                                                                                          8/11
  Installing       : ethtool-2:5.8-1.fc32.x86_64                                                                                                                                                                                  9/11
  Installing       : kubelet-1.19.2-0.x86_64                                                                                                                                                                                     10/11
  Installing       : kubeadm-1.19.2-0.x86_64                                                                                                                                                                                     11/11

~~snip;

```

packageをインストールしたら一度rebootしておいてください。selinuxをdisableにしておきたいので。

systemctlに登録して読み込みます。

```
[root@k8s-tmp ~]# systemctl enable --now kubelet
Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /usr/lib/systemd/system/kubelet.service.
```

## cgroup設定

### DockerのcgroupDriverをsystemdにする。

```
## Create /etc/docker directory.
mkdir /etc/docker

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart Docker
systemctl daemon-reload
systemctl restart docker
```


Dockerと同じcgroup Driverを使う必要があるので設定。cgroupとはkernelの機能でプロセスなどをグルーピングしてくれる機能らっし。
Cgroup Driverを別のを使いたかったり、指定したい場合には`/etc/default/kubelet`に`KUBELET_EXTRA_ARGS=--cgroup-driver=<value>`を保存すればいいらしい。
基本は自動的にkubeletが自動的に判別して設定してくれるらしい。

```
[murata@murata ~]$  docker info | grep -i cgroup
 Cgroup Driver: cgroupfs
```

Centos8を使う人は注意してください。


# kubeadmによる初期設定

[Creating a single control-plane cluster with kubeadm](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)

dnf updateなどはしてあるので飛ばします。

`kubeadm init <args>`をします。\<args\>にはオプションを入れられます。次の項目でCalico for policy and flannel (aka Canal)というネットワークプラグインを使うので、`--pod-network-cidr=10.244.0.0/16`を追加しておきます。後述のCanal側のマニフェストファイル側を変えてもいい。

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/

```
murata:~ $ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
W1002 15:23:13.043966    5853 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.2
[preflight] Running pre-flight checks
        [WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
        [WARNING Hostname]: hostname "k8s-tmp.dev.deroris.local" could not be reached
        [WARNING Hostname]: hostname "k8s-tmp.dev.deroris.local": lookup k8s-tmp.dev.deroris.local on 172.16.200.31:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-tmp.dev.deroris.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.203.203]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-tmp.dev.deroris.local localhost] and IPs [172.16.203.203 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-tmp.dev.deroris.local localhost] and IPs [172.16.203.203 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 14.502912 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-tmp.dev.deroris.local as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-tmp.dev.deroris.local as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 0qoe0u.s8exrc3ahdixzkzu
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.203.203:6443 --token 0qoe0u.s8exrc3ahdixzkzu \
    --discovery-token-ca-cert-hash sha256:7edc25f8a485b832c5b57808df6cc00a9cd8b8955f9349eab86762c6ed413118
```

上にあるように、設定ファイルをhomeディレクトリに持ってきます。

```
murata:~ $ mkdir -p $HOME/.kube
murata:~ $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
murata:~ $ sudo chown $(id -u):$(id -g) $HOME/.kube/config
murata:~ $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ながい。
    server: https://172.16.203.203:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: ながい
    client-key-data: ながい
```

これでkubectlで見れるようになるはず。

```
murata:~ $ kubectl get all -A
NAMESPACE     NAME                                                    READY   STATUS              RESTARTS   AGE
kube-system   pod/coredns-f9fd979d6-9bcmk                             0/1     ContainerCreating   0          2m57s
kube-system   pod/coredns-f9fd979d6-l4whm                             0/1     ContainerCreating   0          2m57s
kube-system   pod/etcd-k8s-tmp.dev.deroris.local                      1/1     Running             0          3m7s
kube-system   pod/kube-apiserver-k8s-tmp.dev.deroris.local            1/1     Running             0          3m7s
kube-system   pod/kube-controller-manager-k8s-tmp.dev.deroris.local   1/1     Running             0          3m7s
kube-system   pod/kube-proxy-nxcct                                    1/1     Running             0          2m57s
kube-system   pod/kube-scheduler-k8s-tmp.dev.deroris.local            1/1     Running             0          3m7s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  3m16s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   3m14s

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   3m14s

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   0/2     2            0           3m14s

NAMESPACE     NAME                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-f9fd979d6   2         2         0       2m57s
```


corednsが動けないみたい。PodのNetwork add-onを入れる。
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network


https://docs.projectcalico.org/getting-started/kubernetes/flannel/flannel


今回はCanalを採用！（なんとなく）を入れる。先ほどのkubeadm initでpod-network-cidrを別の値にした場合には、ymlファイルをDLして該当箇所を書き換えてください。きっと10.244.0.0/16って書いてあるところがあるはず。

```
murata:~ $ curl https://docs.projectcalico.org/manifests/canal.yaml -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  181k  100  181k    0     0   133k      0  0:00:01  0:00:01 --:--:--  133k
murata:~ $ kubectl apply -f ./canal.yaml
configmap/canal-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/canal-flannel created
clusterrolebinding.rbac.authorization.k8s.io/canal-calico created
daemonset.apps/canal created
serviceaccount/canal created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created

```

Canalが産まれようとしている！

```

murata:~ $ kubectl get all -A
NAMESPACE     NAME                                                    READY   STATUS              RESTARTS   AGE
kube-system   pod/calico-kube-controllers-c9784d67d-zj78k             0/1     Pending             0          16s
kube-system   pod/canal-xllfv                                         2/2     Running             0          16s
kube-system   pod/coredns-f9fd979d6-56l7w                             0/1     ContainerCreating   0          3m7s
kube-system   pod/coredns-f9fd979d6-zvwrj                             0/1     ContainerCreating   0          3m7s
kube-system   pod/etcd-k8s-tmp.dev.deroris.local                      1/1     Running             0          3m16s
kube-system   pod/kube-apiserver-k8s-tmp.dev.deroris.local            1/1     Running             0          3m16s
kube-system   pod/kube-controller-manager-k8s-tmp.dev.deroris.local   1/1     Running             0          3m16s
kube-system   pod/kube-proxy-8npdc                                    1/1     Running             0          3m7s
kube-system   pod/kube-scheduler-k8s-tmp.dev.deroris.local            1/1     Running             0          3m16s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  3m24s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   3m23s

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/canal        1         1         1       1            1           kubernetes.io/os=linux   17s
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   3m23s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   0/1     1            0           17s
kube-system   deployment.apps/coredns                   0/2     2            0           3m23s

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-c9784d67d   1         1         0       17s
kube-system   replicaset.apps/coredns-f9fd979d6                   2         2         0       3m7s

murata:~ $ kubectl get all -A
NAMESPACE     NAME                                                    READY   STATUS    RESTARTS   AGE
kube-system   pod/calico-kube-controllers-c9784d67d-zj78k             1/1     Running   0          57s
kube-system   pod/canal-xllfv                                         2/2     Running   0          57s
kube-system   pod/coredns-f9fd979d6-56l7w                             1/1     Running   0          3m48s
kube-system   pod/coredns-f9fd979d6-zvwrj                             1/1     Running   0          3m48s
kube-system   pod/etcd-k8s-tmp.dev.deroris.local                      1/1     Running   0          3m57s
kube-system   pod/kube-apiserver-k8s-tmp.dev.deroris.local            1/1     Running   0          3m57s
kube-system   pod/kube-controller-manager-k8s-tmp.dev.deroris.local   1/1     Running   0          3m57s
kube-system   pod/kube-proxy-8npdc                                    1/1     Running   0          3m48s
kube-system   pod/kube-scheduler-k8s-tmp.dev.deroris.local            1/1     Running   0          3m57s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  4m5s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   4m4s

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/canal        1         1         1       1            1           kubernetes.io/os=linux   58s
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   4m4s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           58s
kube-system   deployment.apps/coredns                   2/2     2            2           4m4s

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-c9784d67d   1         1         1       58s
kube-system   replicaset.apps/coredns-f9fd979d6                   2         2         2       3m48s
```

できた。

デフォルトではマスターノードはworkerにはなれないのでなれるようにしてあげます。

```
murata:~ $ kubectl taint nodes --all node-role.kubernetes.io/master-
node/k8s-tmp.dev.deroris.local untainted
```


# 動作確認

ためしにnginxをデプロイしてみます。

```
kind: Deployment
apiVersion: apps/v1
metadata:
  name: nginx-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-test
  template:
    metadata:
      labels:
        app: nginx-test
    spec:
      containers:
        - args:
          image: nginx:latest
          imagePullPolicy: IfNotPresent
          name: nginx-test
          ports:
            - containerPort: 9100
              protocol: TCP
```

```bash

murata:~ $ cat <<EOF | kubectl apply -f -
> kind: Deployment
> apiVersion: apps/v1
> metadata:
>   name: nginx-test
> spec:
>   replicas: 1
>   selector:
>     matchLabels:
>       app: nginx-test
>   template:
>     metadata:
>       labels:
>         app: nginx-test
>     spec:
>       containers:
>         - args:
>           image: nginx:latest
>           imagePullPolicy: IfNotPresent
>           name: nginx-test
>           ports:
>             - containerPort: 9100
>               protocol: TCP
> EOF
deployment.apps/nginx-test created
```


できた。

```
murata:~ $ kubectl get all
NAME                              READY   STATUS    RESTARTS   AGE
pod/nginx-test-79b49dfdf4-b8lpl   1/1     Running   0          66s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   18m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-test   1/1     1            1           66s

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-test-79b49dfdf4   1         1         1       66s
```

確認

```
murata:~ $ curl 10.244.0.6 -I
HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Fri, 02 Oct 2020 06:42:48 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 11 Aug 2020 14:50:35 GMT
Connection: keep-alive
ETag: "5f32b03b-264"
Accept-Ranges: bytes
```

ちゃんとレスポンスも受け取れる。

# ローカルのPCからいじれるようにする。

**開発用オレオレk8sだからadminコピーしちゃってるけど、本番ではちゃんとすること。**

`/etc/kubernetes/admin.conf` のファイルをローカルに持ってきて環境変数KUBECONFIGに食わせる。

`murata:~/.kube $ scp root@xxxx.local:/etc/kubernetes/admin.conf ~/.kube/oreore.local.config`

`export KUBECONFIG=~/.kube/oreore.local.config`

```
murata:~ $ kubectl get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-5ff9d6cc77-4zc9t   1/1     Running   0          3h5m
pod/nginx-5ff9d6cc77-fnsk5   1/1     Running   0          3h5m


NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes       ClusterIP   10.96.0.1       <none>        443/TCP        3h17m
service/nginx-nodeport   NodePort    10.108.248.35   <none>        80:30222/TCP   3h5m


NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   2/2     2            2           3h5m

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-5ff9d6cc77   2         2         2       3h5m
```

必要ならば、~/.bash_profileなどに記載しておきましょう。

# ローカルPCからデプロイする。

※ここから先はkubeのバージョン関係ないので1.19ではやっていません。

**開発用オレオレk8sだからやってるけど、本番ではちゃんとすること。**

Docker のリポジトリがサーバのローカルにある以上、サーバにDockerイメージをpushしなければならない。
上記で構築したサーバにSSHして云々するのはめんどくさい。でもDockerレジストリを構築するのも嫌だ。

まずDockerを別ホストからいじれるように鍵を用意。
参考: [Docker Engineを立ててクライアントからいじる](https://qiita.com/murata-tomohide/items/526cadd39bf415264af9)

`/etc/systemd/system/multi-user.target.wants/docker.service` の ExecStartに鍵たちと-Hで`tcp://0.0.0.0:2376`を追加します。

```
ExecStart=/usr/bin/dockerd --tlsverify --tlscacert=/etc/docker/ca.pem --tlscert=/etc/docker/server-cert.pem --tlskey=/etc/docker/server-key.pem -H=tcp://0.0.0.0:2376 -H=fd:// --containerd=/run/containerd/containerd.sock
```

`sudo systemctl daemon-reload`して` sudo systemctl restart docker.service` します。


ローカルPCに`export DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://[↑のサーバのIPアドレス]:2376` します。

`docker container ls`や`docker info`でつながっているか確認します。

```
murata:~ $ docker container ls
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
0d08ddfa494d        4fcdc789ef1a           "start_runit"            10 seconds ago      Up 9 seconds                            k8s_calico-node_calico-node-x7nrf_kube-system_fbf69ba9-e31e-4258-83c3-1918aca0229c_60
d2e5b79aa6c5        eb516548c180           "/coredns -conf /etc…"   5 minutes ago       Up 5 minutes                            k8s_coredns_coredns-5c98db65d4-w2lsq_kube-system_f93be9ea-4151-4a21-aa6e-bc876a1ad4b0_2
086373c963d4        k8s.gcr.io/pause:3.1   "/pause"                 5 minutes ago       Up 5 minutes                            k8s_POD_coredns-5c98db65d4-w2lsq_kube-system_f93be9ea-4151-4a21-aa6e-bc876a1ad4b0_8
6c8ad0c23551        nginx                  "nginx -g 'daemon of…"   6 minutes ago       Up 6 minutes                            k8s_nginx_nginx-5ff9d6cc77-4zc9t_default_20a48970-c693-40f3-a11c-9771f6cb3383_2
41a8610f7462        nginx                  "nginx -g 'daemon of…"   6 minutes ago       Up 6 minutes                            k8s_nginx_nginx-5ff9d6cc77-fnsk5_default_5499cc62-bc8b-41b4-a862-47d9c777eb1c_2
59ad16a25a50        k8s.gcr.io/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_nginx-5ff9d6cc77-4zc9t_default_20a48970-c693-40f3-a11c-9771f6cb3383_2
e7024d8c28cf        214ddcd2a33e           "/usr/bin/kube-contr…"   6 minutes ago       Up 6 minutes                            k8s_calico-kube-controllers_calico-kube-controllers-7bd78b474d-7zqt7_kube-system_5be65856-7bee-493d-9e5b-7232f1b402dc_2
e6a0e2a9d930        k8s.gcr.io/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_calico-kube-controllers-7bd78b474d-7zqt7_kube-system_5be65856-7bee-493d-9e5b-7232f1b402dc_10
33c307d9b32b        k8s.gcr.io/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_nginx-5ff9d6cc77-fnsk5_default_5499cc62-bc8b-41b4-a862-47d9c777eb1c_2
0b4f92e99fd6        eb516548c180           "/coredns -conf /etc…"   6 minutes ago       Up 6 minutes                            k8s_coredns_coredns-5c98db65d4-clcsv_kube-system_7a548701-e6f2-4dc3-8f62-e6c1d776cedb_2
6fe27b5b0c2a        k8s.gcr.io/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_coredns-5c98db65d4-clcsv_kube-system_7a548701-e6f2-4dc3-8f62-e6c1d776cedb_8
2062006035f3        34a53be6c9a7           "kube-apiserver --ad…"   6 minutes ago       Up 6 minutes                            k8s_kube-apiserver_kube-apiserver-derori.cloud.oreore.local_kube-system_c05b675d6985ed143dd49b1a689ad67c_3
e96b0325f13f        2c4adeb21b4f           "etcd --advertise-cl…"   6 minutes ago       Up 6 minutes                            k8s_etcd_etcd-derori.cloud.oreore.local_kube-system_0487325602f4b35b7a2cbd7fe4b865e7_2
c22ea2474db9        167bbf6c9338           "/usr/local/bin/kube…"   6 minutes ago       Up 6 minutes                            k8s_kube-proxy_kube-proxy-kw622_kube-system_e5ccb43d-d774-401e-87d9-15849e2db860_3
c770b18181bd        88fa9cb27bd2           "kube-scheduler --bi…"   6 minutes ago       Up 6 minutes                            k8s_kube-scheduler_kube-scheduler-derori.cloud.oreore.local_kube-system_abfcb4f52e957b11256c1f6841d49700_2
955de466e5d2        9f5df470155d           "kube-controller-man…"   6 minutes ago       Up 6 minutes                            k8s_kube-controller-manager_kube-controller-manager-derori.cloud.oreore.local_kube-system_6b98d73b8fe8352c3d11c50e86a2556e_3
5ad55d36f759        k8s.gcr.io/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_calico-node-x7nrf_kube-system_fbf69ba9-e31e-4258-83c3-1918aca0229c_2
a9da827e59ab        k8s.gcr.io/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_etcd-derori.cloud.oreore.local_kube-system_0487325602f4b35b7a2cbd7fe4b865e7_2
f85d3da0d571        k8s.gcr.io/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_kube-controller-manager-derori.cloud.oreore.local_kube-system_6b98d73b8fe8352c3d11c50e86a2556e_3
b8e74e4b19b7        k8s.gcr.io/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_kube-scheduler-derori.cloud.oreore.local_kube-system_abfcb4f52e957b11256c1f6841d49700_2
588f58978a3e        k8s.gcr.io/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_kube-proxy-kw622_kube-system_e5ccb43d-d774-401e-87d9-15849e2db860_3
2bfc7879f0dd        k8s.gcr.io/pause:3.1   "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_kube-apiserver-derori.cloud.oreore.local_kube-system_c05b675d6985ed143dd49b1a689ad67c_3
murata:~ $ docker info
Containers: 46
 Running: 23
 Paused: 0
 Stopped: 23
Images: 13
Server Version: 19.03.1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 5.2.7-200.fc30.x86_64
Operating System: Fedora 30 (Thirty)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.66GiB
Name: derori.cloud.oreore.local
ID: 74UT:MSBM:XOMG:MJJM:PXY2:O4MM:GLAD:6HV4:C3NM:OLWA:ULVM:NIAI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
```

これで、自身のローカルから直接Dockerへのアクセスできるようになったので、ローカルから`kubectl apply`できるはずだ！

適当にDockerfile作ってbiuildしてapplyしてみます。


```
murata:~/tmp/ppp $ cat Dockerfile.dev
FROM nginx:1.15
ADD index.html /usr/share/nginx/html/

murata:~/tmp/ppp $ cat index.html
pipirupi-!!!
murata:~/tmp/ppp $ cat k8s.yaml
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: pipirupi
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pipirupi
  template:
    metadata:
      labels:
        app: pipirupi
    spec:
      containers:
      - args:
        image: pipirupi:v
        imagePullPolicy: IfNotPresent
        name: pipirupi
        ports:
        - containerPort: 80
          protocol: TCP
---
kind: Service
apiVersion: v1
metadata:
  name: pipirupi
spec:
  # externalTrafficPolicy: Local
  type: NodePort
  ports:
  - name: "http-port"
    protocol: TCP
    port: 80
    targetPort: 80
  selector:
    app: pipirupi

```


```
murata:~/tmp/ppp $ docker build -t pipirupi:v . -f Dockerfile.dev
Sending build context to Docker daemon  9.728kB
Step 1/2 : FROM nginx:1.15
 ---> 53f3fd8007f7
Step 2/2 : ADD index.html /usr/share/nginx/html/
 ---> 189cc5fa6d2f
Successfully built 189cc5fa6d2f
Successfully tagged pipirupi:v

murata:~/tmp/ppp $ kubectl apply -f k8s.yaml
deployment.extensions/pipirupi configured
service/pipirupi configured

murata:~/workspace/deroris.github/infrastructure/config/aws (update-countrade-deployer-lambda-policy=) $ kubectl get all  -o wide
NAME                            READY   STATUS      RESTARTS   AGE     IP               NODE                         NOMINATED NODE   READINESS GATES
pod/hello-world                 0/1     Completed   0          56m     192.168.219.80   derori.cloud.oreore.local   <none>           <none>
pod/nginx-5ff9d6cc77-4zc9t      1/1     Running     2          4h23m   192.168.219.78   derori.cloud.oreore.local   <none>           <none>
pod/nginx-5ff9d6cc77-fnsk5      1/1     Running     2          4h23m   192.168.219.76   derori.cloud.oreore.local   <none>           <none>
pod/pipirupi-7f987ff544-q7kx7   1/1     Running     0          12s     192.168.219.83   derori.cloud.oreore.local   <none>           <none>


NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE     SELECTOR
service/kubernetes       ClusterIP   10.96.0.1        <none>        443/TCP        4h35m   <none>
service/nginx-nodeport   NodePort    10.108.248.35    <none>        80:30222/TCP   4h23m   run=nginx
service/pipirupi         NodePort    10.105.153.120   <none>        80:32671/TCP   14m     app=pipirupi


NAME                       READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES         SELECTOR
deployment.apps/nginx      2/2     2            2           4h23m   nginx        nginx:latest   run=nginx
deployment.apps/pipirupi   1/1     1            1           14m     pipirupi     pipirupi:v     app=pipirupi

NAME                                  DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES            SELECTOR
replicaset.apps/nginx-5ff9d6cc77      2         2         2       4h23m   nginx        nginx:latest      pod-template-hash=5ff9d6cc77,run=nginx
replicaset.apps/pipirupi-564bff47cb   0         0         0       12m     pipirupi     pipirupi:vvv      app=pipirupi,pod-template-hash=564bff47cb
replicaset.apps/pipirupi-6bd5b68657   0         0         0       14m     pipirupi     pipirupi:latest   app=pipirupi,pod-template-hash=6bd5b68657
replicaset.apps/pipirupi-7f987ff544   1         1         1       12s     pipirupi     pipirupi:v        app=pipirupi,pod-template-hash=7f987ff544


murata:~/workspace/deroris.github/infrastructure/config/aws (update-countrade-deployer-lambda-policy=) $ curl http://oreore.k8s.local:32671
pipirupi-!!!

```

すごいDockerfile作るのに試行錯誤しちゃったけどこんな感じ。



# まとめ。

まだIngress（ロードバランサ）の設定をしていないため外からの通信はできないけど、kubeadmを使って構築はこんな感じになる。クラスタにサーバを追加する際にはkubeadm joinのコマンドをやれば勝手につないでくれる。