Ubuntu で Kubernetes 三昧その４

[Ubuntu で Kubernetes 三昧その３](https://qiita.com/intrajp/items/9bba127da61a91f23a83)の続きです。

kubeadm, kubelet and kubectlをインストールします。

```
@masternode1:~$ sudo apt-get update
# apt-transport-https may be a dummy package; if so, you can skip that package
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

@masternode1:~$ # If the directory `/etc/apt/keyrings` does not exist, it should be created before the curl command, read the note below.
# sudo mkdir -p -m 755 /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
```
apt レポジトリをインストールします。
```
$ sudo su -
root@masternode1:~#
# This overwrites any existing configuration in /etc/apt/sources.list.d/kubernetes.list
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
```
kubelet, kubeadm と kubectl をインストールします。

```
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
```
kubelet Cgroup ドライバを設定します。

```
# vim kubeadm-config.yaml
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta4
kubernetesVersion: v1.21.0
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
```
上記は、workerノードでも実行します（略）。

次は不明。
```
# kubeadm init --config kubeadm-config.yaml
(表示)
this version of kubeadm only supports deploying clusters with the control plane version >= 1.32.0. Current version: v1.21.0
To see the stack trace of this error execute with --v=5 or higher
```


マスターノードで、kubeadm init を実行します。
```
# kubeadm init

[init] Using Kubernetes version: v1.33.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local masternode1] and IPs [10.96.0.1 10.0.2.15]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost masternode1] and IPs [10.0.2.15 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost masternode1] and IPs [10.0.2.15 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.003047761s
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://10.0.2.15:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-controller-manager is healthy after 2.322241118s
[control-plane-check] kube-scheduler is healthy after 2.50305751s
[control-plane-check] kube-apiserver is healthy after 4.502015627s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node masternode1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node masternode1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: lkutaq.538tayvesh8vu52r
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.2.15:6443 --token lkutaq.538tayvesh8vu52r \
	--discovery-token-ca-cert-hash sha256:40f3948e305bacc8c8f2b6e17b7108b798f63268b2d2727a80676d7a12449e53 
```

マスターノードです。表示されたコマンドを実行します。
```
$  mkdir -p $HOME/.kube
$  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$  sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
マスターノードです。このコマンドを実行します。
```
@masternode1:~$ kubectl get nodes
NAME          STATUS     ROLES           AGE     VERSION
masternode1   NotReady   control-plane   4m45s   v1.33.2
```
ワーカーノードです。このコマンドを実行します。
```
@workernode1:~# kubeadm join 10.0.2.15:6443 --token lkutaq.538tayvesh8vu52r \
        --discovery-token-ca-cert-hash sha256:40f3948e305bacc8c8f2b6e17b7108b798f63268b2d2727a80676d7a12449e53 
[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.00393386s
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
```
マスターノードです。クラスタに worker node が追加されたかを確認します。
```
@masternode1:~$ kubectl get nodes
NAME          STATUS     ROLES           AGE   VERSION
masternode1   NotReady   control-plane   78s   v1.33.2
workernode1   NotReady   <none>          11s   v1.33.2
```
マスターノードです。kube-system ネームスペースで、クラスタの Pod を確認します。
```
$ kubectl get pods -n kube-system
NAME                                  READY   STATUS    RESTARTS   AGE
coredns-674b8bbfcf-44dwf              0/1     Pending   0          16h
coredns-674b8bbfcf-ppcj7              0/1     Pending   0          16h
etcd-masternode1                      1/1     Running   2          16h
kube-apiserver-masternode1            1/1     Running   2          16h
kube-controller-manager-masternode1   1/1     Running   2          16h
kube-proxy-9n8hv                      1/1     Running   0          16h
kube-proxy-km2s7                      1/1     Running   0          16h
kube-scheduler-masternode1            1/1     Running   2          16h
```

CNI (Container Network Interface) をインストールしていないために、coredns Pod が立ち上がっていないのがわかります。なお、etcd, apiserver, controller-manager, kube-proxy, scheduler の Pod は立ち上がっています。
なお、ワーカーノードから kubectl コマンドを実行するとエラーになります。


[Network Plugins](https://kubernetes.io/docs/concepts/cluster-administration/networking/#how-to-implement-the-kubernetes-network-model)をインストールします。
サポートされているのは、[これらのプラグイン](https://kubernetes.io/docs/concepts/cluster-administration/addons/#networking-and-network-policy)になります。

[Ubuntu で Kubernetes 三昧その５-１(Weave Net CNI)](https://qiita.com/intrajp/items/d8bc58ed689a70b64ca7)に続く。

参考：

https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/