Project: OpenYurt (Provisioning / Automation & Configuration)
File: 20221108-yurtadm-join-refactoring.md

title: Yurtadm Join Refactoring authors: "@YTGhost" reviewers: "@rambohe-ch" creation-date: 2022-11-08 last-updated: 2022-11-08 status: provisional <!-- END Remove before PR --> - - - - - - - - - - - `yurtadm join` references `k8s.io/kubernetes` dependencies, especially `k8s.io/kubernetes/cmd/kubeadm`. It should be refactor to remove the dependencies. This proposal wants to import `kubeadm` binary file to replace the code that was imported. Compare to import code, using `kubeadm` binary directly has the following advantages: It's very easy to follow K8s to update kubeadm version, do not need to maintain the code of`kubeadm join`in OpenYurt. reduce the complexity of`yurtadm join` After refactoring, the original join functionality can be maintained by calling `kubeadm join` and some of custom operations required by openyurt. Make the code subsequent maintainability and expandability more guaranteed. Abstract the interface, kubeadm join as one of the implementations. Refactor `yurtadm reset` and so on. As a developer, I can understand the `yurtadm join` process more clearly due to the reduced complexity of the `yurtadm join` part of the code The process of yurtadm join is divided into five phases: init joindata prepare preflight joinnode postcheck Initializes the join data Call `RetrieveBootstrapConfig` and change the server address to the master IP address This phase focuses on initializing the system's environment in `/pkg/yurtadm/cmd/join/phases/prepare.go`, with the following actions: Clean up the `/etc/kubernetes/manifests` folder, i.e. clean up the static Pod manifest folder Turn on packet forwarding (SetIpv4Forward) Turn on `bridge-nf-call-iptables` (SetBridgeSetting) Turn off SELinux (SetSELinux) Check if Kubelet and cni-plugin are installed, if not, they will be installed (CheckAndInstallKubelet) Configure the kubelet service, i.e. configure `/etc/systemd/system/kubelet.service` (SetKubeletService) Configure the kubelet startup parameters, i.e., configure `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` (SetKubeletUnitConfig) Change the connection address to the yurthub listener address (http://127.0.0.1:10261) i.e. configure `/etc/kubernetes/kubelet.conf` (SetKubeletConfigForNode) Configure `/etc/kubernetes/pki/ca.crt` (SetKubeletCaCert) This phase is mainly for preflight checks, which include checking some files, folders and environment configuration work. The relevant code for the checks is a direct call to the `RunJoinNodeChecks` function in the kubeadm source code. The main thing to do in this phase is to start the kubelet, as follows: Get the kubelet configuration from the control plane node, parse it and write it to `/var/lib/kubelet/config.yaml` configure `/var/lib/kubelet/kubeadm-flags.env` Generate the yaml file `/etc/kubernetes/manifests/yurthub.yaml` for the YurtHub static Pod to start YurtHub when the kubelet starts Try to start the kubelet The main thing to do in this phase is to check the health of the components launched on the node, mainly including: the health status of the kubelet the health status of the YurtHub static Pod adding Annotation to the node: `kubeadm.alpha.kubernetes.io/cri-socket` The kubeadm join process is divided into the following phases: preflight kubelet-start This phase mainly performs some preflash checks logic, mainly

basic checks, i.e. `RunJoinNodeChecks` Get the `InitCfg`, which also includes a call to `RetrieveBootstrapConfig`, which will be used in the `kubelet-start` phase This phase will mainly start the kubelet, the main process is. Write the configuration of `tlsBootstrapCfg` to `/etc/kubernetes/bootstrap-kubelet.conf` Take the `CertificateAuthorityData` from `tlsBootstrapCfg` and write it to `/etc/kubernetes/pki/ca.crt`. Check if there is a node in the cluster with the same name as the node you want to join and in Ready state (the note says that an error will occur if there is a control-plane node with the same name) Try to stop the kubelet Configure `/var/lib/kubelet/config.yaml` Configure `/var/lib/kubelet/kubeadm-flags.env` Try to start the kubelet Convert `/etc/kubernetes/bootstrap-kubelet.conf` to `/etc/kubernetes/kubelet.conf`. Add Annotation to the node: `kubeadm.alpha.kubernetes.io/cri-socket` Delete `/etc/kubernetes/bootstrap-kubelet.conf`. Yurtadm will do some operations to initialize the system environment, such as turning on packet forwarding, etc. Yurtadm will change the connection address in `/etc/kubernetes/kubelet.conf` to the yurthub listener address `http://127.0.0.1:10261`. Preflight checks will additionally ignore `FileAvailable--etc-kubernetes-kubelet.conf` and `FileAvailable--etc-kubernetes-pki-ca.crt` errors Configure `/var/lib/kubelet/kubeadm-flags.env` with some additional parameters, such as setting `--rotate-certificates` to `false`, etc. Generate the yaml file `/etc/kubernetes/manifests/yurthub.yaml` for the YurtHub static Pod Additionally check the health status of YurtHub Modified the Server address when calling `RetrieveBootstrapConfig`. According to the above analysis, it can be divided into four phases as follows. init joindata: mainly to initialize the data needed for the later stages prepare: mainly to initialize the system environment joinnode: mainly to call kubeadm join, before the call you need to determine whether the current environment has kubeadm, if not, you need to download postcheck: check the health status of the startup component Consistent with yurtadm join Clean up the `/etc/kubernetes/manifests` folder, i.e. clean up the static Pod manifest folder Turn on packet forwarding (SetIpv4Forward) Turn on `bridge-nf-call-iptables` (SetBridgeSetting) Turn off SELinux (SetSELinux) Check if Kubelet and cni-plugin are installed, if not, they will be installed (CheckAndInstallKubelet) Configure the kubelet service, i.e. configure `/etc/systemd/system/kubelet.service` (SetKubeletService) Configure the kubelet startup parameters, i.e., configure `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` (SetKubeletUnitConfig) Change the connection address to the yurthub listening address (http://127.0.0.1:10261) i.e. configure `/etc/kubernetes/kubelet.conf` (SetKubeletConfigForNode) Configure `/var/lib/kubelet/discovery.conf` for configuring `discovery` Configure `/var/lib/kubelet/kubeadm-join.conf` to configure additional required `KubeletExtraArgs`, the path to `discovery.conf` will also be added to it Use `kubeadm join` and read `/var/lib/kubeadm-join.conf` with `--config` to configure the additional required `KubeletExtraArgs` and `discovery` `kubeadm-join.conf` example: ```yaml apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration discovery: file: kubeConfigPath: /var/lib/kubelet/discovery.conf tlsBootstrapToken: xfn7gd.0ztozki1f5cif7h1 nodeRegistration: criSocket: /var/run/dockershim.sock name: vm-16-6-centos ignorePreflightErrors: FileAvailable--etc-kubernetes-kubelet.conf kubeletExtraArgs: rotate-certificates: "false" pod-infra-container-image: registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 node-labels: openyurt.io/is-edge-worker=true network-plugin: cni ``` `discovery.conf` example: ```yaml apiVersion: v1 clusters: cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM2VENDQWRHZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQ0FYRFRJeU1UQXlOekUxTURBeU1sb1lEekl4TWpJeE1EQXpNVFV3TURJeVdqQVZNUk13RVFZRApWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBCnlFWlpSQUhmdWRIL05nZElKaXJnNW1IcllzT093UHdhWDR6dFJaRG4zdFB4ZTYxY0hBdkhGT0h3M0xqKzFRSmMKWWE4OGZ1NFF6cUh2TnBGeTlZeTE3YVo3c1JuQUhTemcvSkVvcjlxQlVJakN0WlUxMTN5dlRMSDZRZnZBTWl4eQpoTkdIMWRMWnV5SnY5OUUweklMUUNrWGN5SzF6a1ZFeWZKcmZaS2lTalYwQndteWl1QjUwcktmc0pZc21rS2ZICnp0SEtzWmFPaElwcmd3NG02SmttSUZKQjBFTlBhaktKNjRLWng3bXUzOVIySlJNc3A0WXpieWJTQnNacnNGcUIKbnZhTHRlVk9QcHVyK1hYUkptUUFvbkc3Zkl5QU9GY21PTmovekNHbzhpb04xd2NFVU1paWhFaVN6UzVUM2wwQgpwUUdOV0JGVUFyMmZ0NTRaV2syUnd3SURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQXFRd0R3WURWUjBUCkFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVU2Z0RhTUZEaURQaU90eGcxSktxSk1YZkRXV1l3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFHNzAvU3hRTlFZTjJKUC9paUllN2kxdXAyT1VZUXlXYjlqMVF6dnlkNHdRTElMTQpvUzZHZVplblBYRlRnemVFZE5uaUsxcEV6b0JJMUNOa0N4S01QVUx3RWJzRlVDb1M3T2JVTWNoQkw2MG0xOC9GCnFGRG04QVFqRUdXTllOeHVuRXlFVUpUQXBibStndTdCYlIvZzR2Uk5UTWZ3NzFiSzlGci9iUmdBMHZEVEdhK0QKU29RekxKZms2UTRKQW41MHFGRDlCdVViV3ExSkY4eWxqc1BiMnlqWlVqSkhaWmd6cTRJL2FaYjZ5TjJ2MTAxTQpiSHRRR2crUDZFb05rWWZDR25zRGZNYnJhYnZpK3VyTUNNcmZzLzhCQnlUU3c3SjB6UEdkbTQ5b1l1SXE1YnVqCjdJVmR1eE9JempnU2hqSm91bjdpWWpFQXRnR3U3REgzYVd4Qm9tZz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= server: https://apiserver.cluster.local:6443 name: "" contexts: null current-context: "" kind: Config preferences: {} users: null ``` Check the health status of kubelet Check the health status of YurtHub static Pods