Project: OpenYurt (Provisioning / Automation & Configuration)
File: 20221115-static-pod-upgrade-model.md

title: Static Upgrade Model authors: "" reviewers: "" creation-date: last-updated: status: A table of contents is helpful for quickly jumping to sections of a proposal and for highlighting any additional information provided beyond the standard proposal template. a table of contents from markdown are available. - - - - - - - - - - - - Refer to the . If this proposal adds new terms, or defines some, make the changes to the book's glossary when in PR stage. In this proposal, we hope to put forward a new method to control the upgrade of static pods automatically, so we add a new CRD(StaticPod) to control the upgrade of static pods. Meanwhile, we provide two upgrade strategy for static pod: AdvancedRollingUpdate and OTA. We all know that static pods are managed directly by the kubelet daemon on a specific node, without the API server observing them. Generally speaking, if we want to upgrade static pods, we may modify or replace the manifests under `staticPodPath`(Usually it's `/etc/kubernetes/manifests`) on a specific node. When the number of edge static pods becomes very large, this can be a truly tedious and painful task. Therefore, in cloud-edge collaboration scenario, it is very necessary to provide a proper static pod upgrade model like other workloads, such as DaemonSet. Provide common upgrade model for upgrading static pod. Provide AdvancedRollingUpdate upgrade model for static pod in order to get rid of blocking upgrade by notReady nodes. Provide OTA upgrade model for static pod when end user hope to control the time of upgrade. Systemd service(like kubelet) upgrade will not be covered. Do not consider the details of the static pod installation. ```go // StaticPodUpgradeStrategy defines a strategy to upgrade a static pod. type StaticPodUpgradeStrategy struct { // Type of Static Pod upgrade. Can be "AdvancedRollingUpdate" or "OTA". // +optional Type StaticPodUpgradeStrategyType // AdvancedRollingUpdate upgrade config params. Present only if type = "AdvancedRollingUpdate". MaxUnavailable intstr.IntOrString } // StaticPodUpgradeStrategyType is a strategy according to which a static pod gets upgraded. type StaticPodUpgradeStrategyType string const ( // AdvancedRollingUpdateStaticPodUpgradeStrategyType - Replace the old static pod by new ones using AdvancedRollingUpdate upgrade i.e. replace them on each node one after the other. AdvancedRollingUpdateStaticPodUpgradeStrategyType StaticPodUpgradeStrategyType = "AdvancedRollingUpdate" // OTAStaticPodUpgradeStrategyType - Replace the old static pod only when it's killed OTAStaticPodUpgradeStrategyType StaticPodUpgradeStrategyType = "OTA" ) // StaticPodSpec defines the desired state of StaticPod type StaticPodSpec struct { // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster // Important: Run "make" to regenerate code after modifying this file // StaticPodName indicates the static pod desired to be upgraded. StaticPodName string `json:"staticPodName` // StaticPodManifest indicates the Static Pod desired to be upgraded. The corresponding // manifest file name is `ManifestName.yaml`. StaticPodManifest string `json:"staticPodManifest"` // An upgrade strategy to replace existing static pods with new ones. // +optional UpgradeStrategy StaticPodUpgradeStrategy `json:"upgradeStrategy"` // An object that describes the desired upgrade static pod. Template

`json:"template"` } type StaticPodConditionType string const ( // StaticPodUpgradeSuccess means static pods on all nodes have been upgraded to the latest version StaticPodUpgradeSuccess StaticPodConditionType = "Success" // StaticPodUpgradeExecuting means static pods upgrade task is in progress StaticPodUpgradeExecuting StaticPodConditionType = "Upgrading" // StaticPodUpgradeFailed means that exist pods failed to upgrade during the upgrade process StaticPodUpgradeFailed StaticPodConditionType = "Failed" ) // StaticPodCondition describes the state of a StaticPodCondition at a certain point. type StaticPodCondition struct { // Type of StaticPod condition. Type StaticPodConditionType `json:"type,omitempty"` // Status of the condition, one of True, False, Unknown. Status v1.ConditionStatus `json:"status,omitempty"` // Last time the condition transitioned from one status to another. LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"` // The reason for the condition's last transition. Reason string `json:"reason,omitempty"` // A human-readable message indicating details about the transition. Message string `json:"message,omitempty"` } // StaticPodStatus defines the observed state of StaticPod type StaticPodStatus struct { // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster // Important: Run "make" to regenerate code after modifying this file // The number of static pods that should be upgraded. DesiredNumber int32 // The number of static pods that have been upgraded. UpgradedNumber int32 // Represents the latest available observations of StaticPod's current state. Conditions []StaticPodCondition } ``` <div align="center"> <img src="../img/static-pod/process.png" title="Static-Pod-Upgrade"> </div> Check the existence of static pod manifest. Replace static pod manifest. Verify the new static pod status. Controller generate new static pod manifest from CR. Controller creates a ConfigMap to store the new manifest. Controller creates an upgrade-task pod which refers to the ConfigMap. AdvancedRollingUpdate Upgrade is the default static pod upgrade strategy we want to provide. The core capability of AdvancedRollingUpdate Upgrade is to automate the upgrade process for all selected static pods on ready nodes after user submits an upgrade request. This strategy ensures that ready nodes can be upgraded smoothly even when some edge nodes are disconnected from the cloud. <div align="center"> <img src="../img/static-pod/auto.png" title="Static-Pod-Upgrade"> </div> AdvancedRollingUpdate Upgrade strategy supports rolling update and the configuration method is the same with native K8s workload. `Not-ready` nodes will be ignored during upgrade tasks in AdvancedRollingUpdate Upgrade strategy. However, the controller will monitor the status of the `not-ready` node and automatically performs upgrade operation once the node turn ready. OTA upgrade strategy focuses on scenarios where edge side users need to autonomously control the upgrade of static pods. The strategy is accomplished by controller together with YurtHub component. The overall implementation process is the same as . Controller will set the static pod condition `PodNeedUpgrade` to true when user submits a new upgrade request. If the node is `ready`, controller will create an upgrade-task pod to store the static pod manifest in advance, with file name `xxx.yaml.upgrade`. YurtHub will provide two REST APIs for upgrade, just the same as DaemonSet. `GET /pods` This API allows you to get information about the pods on the node. `POST /openyurt.io/v1/namespaces/{ns}/pods/{podname}/upgrade` This API will replace the old static pod manifest with the new one, if it exists. And if the new manifest(`xxx.yaml.upgrade`) doesn't exist, OpenYurt will request the latest version from the cloud. When we adopt this upgrade model, we will record the latest version of static pod in the cluster. So when joining nodes to the cluster we can get the required static pod manifest directly from the cloud. Currently, OpenYurt provides command `yurtadm join` to join nodes. Perhaps we need to adapt this command to support automatically deploying a static pod when joining nodes.