# Cluster Administration

This is the multi-page printable view of this section. Click here to print.Return to the regular view of this page.Cluster AdministrationLower-level detail relevant to creating or administering a Kubernetes cluster.1: Node Shutdowns2: Swap memory management3: Node Autoscaling4: Certificates5: Cluster Networking6: Admission Webhook Good Practices7: Good practices for Dynamic Resource Allocation as a Cluster Admin8: Logging Architecture9: Compatibility Version For Kubernetes Control Plane Components10: Metrics For Kubernetes System Components11: Metrics for Kubernetes Object States12: System Logs13: Traces For Kubernetes System Components14: Proxies in Kubernetes15: API Priority and Fairness16: Installing Addons17: Coordinated Leader ElectionThe cluster administration overview is for anyone creating or administering a Kubernetes cluster. It assumes some familiarity with core Kubernetes concepts.Planning a clusterSee the guides in Setup for examples of how to plan, set up, and configure Kubernetes clusters. The solutions listed in this article are called distros.Note:Not all distros are actively maintained. Choose distros which have been tested with a recent version of Kubernetes.Before choosing a guide, here are some considerations:Do you want to try out Kubernetes on your computer, or do you want to build a high-availability, multi-node cluster? Choose distros best suited for your needs.Will you be using a hosted Kubernetes cluster, such as Google Kubernetes Engine, or hosting your own cluster?Will your cluster be on-premises, or in the cloud (IaaS)? Kubernetes does not directly support hybrid clusters. Instead, you can set up multiple clusters.If you are configuring Kubernetes on-premises, consider which networking model fits best.Will you be running Kubernetes on "bare metal" hardware or on virtual machines (VMs)?Do you want to run a cluster, or do you expect to do active development of Kubernetes project code? If the latter, choose an actively-developed distro. Some distros only use binary releases, but offer a greater variety of choices.Familiarize yourself with the components needed to run a cluster.Managing a clusterLearn how to manage nodes.Read about Node autoscaling.Learn how to set up and manage the resource quota for shared clusters.Securing a clusterGenerate Certificates describes the steps to generate certificates using different tool chains.Kubernetes Container Environment describes the environment for Kubelet managed containers on a Kubernetes node.Controlling Access to the Kubernetes API describes how Kubernetes implements access control for its own API.Authenticating explains authentication in Kubernetes, including the various authentication options.Authorization is separate from authentication, and controls how HTTP calls are handled.Using Admission Controllers explains plug-ins which intercepts requests to the Kubernetes API server after authentication and authorization.Admission Webhook Good Practices provides good practices and considerations when designing mutating admission webhooks and validating admission webhooks.Using Sysctls in a Kubernetes Cluster describes to an administrator how to use the sysctl command-line tool to set kernel parameters .Auditing describes how to interact with Kubernetes' audit logs.Securing the kubeletControl Plane-Node communicationTLS bootstrappingKubelet authentication/authorizationOptional Cluster ServicesDNS Integration describes how to resolve a DNS name directly to a Kubernetes service.Logging and Monitoring Cluster Activity explains how logging in Kubernetes works and how to implement it.1 - Node ShutdownsIn a Kubernetes cluster, a node can be shut down in a planned graceful way or unexpectedly because of reasons such as a power outage or something else external. A node shutdown could lead to workload failure if the node is not drained before the shutdown. A node shutdown can be either graceful or non-graceful.Graceful node shutdownThe kubelet attempts to detect node system shutdown and terminates pods running on the node.Kubelet ensures that pods follow the normal pod termination process during the node shutdown. During node shutdown, the kubelet does not accept new Pods (even if those Pods are already bound to the node).Enabling graceful node shutdownFEATURE STATE: Kubernetes v1.21 [beta] (enabled by default: true)On Linux, the graceful node shutdown feature is controlled with the GracefulNodeShutdown feature gate which is enabled by default in 1.21.Note:The graceful node shutdown feature depends on systemd since it takes advantage of systemd inhibitor locks to delay the node shutdown with a given duration.FEATURE STATE: Kubernetes v1.34 [beta] (enabled by default: true)On Windows, the graceful node shutdown feature is controlled with the WindowsGracefulNodeShutdown feature gate which is introduced in 1.32 as an alpha feature. In Kubernetes 1.34 the feature is Beta and is enabled by default.Note:The Windows graceful node shutdown feature depends on kubelet running as a Windows service, it will then have a registered service control handler to delay the preshutdown event with a given duration.Windows graceful node shutdown can not be cancelled.If kubelet is not running as a Windows service, it will not be able to set and monitor the Preshutdown event, the node will have to go through the Non-Graceful Node Shutdown procedure mentioned above.In the case where the Windows graceful node shutdown feature is enabled, but the kubelet is not running as a Windows service, the kubelet will continue running instead of failing. However, it will log an error indicating that it needs to be run as a Windows service.Configuring graceful node shutdownNote that by default, both configuration options described below, shutdownGracePeriod and shutdownGracePeriodCriticalPods, are set to zero, thus not activating the graceful node shutdown functionality. To activate the feature, both options should be configured appropriately and set to non-zero values.Once the kubelet is notified of a node shutdown, it sets a NotReady condition on the Node, with the reason set to "node is shutting down". The kube-scheduler honors this condition and does not schedule any Pods onto the affected node; other third-party schedulers are expected to follow the same logic. This means that new Pods won't be scheduled onto that node and therefore none will start.The kubelet also rejects Pods during the PodAdmission phase if an ongoing node shutdown has been detected, so that even Pods with a toleration for node.kubernetes.io/not-ready:NoSchedule do not start there.When kubelet is setting that condition on its Node via the API, the kubelet also begins terminating any Pods that are running locally.During a graceful shutdown, kubelet terminates pods in two phases:Terminate regular pods running on the node.Terminate critical pods running on the node.The graceful node shutdown feature is configured with two KubeletConfiguration options:shutdownGracePeriod:Specifies the total duration that the node should delay the shutdown by. This is the total grace period for pod termination for both regular and critical pods.shutdownGracePeriodCriticalPods:Specifies the duration used to terminate critical pods during a node shutdown. This value should be less than shutdownGracePeriod.Note:There are cases when Node termination was cancelled by the system (or perhaps manually by an administrator). In either of those situations the Node will return to the Ready state. However, Pods which already started the process of termination will not be restored by kubelet and will need to be re-scheduled.For example, if shutdownGracePeriod=30s, and shutdownGracePeriodCriticalPods=10s, kubelet will delay the node shutdown by 30 seconds. During the shutdown, the first 20 (30-10) seconds would be reserved for gracefully terminating normal pods, and the last 10 seconds would be reserved for terminating critical pods.Note:When pods were evicted during the graceful node shutdown, they are marked as shutdown. Running kubectl get pods shows the status of the evicted pods as Terminated. And kubectl describe pod indicates that the pod was evicted because of node shutdown:Reason: Terminated Message: Pod was terminated in response to imminent node shutdown. Pod Priority based graceful node shutdownFEATURE STATE: Kubernetes v1.24 [beta] (enabled by default: true)To provide more flexibility during graceful node shutdown around the ordering of pods during shutdown, graceful node shutdown honors the PriorityClass for Pods, provided that you enabled this feature in your cluster. The feature allows cluster administrators to explicitly define the ordering of pods during graceful node shutdown based on priority classes.The Graceful Node Shutdown feature, as described above, shuts down pods in two phases, non-critical pods, followed by critical pods. If additional flexibility is needed to explicitly define the ordering of pods during shutdown in a more granular way, pod priority based graceful shutdown can be used.When graceful node shutdown honors pod priorities, this makes it possible to do graceful node shutdown in multiple phases, each phase shutting down a particular priority class of pods. The kubelet can be configured with the exact phases and shutdown time per phase.Assuming the following custom pod priority classes in a cluster,Pod priority class namePod priority class valuecustom-class-a100000custom-class-b10000custom-class-c1000regular/unset0Within the kubelet configuration the settings for shutdownGracePeriodByPodPriority could look like:Pod priority class valueShutdown period10000010 seconds10000180 seconds1000120 seconds060 secondsThe corresponding kubelet config YAML configuration would be:shutdownGracePeriodByPodPriority: - priority: 100000 shutdownGracePeriodSeconds: 10 - priority: 10000 shutdownGracePeriodSeconds: 180 - priority: 1000 shutdownGracePeriodSeconds: 120 - priority: 0 shutdownGracePeriodSeconds: 60 The above table implies that any pod with priority value >= 100000 will get just 10 seconds to shut down, any pod with value >= 10000 and < 100000 will get 180 seconds to shut down, any pod with value >= 1000 and < 10000 will get 120 seconds to shut down. Finally, all other pods will get 60 seconds to shut down.One doesn't have to specify values corresponding to all of the classes. For example, you could instead use these settings:Pod priority class valueShutdown period100000300 seconds1000120 seconds060 secondsIn the above case, the pods with custom-class-b will go into the same bucket as custom-class-c for shutdown.If there are no pods in a particular range, then the kubelet does not wait for pods in that priority range. Instead, the kubelet immediately skips to the next priority class value range.If this feature is enabled and no configuration is provided, then no ordering action will be taken.Using this feature requires enabling the GracefulNodeShutdownBasedOnPodPriority feature gate, and setting ShutdownGracePeriodByPodPriority in the kubelet config to the desired configuration containing the pod priority class values and their respective shutdown periods.Note:The ability to take Pod priority into account during graceful node shutdown was introduced as an Alpha feature in Kubernetes v1.23. In Kubernetes 1.34 the feature is Beta and is enabled by default.Metrics graceful_shutdown_start_time_seconds and graceful_shutdown_end_time_seconds are emitted under the kubelet subsystem to monitor node shutdowns.Non-graceful node shutdown handlingFEATURE STATE: Kubernetes v1.28 [stable] (enabled by default: true)A node shutdown action may not be detected by kubelet's Node Shutdown Manager, either because the command does not trigger the inhibitor locks mechanism used by kubelet or because of a user error, i.e., the ShutdownGracePeriod and ShutdownGracePeriodCriticalPods are not configured properly. Please refer to above section Graceful Node Shutdown for more details.When a node is shutdown but not detected by kubelet's Node Shutdown Manager, the pods that are part of a StatefulSet will be stuck in terminating status on the shutdown node and cannot move to a new running node. This is because kubelet on the shutdown node is not available to delete the pods so the StatefulSet cannot create a new pod with the same name. If there are volumes used by the pods, the VolumeAttachments will not be deleted from the original shutdown node so the volumes used by these pods cannot be attached to a new running node. As a result, the application running on the StatefulSet cannot function properly. If the original shutdown node comes up, the pods will be deleted by kubelet and new pods will be created on a different running node. If the original shutdown node does not come up, these pods will be stuck in terminating status on the shutdown node forever.To mitigate the above situation, a user can manually add the taint node.kubernetes.io/out-of-service with either NoExecute or NoSchedule effect to a Node marking it out-of-service. If a Node is marked out-of-service with this taint, the pods on the node will be forcefully deleted if there are no matching tolerations on it and volume detach operations for the pods terminating on the node will happen immediately. This allows the Pods on the out-of-service node to recover quickly on a different node.During a non-graceful shutdown, Pods are terminated in the two phases:Force delete the Pods that do not have matching out-of-service tolerations.Immediately perform detach volume operation for such pods.Note:Before adding the taint node.kubernetes.io/out-of-service, it should be verified that the node is already in shutdown or power off state (not in the middle of restarting).The user is required to manually remove the out-of-service taint after the pods are moved to a new node and the user has checked that the shutdown node has been recovered since the user was the one who originally added the taint.Forced storage detach on timeoutIn any situation where a pod deletion has not succeeded for 6 minutes, kubernetes will force detach volumes being unmounted if the node is unhealthy at that instant. Any workload still running on the node that uses a force-detached volume will cause a violation of the CSI specification, which states that ControllerUnpublishVolume "must be called after all NodeUnstageVolume and NodeUnpublishVolume on the volume are called and succeed". In such circumstances, volumes on the node in question might encounter data corruption.The forced storage detach behaviour is optional; users might opt to use the "Non-graceful node shutdown" feature instead.Force storage detach on timeout can be disabled by setting the disable-force-detach-on-timeout config field in kube-controller-manager. Disabling the force detach on timeout feature means that a volume that is hosted on a node that is unhealthy for more than 6 minutes will not have its associated VolumeAttachment deleted.After this setting has been applied, unhealthy pods still attached to volumes must be recovered via the Non-Graceful Node Shutdown procedure mentioned above.Note:Caution must be taken while using the Non-Graceful Node Shutdown procedure.Deviation from the steps documented above can result in data corruption.What's nextLearn more about the following:Blog: Non-Graceful Node Shutdown.Cluster Architecture: Nodes.2 - Swap memory managementKubernetes can be configured to use swap memory on a node, allowing the kernel to free up physical memory by swapping out pages to backing storage. This is useful for multiple use-cases. For example, nodes running workloads that can benefit from using swap, such as those that have large memory footprints but only access a portion of that memory at any given time. It also helps prevent Pods from being terminated during memory pressure spikes, shields nodes from system-level memory spikes that might compromise its stability, allows for more flexible memory management on the node, and much more.To learn about configuring swap in your cluster, read Configuring swap memory on Kubernetes nodes.Operating system supportLinux nodes support swap; you need to configure each node to enable it. By default, the kubelet will not start on a Linux node that has swap enabled.Windows nodes require swap space. By default, the kubelet does not start on a Windows node that has swap disabled.How does it work?There are a number of possible ways that one could envision swap use on a node. If kubelet is already running on a node, it would need to be restarted after swap is provisioned in order to identify it.When kubelet starts on a node in which swap is provisioned and available (with the failSwapOn: false configuration), kubelet will:Be able to start on this swap-enabled node.Direct the Container Runtime Interface (CRI) implementation, often referred to as the container runtime, to allocate zero swap memory to Kubernetes workloads by default.Swap configuration on a node is exposed to a cluster admin via the memorySwap in the KubeletConfiguration. As a cluster administrator, you can specify the node's behaviour in the presence of swap memory by setting memorySwap.swapBehavior.Swap behaviorsYou need to pick a swap behavior to use. Different nodes in your cluster can use different swap behaviors.The swap behaviors you can choose for Linux nodes are:NoSwap (default)Workloads running as Pods on this node do not and cannot use swap.LimitedSwapKubernetes workloads can utilize swap memory.Note:If you choose the NoSwap behavior, and you configure the kubelet to tolerate swap space (failSwapOn: false), then your workloads don't use any swap.However, processes outside of Kubernetes-managed containers, such as systemi services (and even the kubelet itself!) can utilize swap.You can read configuring swap memory on Kubernetes nodes to learn about enabling swap for your cluster.Container runtime integrationThe kubelet uses the container runtime API, and directs the container runtime to apply specific configuration (for example, in the cgroup v2 case, memory.swap.max) in a manner that will enable the desired swap configuration for a container. For runtimes that use control groups, or cgroups, the container runtime is then responsible for writing these settings to the container-level cgroup.Observability for swap useNode and container level metric statisticsKubelet now collects node and container level metric statistics, which can be accessed at the /metrics/resource (which is used mainly by monitoring tools like Prometheus) and /stats/summary (which is used mainly by Autoscalers) kubelet HTTP endpoints. This allows clients who can directly request the kubelet to monitor swap usage and remaining swap memory when using LimitedSwap. Additionally, a machine_swap_bytes metric has been added to cadvisor to show the total physical swap capacity of the machine. See this page for more info.For example, these /metrics/resource are supported:node_swap_usage_bytes: Current swap usage of the node in bytes.container_swap_usage_bytes: Current amount of the container swap usage in bytes.container_swap_limit_bytes: Current amount of the container swap limit in bytes.Using kubectl top --show-swapQuerying metrics is valuable, but somewhat cumbersome, as these metrics are designed to be used by software rather than humans. In order to consume this data in a more user-friendly way, the kubectl top command has been extended to support swap metrics, using the --show-swap flag.In order to receive information about swap usage on nodes, kubectl top nodes --show-swap can be used:kubectl top nodes --show-swap This will result in an output similar to:NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%) SWAP(bytes) SWAP(%) node1 1m 10% 2Mi 10% 1Mi 0% node2 5m 10% 6Mi 10% 2Mi 0% node3 3m 10% 4Mi 10% <unknown> <unknown> In order to receive information about swap usage by pods, kubectl top nodes --show-swap can be used:kubectl top pod -n kube-system --show-swap This will result in an output similar to:NAME CPU(cores) MEMORY(bytes) SWAP(bytes) coredns-58d5bc5cdb-5nbk4 2m 19Mi 0Mi coredns-58d5bc5cdb-jsh26 3m 37Mi 0Mi etcd-node01 51m 143Mi 5Mi kube-apiserver-node01 98m 824Mi 16Mi kube-controller-manager-node01 20m 135Mi 9Mi kube-proxy-ffgs2 1m 24Mi 0Mi kube-proxy-fhvwx 1m 39Mi 0Mi kube-scheduler-node01 13m 69Mi 0Mi metrics-server-8598789fdb-d2kcj 5m 26Mi 0Mi Nodes to report swap capacity as part of node statusA new node status field is now added, node.status.nodeInfo.swap.capacity, to report the swap capacity of a node.As an example, the following command can be used to retrieve the swap capacity of the nodes in a cluster:kubectl get nodes -o go-template='{{range .items}}{{.metadata.name}}: {{if .status.nodeInfo.swap.capacity}}{{.status.nodeInfo.swap.capacity}}{{else}}<unknown>{{end}}{{"\n"}}{{end}}' This will result in an output similar to:node1: 21474836480 node2: 42949664768 node3: <unknown> Note:The <unknown> value indicates that the .status.nodeInfo.swap.capacity field is not set for that Node. This probably means that the node does not have swap provisioned, or less likely, that the kubelet is not able to determine the swap capacity of the node.Swap discovery using Node Feature Discovery (NFD)Node Feature Discovery is a Kubernetes addon for detecting hardware features and configuration. It can be utilized to discover which nodes are provisioned with swap.As an example, to figure out which nodes are provisioned with swap, use the following command:kubectl get nodes -o jsonpath='{range .items[?(@.metadata.labels.feature\.node\.kubernetes\.io/memory-swap)]}{.metadata.name}{"\t"}{.metadata.labels.feature\.node\.kubernetes\.io/memory-swap}{"\n"}{end}' This will result in an output similar to:k8s-worker1: true k8s-worker2: true k8s-worker3: false In this example, swap is provisioned on nodes k8s-worker1 and k8s-worker2, but not on k8s-worker3.Risks and caveatsCaution:It is deeply encouraged to encrypt the swap space. See Memory-backed volumes memory-backed volumes for more info.Having swap available on a system reduces predictability. While swap can enhance performance by making more RAM available, swapping data back to memory is a heavy operation, sometimes slower by many orders of magnitude, which can cause unexpected performance regressions. Furthermore, swap changes a system's behaviour under memory pressure. Enabling swap increases the risk of noisy neighbors, where Pods that frequently use their RAM may cause other Pods to swap. In addition, since swap allows for greater memory usage for workloads in Kubernetes that cannot be predictably accounted for, and due to unexpected packing configurations, the scheduler currently does not account for swap memory usage. This heightens the risk of noisy neighbors.The performance of a node with swap memory enabled depends on the underlying physical storage. When swap memory is in use, performance will be significantly worse in an I/O operations per second (IOPS) constrained environment, such as a cloud VM with I/O throttling, when compared to faster storage mediums like solid-state drives or NVMe. As swap might cause IO pressure, it is recommended to give a higher IO latency priority to system critical daemons. See the relevant section in the recommended practices section below.Memory-backed volumesOn Linux nodes, memory-backed volumes (such as secret volume mounts, or emptyDir with medium: Memory) are implemented with a tmpfs filesystem. The contents of such volumes should remain in memory at all times, hence should not be swapped to disk. To ensure the contents of such volumes remain in memory, the noswap tmpfs option is being used.The Linux kernel officially supports the noswap option from version 6.3 (more info can be found in Linux Kernel Version Requirements). However, the different distributions often choose to backport this mount option to older Linux versions as well.In order to verify whether the node supports the noswap option, the kubelet will do the following:If the kernel's version is above 6.3 then the noswap option will be assumed to be supported.Otherwise, kubelet would try to mount a dummy tmpfs with the noswap option at startup. If kubelet fails with an error indicating of an unknown option, noswap will be assumed to not be supported, hence will not be used. A kubelet log entry will be emitted to warn the user about memory-backed volumes might swap to disk. If kubelet succeeds, the dummy tmpfs will be deleted and the noswap option will be used.If the noswap option is not supported, kubelet will emit a warning log entry, then continue its execution.See the section above with an example for setting unencrypted swap. However, handling encrypted swap is not within the scope of kubelet; rather, it is a general OS configuration concern and should be addressed at that level. It is the administrator's responsibility to provision encrypted swap to mitigate this risk.EvictionsConfiguring memory eviction thresholds for swap-enabled nodes can be tricky.With swap being disabled, it is reasonable to configure kubelet's eviction thresholds to be a bit lower than the node's memory capacity. The rationale is that we want Kubernetes to start evicting Pods before the node runs out of memory and invokes the Out Of Memory (OOM) killer, since the OOM killer is not Kubernetes-aware, therefore does not consider things like QoS, pod priority, or other Kubernetes-specific factors.With swap enabled, the situation is more complex. In Linux, the vm.min_free_kbytes parameter defines the memory threshold for the kernel to start aggressively reclaiming memory, which includes swapping out pages. If the kubelet's eviction thresholds are set in a way that eviction would take place before the kernel starts reclaiming memory, it could lead to workloads never being able to swap out during node memory pressure. However, setting the eviction thresholds too high could result in the node running out of memory and invoking the OOM killer, which is not ideal either.To address this, it is recommended to set the kubelet's eviction thresholds to be slightly lower than the vm.min_free_kbytes value. This way, the node can start swapping before kubelet would start evicting Pods, allowing workloads to swap out unused data and preventing evictions from happening. On the other hand, since it is just slightly lower, kubelet is likely to start evicting Pods before the node runs out of memory, thus avoiding the OOM killer.The value of vm.min_free_kbytes can be determined by running the following command on the node:cat /proc/sys/vm/min_free_kbytes Unutilized swap spaceUnder the LimitedSwap behavior, the amount of swap available to a Pod is determined automatically, based on the proportion of the memory requested relative to the node's total memory (For more details, see the section below).This design means that usually there would be some portion of swap that will remain restricted for Kubernetes workloads. For example, since Kubernetes 1.34 does not permit swap use for Pods in the Guaranteed QoS class, the amount of swap that's proportional to the memory request for Guaranteed pods would remain unused by Kubernetes workloads.This behavior carries some risk in a situation where many pods are not eligible for swapping. On the other hand, it effectively keeps some system-reserved amount of swap memory that can be used by processes outside of Kubernetes' scope, such as system daemons and even kubelet itself.Good practice for using swap in a Kubernetes clusterDisable swap for system-critical daemonsDuring the testing phase and based on user