Question: How do I configure Amazon EKS worker nodes to clean up the image cache at a specified percent of disk usage?

I want to use Amazon Elastic Kubernetes Service (Amazon EKS) worker nodes to clean up the image cache at a specified percent of disk usage.

Short descriptionTo clean up the image cache with Amazon EKS worker nodes, use the following kubelet garbage collection (from the Kubernetes website) arguments:The --image-gc-high-threshold argument defines the percent of disk usage that initiates image garbage collection. The default is 85%.The --image-gc-low-threshold argument defines the percent of disk usage that image garbage collection tries to free. The default is 80%.Note: In the following resolution, the kubelet garbage collection arguments clean up the image cache in the worker node when the disk usage reaches 70%. The --image-gc-high-threshold is set to 70%. The --image-gc-low-threshold is set to 50%.To add kubelet garbage collection arguments to a new worker node, complete the steps in the following sections:Add the kubelet garbage collection arguments to the default kubelet-config file in the Amazon Machine Image (AMI) using UserData for a new worker nodeVerify that the new kubelet garbage collection arguments are in the node configz endpointTo add kubelet garbage collection arguments to an existing worker node, complete the steps in the following sections:Add the kubelet garbage collection arguments to an existing worker nodeVerify that the new kubelet garbage collection arguments are in the node configz endpoint

ResolutionNote: The following resolution applies to Amazon EKS-optimized Linux AMIs.Add the kubelet garbage collection arguments to the default kubelet-config file in the AMI using UserData for a new worker node1.    Launch an Amazon Elastic Compute Cloud (Amazon EC2) instance.Use the sed commands to add --image-gc-low-threshold and --image-gc-high-threshold in the kubelet-config.json file in the Amazon EKS-optimized Linux AMI. The sed commands are in the UserData section of Step 3: Configure instance details.For example:#!/bin/bashset -o xtrace# Inject imageGCHighThresholdPercent value unless it has already been set.if ! grep -q imageGCHighThresholdPercent /etc/kubernetes/kubelet/kubelet-config.json; then     sed -i '/"apiVersion*/a \ \ "imageGCHighThresholdPercent": 70,' /etc/kubernetes/kubelet/kubelet-config.jsonfi# Inject imageGCLowThresholdPercent value unless it has already been set.if ! grep -q imageGCLowThresholdPercent /etc/kubernetes/kubelet/kubelet-config.json; then     sed -i '/"imageGCHigh*/a \ \ "imageGCLowThresholdPercent": 50,' /etc/kubernetes/kubelet/kubelet-config.jsonfi/etc/eks/bootstrap.sh your-cluster-nameNote: Replace your-cluster-name with your Amazon EKS cluster name. If you're using a different kubelet config file, then update the file name in the sed commands.2.    Add the worker node to your Amazon EKS cluster.3.    To verify the changes, follow the steps in the Verify that the new kubelet garbage collection arguments are in the node configz endpoint section.Add the kubelet garbage collection arguments to an existing worker nodeImportant: The following steps require that you connect to an existing worker node with SSH and have sudo access. You must complete these steps on all the existing worker nodes in your Amazon EKS cluster.1.    Connect to an existing worker node using SSH.2.    Open the /etc/kubernetes/kubelet/kubelet-config.json file in your worker nodes.If you launched the worker node using EKSCTL, then open /etc/eksctl/kubelet.yaml. See the following example:sudo vi /etc/kubernetes/kubelet/kubelet-config.json#WORKER NODES LAUNCHED USING EKSCTLsudo vi /etc/eksctl/kubelet.yaml3.    Add the kubelet garbage collection arguments to the kubelet-config.json file or kubelet.yaml  file, depending on how you launched your worker nodes. Then, save the file. See the following example:{  "kind": "KubeletConfiguration",  "apiVersion": "kubelet.config.k8s.io/v1beta1",  .  .  .  "imageGCHighThresholdPercent": 70,         ==> Add the argument under the same alignment as the "kind"  "imageGCLowThresholdPercent": 50,  "maxPods": ...}#WORKER NODES LAUNCHED USING EKSCTLkind: KubeletConfigurationkubeReserved:  cpu: 70m  ephemeral-storage: 1Gi  memory: 1843MiserverTLSBootstrap: trueimageGCHighThresholdPercent: 70        ==> Add the arguments under the alignment "Kind" in the yaml fileimageGCLowThresholdPercent: 504.    To restart the kubelet service in the worker node, run the following command:sudo service kubelet restart5.    To verify the changes, follow the steps in the Verify that the new kubelet garbage collection arguments are in the node configz endpoint section.Verify that the new kubelet garbage collection arguments are in the node configz endpoint1.    To get the name of your worker nodes, run the following command:kubectl get nodes2.    To open a connection to the API server, run the following command:kubectl proxy3.    To check the node configz, open a new terminal, and then run the following command:curl -sSL "http://localhost:8001/api/v1/nodes/node_name/proxy/configz" | python3 -m json.toolNote: Replace node_name with your node name from the list of nodes retrieved in step 1. If curl and python aren't available, then open the URL in a web browser.The preceding command returns output from the kubeletconfig. This output includes your settings from the bootstrap.sh file. See the following example:{"kubeletconfig": {  .  .  "imageGCHighThresholdPercent": 70,          <=== The new value is set to 70 as given in UserData  "imageGCLowThresholdPercent": 50,           <=== The new value is set to 50 as given in UserData  .  .}}