Package: eks-rolling-update
EKS Rolling UpdateEKS Rolling Update is a utility for updating the launch configuration or template of worker nodes in an EKS cluster.IntroRequirementsInstallationUsageConfigurationContributingLicenseIntroEKS Rolling Update is a utility for updating the launch configuration or template of worker nodes in an EKS cluster. It
updates worker nodes in a rolling fashion and performs health checks of your EKS cluster to ensure no disruption to service.
To achieve this, it performs the following actions:Pauses Kubernetes Autoscaler (Optional)Finds a list of worker nodes that do not have a launch config or template that matches their ASGScales up the desired capacityEnsures the ASGs are healthy and that the new nodes have joined the EKS clusterCordons the outdated worker nodesSuspends AWS Autoscaling actions while update is in progressDrains outdated EKS outdated worker nodes one by oneTerminates EC2 instances of the worker nodes one by oneDetaches EC2 instances from the ASG one by oneScales down the ASG to original count (in case of failure)Resumes AWS Autoscaling actionsResumes Kubernetes Autoscaler (Optional)RequirementskubectlinstalledKUBECONFIGenvironment variable set, or config available in${HOME}/.kube/configper defaultAWS credentialsconfiguredIAM RequirementsThe following IAM permissions are required:autoscaling:DescribeAutoScalingGroups
autoscaling:TerminateInstanceInAutoScalingGroup
autoscaling:SuspendProcesses
autoscaling:ResumeProcesses
autoscaling:UpdateAutoScalingGroup
autoscaling:CreateOrUpdateTags
autoscaling:DeleteTags
ec2:DescribeLaunchTemplates
ec2:DescribeInstanceInstallationFrom PyPipip3 install eks-rolling-updateFrom sourcevirtualenv -p python3 venv
source venv/bin/activate
pip3 install -r requirements.txtUsageusage: eks_rolling_update.py [-h] --cluster_name CLUSTER_NAME [--plan]

Rolling update on cluster

optional arguments:
  -h, --help            show this help message and exit
  --cluster_name CLUSTER_NAME, -c CLUSTER_NAME
                        the cluster name to perform rolling update on
  --plan, -p            perform a dry run to see which instances are out of
                        dateExample:eks_rolling_update.py -c my-eks-clusterConfigurationCore ConfigurationEnvironment VariableDescriptionDefaultRUN_MODEOverall strategy for handling multiple ASGs & identifying nodes to roll. SeeRun Modessection below1DRY_RUNIf True, only a query will be run to determine which worker nodes are outdated without running an update operationFalseCLUSTER_HEALTH_WAITNumber of seconds to wait after ASG has been scaled up before checking health of nodes with the cluster90CLUSTER_HEALTH_RETRYNumber of attempts to validate the health of the cluster after ASG has been scaled1GLOBAL_MAX_RETRYNumber of attempts of a node health or instance termination check12GLOBAL_HEALTH_WAITNumber of seconds to wait before retrying a health node health or instance termination check20BETWEEN_NODES_WAITNumber of seconds to wait after removing a node before continuing on0ASG & Node-Related ControlsEnvironment VariableDescriptionDefaultASG_DESIRED_STATE_TAGTemporary tag which will be saved to the ASG to store the state of the EKS cluster prior to updateeks-rolling-update:desired_capacityASG_ORIG_CAPACITY_TAGTemporary tag which will be saved to the ASG to store the state of the EKS cluster prior to updateeks-rolling-update:original_capacityASG_ORIG_MAX_CAPACITY_TAGTemporary tag which will be saved to the ASG to store the state of the EKS cluster prior to updateeks-rolling-update:original_max_capacityASG_NAMESList of space-delimited ASG names. Out of ASGs attached to the cluster, only these will be processed for rolling update. If this is left empty all ASGs of the cluster will be processed.""BATCH_SIZE# of instances to scale the ASG by at a time. When set to 0, batching is disabled. SeeBatchingsection0MAX_ALLOWABLE_NODE_AGEThe max age each node allowed to be. This works withRUN_MODE4 as node rolling is updating based on age of node6EXCLUDE_NODE_LABEL_KEYSList of space-delimited keys for node labels. Nodes with a label using one of these keys will be excluded from the node count when scaling the cluster.spotinst.io/node-lifecycleASG_USE_TERMINATION_POLICYPrefer ASG termination policy (instance terminate/detach handled by ASG according to configured termination policy)FalseINSTANCE_WAIT_FOR_STOPPINGOnly wait for terminated instances to be instoppingorshutting-downstate, instead of fullyterminatedorstoppedFalseK8S Node & Pod ControlsEnvironment VariableDescriptionDefaultK8S_AUTOSCALER_ENABLEDIf True Kubernetes Autoscaler will be paused before running updateFalseK8S_AUTOSCALER_NAMESPACENamespace where Kubernetes Autoscaler is deployeddefaultK8S_AUTOSCALER_DEPLOYMENTDeployment name of Kubernetes Autoscalercluster-autoscalerK8S_AUTOSCALER_REPLICASNumber of replicas to scale back up to after Kubernentes Autoscaler paused2K8S_CONTEXTContext from the Kubernetes config to use. If this is left undefined thecurrent-contextis usedNoneK8S_PROXY_BYPASSSet totrueto ignoreHTTPS_PROXYandHTTP_PROXYand disable use of any configured proxy when talking to the K8S APIFalseTAINT_NODESReplace the defaultcordon-before-drain strategy withNoScheduletainting, as a workaround for K8S <1.19prematurely removing cordoned nodesfromService-managedLoadBalancersFalseEXTRA_DRAIN_ARGSAdditional space-delimited args to supply to thekubectl drainfunction, e.g--force=true. Seekubectl drain -h""ENFORCED_DRAININGIf draining fails for a node due to corruptedPodDisruptionBudgets or failing pods, retry draining with--disable-eviction=trueand--force=truefor this node to prevent aborting the script. This is useful to get the rolling update done in development and testing environments andshould not be used in productive environmentssince this will bypass checkingPodDisruptionBudgetsFalseRun ModesThere are a number of different values which can be set for theRUN_MODEenvironment variable.1is the default.Mode NumberDescription1Scale up and cordon/taint the outdated nodes of each ASG one-by-one, just before we drain them.2Scale up and cordon/taint the outdated nodes of all ASGs all at once at the beginning of the run.3Cordon/taint the outdated nodes of all ASGs at the beginning of the run but scale each ASG one-by-one.4Roll EKS nodes based on age instead of launch config (works withMAX_ALLOWABLE_NODE_AGEwith default 6 days value).Each of them have different advantages and disadvantages.Scaling up all ASGs at once may cause AWS EC2 instance limits to be exceededOnly cordoning the nodes on a per-ASG basis will mean that pods are likely to be moved more than onceCordoning the nodes for all ASGs at once could cause issues if new pods needs to start during the processBatchingEKS Rolling Update can batch scale-out the ASG to progressively reach the desired instance count before it begins
draining the nodes.This is intended for use in cases where a large ASG scale-out may result in instances failing to register with
EKS. Such a scenario is more likely to occur with larger ASGs where (for example) a 100 instance ASG may be asked
to scale to 200 (temporarily). Users may find that some instances never register, and this causes EKS Rolling
Update to hang indefinitely waiting for the registered EKS node count to match the instance count.If this happens, you may want to consider batching.For example, if the ASG will be scaled from 100 instances to 200 instances, specifying a batch size of 10 will
result in the ASG first scaling to 110, then 120, 130, etc instances until 200 is reached. Once the desired
count is reached, the tool will proceed with the normal draining/scale-in operations.ExamplesPlan$ python eks_rolling_update.py --cluster_name YOUR_EKS_CLUSTER_NAME --planApply Changes$ python eks_rolling_update.py --cluster_name YOUR_EKS_CLUSTER_NAMECluster AutoscalerIf usingcluster-autoscaler, you must leteks-rolling-updateknow that cluster-autoscaler is running in your cluster by exporting the following environment variables:$ export  K8S_AUTOSCALER_ENABLED=true \
          K8S_AUTOSCALER_NAMESPACE="${CA_NAMESPACE}" \
          K8S_AUTOSCALER_DEPLOYMENT="${CA_DEPLOYMENT_NAME}"Disable operations oncluster-autoscaler$ unset K8S_AUTOSCALER_ENABLEDConfigure tool via.envfileRather than using environment variables, you can use a.envfile within your working directory to load
updater settings. e.g:$ cat .env
DRY_RUN=1DockerAlthough no public Docker image is currently published for this project, feel free to use the includedDockerfileto build your own image.makedocker-distversion=1.0.DEVAfter building the image, run using the commanddockerrun-ti--rm\-eAWS_DEFAULT_REGION\-v"${HOME}/.aws:/root/.aws"\-v"${HOME}/.kube/config:/root/.kube/config"\eks-rolling-update:latest\-cmy-clusterPass in any additional environment variables and options as described elsewhere in this file.ContributingPlease readCONTRIBUTING.mdfor details on our code of conduct, and the process for submitting pull requests to us.LicenseThis project is licensed under the Apache 2.0 License - see theLICENSEfile for details