Question: How do I return an EKS Anywhere cluster to a working state when the cluster upgrade fails?

I want to use the eksctl command to upgrade an Amazon Elastic Kubernetes Service (Amazon EKS) Anywhere management cluster. However, the upgrade process fails or is interrupted before completion.

ResolutionWhen you upgrade an Amazon EKS Anywhere management cluster, the process includes two phases: the verification phase and upgrade phase. The recovery steps for a failed upgrade depend on which phase of the upgrade was interrupted.Verification phaseWhen you upgrade an EKS Anywhere cluster, eksctl runs a set of preflight checks to make sure that your cluster is ready. This occurs before the upgrade, and eksctl modifies your cluster to match the updated specification.When eksctl performs these checks, you see a message that's similar to the following example:Performing setup and validations   Connected to server   Authenticated to vSphere   Datacenter validated   Network validatedCreating template. This might take a while.   Datastore validated   Folder validated   Resource pool validated   Datastore validated   Folder validated   Resource pool validated   Datastore validated   Folder validated   Resource pool validated   Machine config tags validated   Control plane and Workload templates validated   Vsphere provider validation   Validate certificate for registry mirror   Control plane ready   Worker nodes ready   Nodes ready   Cluster CRDs ready   Cluster object present on workload cluster   Upgrade cluster kubernetes version increment   Validate authentication for git provider   Validate immutable fields   Upgrade preflight validations passNext, eksctl continues to verify CAPI controllers that run in your management cluster. If any of these controllers need an upgrade, then eksctl also upgrades them. During this process, eksctl also creates a KinD bootstrap cluster to upgrade your management cluster. You see a message that reflects this process:Ensuring etcd CAPI providers exist on management cluster before Pausing EKS-A cluster controller reconcilePausing GitOps cluster resources reconcileUpgrading core componentsCreating bootstrap clusterProvider specific pre-capi-install-setup on bootstrap clusterProvider specific post-setupInstalling cluster-api providers on bootstrap clusterIf any of these checks or actions fail, then the upgrade stops and your management cluster remains in the same original version.For more details about the specific check that failed, check the eksctl logs.Issues during the verification phaseTo recover from a failure at this phase, complete the following steps:1.    Troubleshoot and fix the problem that caused the verification to fail.2.    Run the eksctl anywhere cluster upgrade command again. It's a best practice to use the -v9 flag.Upgrade phaseIn the upgrade phase, eksctl performs the following main actions:Moves your management cluster CAPI objects (such as machines, KubeadmControlPlane, and EtcdadmCluster)to the bootstrap clusterUpgrades the etcd and control plane componentsUpgrades the worker node componentsDuring this phase, you see a message that's similar to the following example:Moving cluster management from bootstrap to workload clusterApplying new EKS-A cluster resourceResuming EKS-A controller reconcileUpdating Git Repo with new EKS-A cluster specGitOps field not specified, update git repo skippedForcing reconcile Git repo with latest commitGitOps not configured, force reconcile flux git repo skippedResuming GitOps cluster resources kustomizationWriting cluster config file    Cluster upgraded!eksctl uses a rolling process to perform the upgrade in place, similar to Kubernetes deployments. It also creates a new virtual machine (VM) with this upgrade, and then it removes the old VM. This process applies to each component, one at a time, until all control plane components are upgraded.If a VM fails to run, then the upgrade fails and stops after a set timeout interval. The rolling process keeps the old VM running to make sure that your cluster remains in the Ready state.Issues during the upgrade phaseTo recover from a failure during this phase, complete the following steps:1.    Troubleshoot and fix the problem that caused the upgrade to fail. Check the eksctl logs for details about the failure.2.    To facilitate the recovery process, set up an environment variable:CLUSTER_NAME: The name of your clusterCLITOOLS_CONT: The name of the container that runs image cli-tools left in your environment after upgrade interruptionKINDKUBE: The Kubeconfig file that you use to access the KinD bootstrap clusterMGMTKUBE: The Kubeconfig file that you use to access your management clusterEKSA_VSPHERE_USERNAME and EKSA_VSPHERE_PASSWORD: Credentials to access vCenterSee the following example of these variables:CLUSTER_NAME=cluster3CLITOOLS_CONT=eksa_1681572481616591501KINDKUBE=$CLUSTER_NAME/generated/${CLUSTER_NAME}.kind.kubeconfigMGMTKUBE=$CLUSTER_NAME/$CLUSTER_NAME-eks-a-cluster.kubeconfigEKSA_VSPHERE_USERNAME=xxxxxEKSA_VSPHERE_PASSWORD=yyyyy3.    Make sure that your management cluster CAPI components, such as machines and clusters, are in the Ready state. Also, make sure that kubeApi-server in your management cluster is responsive. To do this, run the following command:kubectl --kubeconfig $KINDKUBE -n eksa-system get machinesdocker exec -i $CLITOOLS_CONT clusterctl describe cluster cluster3  --kubeconfig $KINDKUBE -n eksa-systemkubectl --kubeconfig $MGMTKUBE -n kube-system get nodeYou receive an output that's similar to the following example:NAME                            CLUSTER    NODENAME                        PROVIDERID                                       PHASE     AGE     VERSIONcluster3-2snw8                  cluster3   cluster3-2snw8                  vsphere://4230efe1-e1f5-c8e5-9bff-12eca320f5db   Running   3m13s   v1.23.17-eks-1-23-19cluster3-etcd-chkc5             cluster3                                   vsphere://4230826c-b25d-937a-4728-3e607e6af579   Running   4m14scluster3-md-0-854976576-tw6hr   cluster3   cluster3-md-0-854976576-tw6hr   vsphere://4230f2e5-0a4b-374c-f06b-41ac1f80e41f   Running   4m30s   v1.22.17-eks-1-22-24$ docker exec -i $CLITOOLS_CONT clusterctl describe cluster cluster3  --kubeconfig $KINDKUBE -n eksa-systemNAME                                               READY  SEVERITY  REASON  SINCE  MESSAGECluster/cluster3                                   True                     49s├─ClusterInfrastructure - VSphereCluster/cluster3  True                     4m53s├─ControlPlane - KubeadmControlPlane/cluster3      True                     49s│ └─Machine/cluster3-2snw8                         True                     2m51s└─Workers  ├─MachineDeployment/cluster3-md-0                True                     4m53s  │ └─Machine/cluster3-md-0-854976576-tw6hr        True                     4m53s  └─Other    └─Machine/cluster3-etcd-chkc5                  True                     3m55s    $ kubectl --kubeconfig $MGMTKUBE -n kube-system get nodeNAME                             STATUS   ROLES                  AGE   VERSIONcluster3-md-0-854976576-tw6hr    Ready    [none]                 18m   v1.22.17-eks-a51510bcluster3-2snw8                   Ready    control-plane,master   19m   v1.23.17-eks-a51510b4.    Back up your management cluster CAPI components:mkdir ${CLUSTER_NAME}-backupdocker exec -i $CLITOOLS_CONT clusterctl move --to-directory ${CLUSTER_NAME}-backup  --kubeconfig $KINDKUBE -n eksa-system5.    Move your management cluster CAPI components back to your management cluster:$ docker exec -i $CLITOOLS_CONT clusterctl move --to-kubeconfig $MGMTKUBE  --kubeconfig $KINDKUBE -n eksa-systemPerforming move...Discovering Cluster API objectsMoving Cluster API objects Clusters=1Moving Cluster API objects ClusterClasses=0Creating objects in the target clusterDeleting objects from the source clusterYou receive an output that's similar to the following example:$ docker exec -i $CLITOOLS_CONT clusterctl move --to-kubeconfig $MGMTKUBE  --kubeconfig $KINDKUBE -n eksa-systemPerforming move...Discovering Cluster API objectsMoving Cluster API objects Clusters=1Moving Cluster API objects ClusterClasses=0Creating objects in the target clusterDeleting objects from the source cluster6.    Make sure that management cluster CAPI components, such as machines and clusters, are no longer in the KinD bootstrap cluster. Verify that they show up in the management cluster. To do this, run the following commands:kubectl --kubeconfig $KINDKUBE -n eksa-system get cluster -n eksa-systemkubectl --kubeconfig $MGMTKUBE get machines -n eksa-systemYou receive an output that's similar to the following example:$ kubectl --kubeconfig $KINDKUBE -n eksa-system get cluster -n eksa-systemNo resources found in eksa-system namespace.$ kubectl --kubeconfig $MGMTKUBE get machines -n eksa-systemNAME                           CLUSTER    NODENAME                       PROVIDERID                                       PHASE     AGE     VERSIONcluster2-4n7qd                 cluster2   cluster2-4n7qd                 vsphere://4230fb07-2823-3474-c41f-b7223dec3089   Running   2m27s   v1.23.17-eks-1-23-19cluster2-etcd-h4tpl            cluster2                                  vsphere://42303b36-1991-67a9-e942-dd9959760649   Running   2m27scluster2-md-0-fd6c558b-6cfvq   cluster2   cluster2-md-0-fd6c558b-6cfvq   vsphere://423019a3-ad3f-1743-e7a8-ec8772d3edc2   Running   2m26s   v1.22.17-eks-1-22-247.    Run the upgrade again. Use the flags --force-cleanup -v9 flag:eksctl anywhere upgrade cluster -f cluster3/cluster3-eks-a-cluster.yaml --force-cleanup -v9Related informationUpgrade vSphere, CloudStack, Nutanix, or Snow clusterEKS-A troubleshootingThe Cluster API Book (on the Kubernetes website)