Question: How does EKS Anywhere cluster bootstrapping work?

I want to understand the bootstrapping process for Amazon Elastic Kubernetes Service (Amazon EKS) Anywhere.

ResolutionThe bootstrap clusterWhen you create an initial standalone cluster or management cluster, Amazon EKS Anywhere also creates a bootstrap cluster. This is a temporary, single node Kubernetes in Docker (KinD) cluster that's created on a separate Administrative machine to facilitate your main cluster's creation.EKS Anywhere creates a bootstrap cluster on the Administrative machine that hosts CAPI and CAPX operators. To create the bootstrap cluster, EKS Anywhere must complete the following steps:Pull the KinD node imagePrepare the nodeWrite the configurationStart the control planeInstall the CNIInstall the StorageClass on the KinD-based single node clusterCluster creationWhen the bootstrap cluster is running and properly configured on the Administrative machine, the creation of the target cluster begins. EKS Anywhere uses kubectl to apply a target cluster configuration in the following process:1.    After etcd, the control plane, and the worker nodes are ready, the target cluster receives its networking configuration.2.    The target cluster receives its default storage class installation.3.    CAPI providers are configured on the target cluster. This allows the target cluster to control and run the components that it needs to manage itself.4.    After CAPI runs on the target cluster, CAPI objects are moved from the bootstrap cluster to the target cluster's CAPI service. This happens internally with the clusterctl command.5.    The target cluster receives Kubernetes CRDs and other add-ons that are specific to EKS Anywhere.6.    The cluster configuration is saved.The bootstrapping process creates a YAML file that's located in CLUSTER_NAME/generated/CLUSTER_NAME-eks-a-cluster.yaml.When boostrapping succeeds, this YAML file moves to CLUSTER_NAME/CLUSTER_NAME-eks-a-cluster.yamlSimilarly, Kubeconfig moves from CLUSTER_NAME/generated/CLUSTER_NAME.kind.kubeconfig to CLUSTER_NAME/CLUSTER_NAME-eks-a-cluster.kubeconfig.When etcd, the control plane, and the worker nodes are ready, the workload cluster receives its networking configuration. When the cluster is active and the CAPI service is running on the new cluster, the bootstrap cluster is no longer needed. Then, the service deletes the bootstrap cluster.Package workflowsDuring the bootstrapping process, EKS Anywhere uses the following logic in its workflows for the target cluster creation, cluster upgrade, and cluster deletion.Cluster creationFor the full package workflow during cluster creation, see create.go on GitHub. During this workflow, EKS Anywhere uses the following logic:Setups and validationsNote: If this step fails, then either preflights failed or the registry isn't set up properly.Create a new bootstrap clusterCreate a new KinD clusterProvider specific pre-capi-install-setup on the bootstrap clusterInstall cluster-api providers on bootstrap clusterProvider specific post-setupCreate a new workload clusterWait for external etcd to be readyWait for control plane to become availableWait for workload kubeconfig generationInstall networking the on workload clusterInstall machine health checks on the bootstrap clusterWait for control plane and worker machines to be readyInstall resources on managementInstall the eks-a components taskInstall the Git ops managerMove cluster managementWrite ClusterConfigDelete the bootstrap clusterInstall curated packagesCluster upgradeFor the full package workflow during a cluster upgrade, see upgrade.go on GitHub. During this workflow, EKS Anywhere uses the following logic:Setups and validationsUpdate secretsVerify etcd CAPI components existUpgrade core componentsVerify the needed upgradePause eks-a reconciliationCreate the bootstrap clusterInstall CAPIMove management to the bootstrap clusterMove management to the workload clusterUpgrade the workload clusterDelete the bootstrap clusterUpdate the workload cluster and Git resourcesResume eks-a reconciliationWrite ClusterConfigCluster deletionFor the full package workflow during a cluster's deletion, see delete.go on GitHub. During this workflow, EKS Anywhere uses the following logic:Setups and validatationsCreate a management clusterInstall CAPIMove cluster managementDelete the workload clusterClean up the Git repositoryDelete package resourcesDelete the management clusterErrors during cluster creationIf you encounter issues or errors, then look for logs in the Administrative machine and the capc-controller-manager. View the capc-controller-manager logs with kubectl in the capc-system namespace. For further troubleshooting, check the status of the CAPI objects for your cluster, located in the eksa-system namespace.You might also find related information on errors in the logs of other CAPI managers, such as capi-kubeadm-bootstrap-controller, capi-kubeadm-control-plane-controller and capi-controller-manager. These managers work together, and you can locate each in their own namespace with the kubectl get pods -A command. For more information, see the troubleshooting guide for EKS Anywhere.For a script to fix linting errors during the bootstrapping process, see bootstrapper.go on GitHub.

Related informationKinD quick start (on the KinD website)Cluster creation workflow