Package: cdk-k3s-cluster
What is cdk-k3s-cluster?cdk-k3s-clusteris a new JSII construct library for AWS CDK that deploys a scalable KubernetesK3scluster onGraviton2 Arm-based(mg6 by default)Spotinstances with oneclickcommand on AWS.What problem does cdk-k3s-cluster solve?cdk-k3s-clusteris just another way to deploy Kubernetes (K3sspecifically) on AWS.K3sis a minimalist Kubernetes distribution from Rancher often related to Edge and IoT use cases. There is a gazillion of articles on the Internet about how to setupK3son a Raspberry cluster. The greatAlex Ellishas ablog poston it, for example. Since I don't have a Raspberry cluster and I was not planning to buy one I thought I'd use the power of the cloud to mimic it. The result of this experiment is that you could deploy an ephemeral, cheap Kubernetes Arm-based cluster on AWS in a matter of minutes. Unleash your imagination re how you can use it (e.g. an ephemeral cluster part of your deployment pipeline in the cloud?). Note the class today only supports Arm-based instances for deomstration purposes but can easily be adapted to support x86-based instances.How do you deploy and consume cdk-k3s-cluster?The cdk-k3s-cluster Cluster APITheClusterAPI available in thecdk-k3s-clusterlibrary, not to be confused with the Kubernetes Cluster API project, allows you to build the k3s cluster on AWS with AWS CDK. This library is currently available in bothNPMandPyPi.Creating a default cluster could be as simple as:import*ask3sfrom'cdk-k3s-cluster'newk3s.Cluster(stack,'Cluster')See below for a complete example.Deploymentcdk-k3s-clusterfirst deploys a Graviton2 EC2 instance (m6g.largeby default) where it starts theK3scontrol plane. Thekubeconfigfile generated and theK3stoken are copied to an S3 bucket. It then creates an ASG that spins up the number of Graviton2 worker nodes that you specify (3 by default - the ceiling limit is based on your account limits). These instances launchK3sworker nodes that join the cluster by downloading the token from S3. Ultimately the CDK outputs the link to thekubeconfigfile on your private S3 bucket for you to copy it and use it withkubectl. This construct can deploy the worker node instances using either theon-demandor theSpotlife cycles to reduce costs further (Spotis used by default). This construct cretes by default a new VPC but it can be configured to deploy either on thedefaultVPC in your account or on a specific existing VPC id.This is a high level view of the architecture and the deployment flow described above:This is a screenshot of the cluster creation user experience using CDK:ConsumptionOnce thecdk-k3s-clusterhas been deployed, this a high level view of the consumption flow:This is an example of the cluster consumption user experience using theaws cli(used to copy thekubeconfigfile from S3) andkubectl:$awss3cps3://k3sCluster-clusterk3sbucketxxxxxxxxxxxxx/kubeconfig.yaml.
download:s3://k3sCluster-clusterk3sbucketxxxxxxxxxxxxx/kubeconfig.yamlto./kubeconfig.yaml

$kubectlgetnodes--kubeconfig=./kubeconfig.yaml
NAMESTATUSROLESAGEVERSION
ip-172-31-43-198.us-west-2.compute.internalReadymaster2m49sv1.16.9+k3s1
ip-172-31-10-252.us-west-2.compute.internalReady<none>15sv1.16.13+k3s1
ip-172-31-18-126.us-west-2.compute.internalReady<none>8sv1.16.13+k3s1
ip-172-31-60-174.us-west-2.compute.internalReady<none>1sv1.16.13+k3s1How does this relate to EKSIt doesn't. As noted above,cdk-k3s-clusteris just yet another experimental and peculiar way to run Kubernetes on AWS.What are the running costs for a cluster built with cdk-k3s-cluster?It obviously depends how many worker nodes you deploy. If we stick to all the defaults (Spotlifecycle and them6g.mediuminstance type with 1 Graviton2 CPU and 4GB of memory), as of August 2020 the unit cost inOregonis $0.0177 which translates to a unit cost of $0.1416 per 8 hours (a work day). A 3 worker nodes cluster would be $0.5664 (including the control plane) per 8 hours (or $1.6992 per 24 hours full day).A100 worker nodes cluster(or 101 vCPUs and 404GB of memory) would be$0.0177 x 101 = $1.7877 per houror $0.0177 x 101 x 24 =$42.9048 per day.This does not include, for example, the costs of the S3 bucket (probably marginal) or network traffic.Getting startedcdk-k3s-clusteris available in bothNPMandPyPimodules ready to be imported into your CDK program.I am a Typescript type of personThis is an example of how to consume theNPMmodule with a CDK application written in Typescript. If you need to setup your Typescript environmentthis is a good guide.$mkdirmyk3scluster-typescript
$cdmyk3scluster-typescript# initialize the AWS CDK project$cdkinit-ltypescript# install the cdk-k3s-cluster npm module$yarnaddcdk-k3s-clusterUpdate your./bin/myk3scluster-typescript.tsfile with the following content.import*ascdkfrom'@aws-cdk/core';import*asec2from'@aws-cdk/aws-ec2';import*ask3sfrom'cdk-k3s-cluster';constapp=newcdk.App();constenv={region:app.node.tryGetContext('region')||process.env.CDK_INTEG_REGION||process.env.CDK_DEFAULT_REGION,account:app.node.tryGetContext('account')||process.env.CDK_INTEG_ACCOUNT||process.env.CDK_DEFAULT_ACCOUNT};conststack=newcdk.Stack(app,'k8sCluster',{env})newk3s.Cluster(stack,'Cluster',{vpc:k3s.VpcProvider.getOrCreate(stack),spotWorkerNodes:true,workerMinCapacity:3,workerInstanceType:newec2.InstanceType('m6g.medium'),controlPlaneInstanceType:newec2.InstanceType('m6g.medium')bucketRemovalPolicy:cdk.RemovalPolicy.DESTROY})In thistypescriptexample, we are using all of the properties available today. Note that we set the bucket removal policy toDESTROY(this will remove completely the S3 bucket - the safe default behavior is to leave the bucket in the account). For an up to date list of all the properties please refer to theAPI.mdfile in this repo.I am a Python type of personThis is an example of how to consume thePyPimodule with a CDK application written in Typescript. If you need to setup your Python environmentthis is a good guide.$mkdirmyk3scluster-python
$cdmyk3scluster-python# initialize the AWS CDK project$cdkinit-lpython# activate the Python virtual environment$source.env/bin/activate# install the dependencies$pipinstallcdk-k3s-clusterUpdate your./app.pyfile with the following content. Note how, in this example, we are only using two of the parameters available. To use all the defaults you could usecluster = k3s.Cluster(self, "MyK3sClusters"):#!/usr/bin/env python3fromaws_cdkimportcoreimportcdk_k3s_clusterask3sclassK3sCluster(core.Stack):def__init__(self,scope:core.Construct,id:str,**kwargs):super().__init__(scope,id,**kwargs)k3s.Cluster(self,"MyK3sClusters",worker_min_capacity=5,spot_worker_nodes=True)app=core.App()K3sCluster(app,"K3sCluster")app.synth()Deploying the stack (regardless of the language you have used)deploy the CDK stack:# see the difference before the deployment$cdkdiff# deploy it$cdkdeployIf you want to deploy in an existing VPC use eithercdk deploy --context use_default_vpc=1orcdk deploy --context use_vpc_id=<vpc id>Clean upCleaning up the environment is as easy as runningcdk destroyfrom where you left your prompt.Known issues and limitationsFirst and foremost this is a learning experiment. We have done limited tests with it.We have not tested this beyond a merekubectl get nodestest. Let alone trying anything likearkadecdk-k3s-clusteronly deploys Arm-based instances. It would be trivial to add x86 based instances support but it's not there todayAll the control plane and worker nodes are deployed in public subnets and the SGs are fairly permissive in terms of "source". Picking private subnets would have probably broken the use case of deploying into thedefaultVPC (which is handy). This prototype over-indexes more on deployment convenience and ease of use than on best practices. Be mindful of thatThe control plane instance always deployson-demandwhile for worker nodes you can pick betweenon-demandandSpotThe ASG for worker nodes is configured with a single parameter that becomes themin,maxanddesiredcount for the ASGFor simplicity, both the control plane instance and the worker nodes share the same instance typeCreditsThis library has been authored byMassimoandPahud.