Package: airflow-cdk
airflow-cdkThis project makes it simple to deploy airflow via ECS fargate using the aws cdk in Python.UsageThere are two main ways that this package can be used.Standalone PackageFor those already familiar with the aws cdk, add this project
as a dependency i.e.pip install airflow-cdkand/or add torequirement.txtand use theFargateAirflowconstruct like so.fromaws_cdkimportcorefromairflow_cdkimportFargateAirflowapp=core.App()FargateAirflow(app,"airflow-cdk",postgres_password="replacethiswithasecretpassword")app.synth()cdk deployThat's it.CloningYou can also clone this repository and alter theFargateAirflowconstruct
to your heart's content.That also provides you an added benefit of utilizing thetasks.pytasks
withinvoketo do things like create new dags easily
i.e.inv new-dagYou would then also easily be able to use the existing docker-compose for local development
with some minor modifications for your setup.The easiest way to get started would be just a one-line change to theapp.pyexample above
and to thedocker-compose.ymlfile.fromaws_cdkimportcorefromairflow_cdkimportFargateAirflowapp=core.App()FargateAirflow(app,"airflow-cdk",postgres_password="replacethiswithasecretpassword"),# this is the only change to make when cloningbase_image=aws_ecs.ContainerImage.from_asset(".")app.synth()Then, in thedocker-compose.ymlfile, simply delete, comment out, or change the image name
for theimage: knowsuchagency/airflow-cdkline inx-airflow.Now the same container that would be created bydocker-compose buildwill be deployed to ECS for your web, worker, and
scheduler images bycdk deploy.ComponentsThe following aws resources will be deployed as ecs tasks within the same cluster and vpc by default:an airflow webserver taskand an internet-facing application load-balanceran airflow scheduler taskan airflow worker task(note) it will auto-scale based on cpu and memory usage up to a total of 16 instances at a time by default starting from 1a rabbitmq brokeran application load balancer that will allow you to log in to
the rabbitmq management console with the default user/pw guest/guestan rds instancean s3 bucket for logsWhy is this awesome?Apart from the fact that we're able to describe our infrastructure using the same language and codebase we use to author
our dags?Since we're using cloudformation under-the-hood, whenever we change a part of our code or infrastructure, only those
changes that are different from our last deployment will be deployed.Meaning, if all we do is alter the code we want to run on our deployment, we simply re-build and publish our docker
container (which is done for us if we useaws_ecs.ContainerImage.from_asset(".")) prior tocdk deploy!Existing users of airflow will know how tricky it can be to manage deployments when you want to distinguish between
pushing changes to your codebase i.e. dags and actual infrastructure deployments.We just have to be careful not to deploy while we have some long-running worker task we don't want to interrupt since
fargate will replace those worker instances with new ones running our updated code.
Now there's basically no distinction.NotesBefore runningcdk destroy, you will want to empty the s3 bucket that's created otherwise the command may fail at
that stage and the bucket can be left in a state that makes it difficult to delete later onTODOscreate a custom component to deploy airflow to an ec2 clusterimprove documentation(possibly) subsume theairflow stable helm chartas a cdk8s chartContributions Welcome!