Distributed data pipelines made easy with AWS EKS and Prefect | by Anna Geller | Towards Data Science

Building distributed systems for ETL & ML data pipelines is hard. If you tried implementing one yourself, you may have experienced that tying together a workflow orchestration solution with distributed multi-node compute clusters such as Spark or Dask may prove difficult to properly set up and manage. In this article, I want to show you how you can obtain a highly available, scalable, distributed system that will make the orchestration of your data pipelines for ETL & ML much more enjoyable and will free up your time to work with data and generate value out of it rather than spend time on maintaining clusters.
The entire setup is, to a large extent, automated by AWS and Prefect. Plus, it will cost you almost nothing in the beginning. Why? Because I will show you how to set up a production-ready and secure workflow orchestration system with a serverless Kubernetes cluster as your distributed, fault-tolerant, highly available, self-healing & automatically scalable execution layer within minutes!
All that you need to do is:
create AWS account & IAM user with programmatic access to create AWS ECR, ECS & EKS resources
run a few shell commands from this article to set up a Kubernetes cluster by using AWS EKS on Fargate (which will cost you just 0.10$ per hour for the control plane! Before, it was 0.20$ but AWS halved the price in early 2020)
sign up for a free Prefect Cloud account (free Developer account gives you access to all features but only for you — if you want to use it in your team, you need to upgrade to either Team or Enterprise plan)
generate authentification tokens within Prefect Cloud UI to connect your Prefect Cloud account to your serverless Kubernetes cluster on AWS
run your distributed data pipelines and generate value out of your data!
Let’s get started!
In December 2019, AWS launched a new Fargate feature which, to many, was considered a game-changer — they introduced an option to use AWS EKS on ECS Fargate, which is a way of saying: AWS made the Fargate service an orchestrator not only for the ECS but also for the EKS. Up to that point, AWS Fargate was a serverless way of running containers only on AWS ECS.
EKS and Fargate make it straightforward to run Kubernetes-based applications on AWS by removing the need to provision and manage infrastructure for pods.
What does it mean for us? It means that we can now have a serverless Kubernetes cluster on EKS which is only charging us for running pods, rather than their underlying EC2 instances. It also means, among other benefits:
no more worker node maintenance,
no more guessing capacity,
no more EC2 autoscaling groups to scale the worker nodes up and down.
AWS takes care of all that for you. All that you need to do is to write YAML files for your deployments and interact with the EKS via kubectl. In short: your only task now is to write your ETL & ML code that adds value to your business and AWS takes care of the Ops, i.e. operating, maintaining, and scaling your Kubernetes cluster.
Considering that we are charged only for the actual vCPU and memory of running pods, this provides a great foundation for a modern data platform.
One possible disadvantage of using almost any serverless option is the issue of cold start when the container orchestration system needs to first allocate and prepare the compute resources (a.o. pulling the latest version of the image from the docker registry to the allocated worker node and building the image) which may add some additional latency before the container (or your K8s pod) will turn into a running state.
If your data workloads require a very low level of latency, you may opt for the AWS EKS cluster with the traditional data plane and follow the instructions from this blog post to set up a non-serverless cluster on AWS EKS and to connect it to your Prefect Cloud environment.
However, you can have both! AWS allows you to mix the two:
you can have the same AWS EKS cluster running pods in a serverless way in the default namespace (this is set up by generating a Fargate Profile)
and you can have an EC2 instance (ex. with GPU) for your Data Science models connected to the same Kubernetes cluster on EKS but within a different namespace or using different Kubernetes labels. When you then create a deployment for a pod that doesn’t match the namespace and labels defined in the Fargate Profile, it will be scheduled to the EC2-worker node which you maintain and which is available with no latency.
As you can see, AWS designed EKS on Fargate with a lot of foresight which allows you to mix the serverless and non-serverless options to save your time, money, and maintenance efforts. You can find out more about that in this video in which AWS introduced the service.
You need to have an AWS account with either admin access or at least a user with IAM permissions for creating ECR, EKS and ECS resources. Then, you must have AWS CLI configured for this account and eksctl installed, as described in this AWS docs.
Kubernetes on AWS works well with AWS ECR, which is a registry for your Docker images. To authenticate your terminal with your ECR account, run:
if you use the new AWS CLI v2:
aws ecr get-login-password --region <YOUR_AWS_REGION> | docker login --username AWS --password-stdin <YOUR_ECR_REGISTRY_ID>.dkr.ecr.<YOUR_AWS_REGION>.amazonaws.com
if you use the old AWS CLI version:
$(aws ecr get-login --no-include-email --region <YOUR_AWS_REGION>)
Note: <YOUR_AWS_REGION> could be ex. us-east-1, eu-central-1, and more.
If you get Login Succeeded message, you can create your ECR repositories for your data pipelines. We will create two data pipelines: dask-k8 and basic-etl-prefect-flow — use the same names to follow this walkthrough, but in general, it’s easiest to give your ECR repository the same name as your Prefect flow to avoid confusion.
aws ecr create-repository --repository-name dask-k8aws ecr create-repository --repository-name basic-etl-prefect-flow
Then, all you need to do is to run the following command, which will deploy Kubernetes control plane and Fargate Profile in your VPC:
eksctl create cluster --name fargate-eks --region <YOUR_AWS_REGION> --fargate
I picked the name fargate-eks for the cluster, but feel free to change it. The --fargate flag ensures that we create a new Fargate profile for use with this cluster. EKS allows you to create a custom Fargate profile if needed.
The provision of all resources may take several minutes. When finished, you should see output similar to this one:
Then, if you check your context:
kubectl config current-context
You should get output similar to this one:
<YOUR_AWS_USER_NAME>@fargate-eks.<YOUR_AWS_REGION>.eksctl.io
This way you can see that you are connected to a serverless Kubernetes cluster running on AWS Fargate! To prove it further, run:
➜  ~ kubectl get nodesNAME                                                       STATUS   ROLES    AGE   VERSIONfargate-ip-192-168-163-163.eu-central-1.compute.internal   Ready    <none>   15m   v1.17.9-eks-a84824fargate-ip-192-168-180-51.eu-central-1.compute.internal    Ready    <none>   15m   v1.17.9-eks-a84824
In the output, you should see at least 1 Fargate node waiting for your pod deployments.
Note: those nodes are running inside of your VPC but they are not visible within your EC2 dashboard. You cannot SSH to those nodes either, as they are fully managed and deployed by Fargate in a serverless fashion.
The advantage of combining this AWS EKS cluster with Prefect is that the entire Kubernetes pod deployment and scheduling is abstracted away from you by Prefect. This means that you don’t even need to know much about Kubernetes in order to derive value from it. In the next section, we will connect this cluster to our Prefect Cloud account and start building distributed ETL & ML data pipelines.
Let’s first sign up for a free Developer account on https://cloud.prefect.io/.
At first, you will be welcomed by a clean UI showing you your flows, agents, and the general overview of the recent flow runs and the next scheduled jobs. The flows themselves can be organized into several projects. When you start building your data pipelines, this main dashboard lets you quickly identify the current status of all your data pipelines. This dashboard is extremely helpful: imagine that you log into your account in the morning and see that all your pipelines are in a green status! You only need to dive deeper if you see any red bars indicating some failed data flows.
I know companies who would dream of having this kind of dashboard and this level of transparency about their ETL & ML data pipelines and their status, while at the same time being able to see that all agents executing the work are healthy: at the moment you can see that I currently have a Fargate agent ready to deploy the flows on AWS ECS. For now, we focus on AWS EKS.
Let’s continue with the setup and install prefect on your computer. The following command will install Prefect with AWS and Kubernetes extras (instead of [kubernetes,aws] you could use [all_extras] if you want to install all Prefect extensions to external systems):
pip install "prefect[kubernetes,aws]"
Now to make sure that you use Prefect Cloud and not the open-source version Prefect Core, switch the context to Cloud:
prefect backend cloud
After you registered for a free account, you need to create a Personal Access Token to authenticate your local terminal with Prefect Cloud. This will allow to register your flows (i.e. your ETL & ML data pipelines) to the Prefect Cloud directly from your computer. Go to the side bar: User → Personal Access Token → + CREATE TOKEN button.
Choose some meaningful name ex. MyTokenToRegisterFlows.
Then copy the token and within your terminal run the following commands:
prefect auth login -t <MyTokenToRegisterFlows>
Now you can register your flows to be orchestrated from Prefect Cloud!
The last part is to create a RunnerToken for your Kubernetes Agent and to register the agent. Go to the sidebar: Team → API Tokens → + CREATE TOKEN button.
Alternatively, you could do the same from your terminal:
prefect auth create-token -n MyEKS_on_Fargate_K8s_Token -r RUNNER
It’s very important to select the RUNNER scope, otherwise your agent will not be able to execute the flows on your behalf.
Click on CREATE and copy the generated API token.
Now we are reaching the most exciting part: with the command below, you will be able to set your serverless AWS Kubernetes cluster as your execution layer (i.e. agent) for your Prefect data pipelines:
Now you should be able to see a new Kubernetes Agent in your Prefect Cloud account:
We can also see a new pod corresponding to the Prefect agent:
➜  ~ kubectl get podsNAME                             READY   STATUS    RESTARTS   AGEprefect-agent-68785f47d4-pv9kt   1/1     Running   0          5m59s
Now everything is set up and we can start creating our flows. Prefect Docs include a variety of useful tutorials that will quickly show you how to adapt your Python ETL & ML code to make it run on Prefect. In a nutshell, you just need to decorate your Python functions with @task decorator, add the task and flow imports and create a flow object:
Note: adding log_stdout=True ensures that the printed output will appear in the Prefect Cloud flow logs.
Now we can create a project to organize our flows either from the UI or by using the terminal:
➜ prefect create project Medium_AWS_PrefectMedium_AWS_Prefect created
If we now run this script, we should get a link to the Prefect Cloud UI from which we can trigger or schedule the flow:
➜ python3 basic-prefect-etl-flow.py Result check: OKFlow: https://cloud.prefect.io/anna/flow/888046e6-f366-466a-b9b5-4113cd437e4d
When we trigger the flow, we will see that it stays in the scheduled state and doesn’t run.
This is because to run your flows on the AWS EKS cluster, your flow must include information on where your code is stored.
There are several options for how you can store your code: EKS could pull it from ECR, S3, or Github.
The easiest option is to dockerize your flow and push the image to ECR. Luckily, Prefect team made it really easy — we only need to:
create an ECR repository for our flow (this step can be automated with a CI/CD pipelines in your production environments)
add Docker storage to the code.
If you remember the AWS Setup section, we already created two ECR respositories: dask-k8 and basic-etl-prefect-flow. Therefore, we only need to add storage=Docker() argument to our flow code so that it can be executed by our serverless Kubernetes Agent:
Some important notes to the code above:
under the hood, Prefect checks your Prefect version and extends the corresponding Prefect Docker image to include your flow and its dependencies,
make sure to set your ECR registry ID so that your flow will be dockerized and pushed to the ECR repository dask-k8 that we created earlier
image_tag='latest' is used to disable versioning of your ECR images. Setting it to latest, or any other particular tag will make sure that every time you register a flow, you overwrite the previous ECR image tagged as latest. This works well for me because I already use Git to version-control my code and I don’t need ECR versioning. Also, it can save you some money on AWS, as with each new version of the image on ECR, you pay more for storage of those ECR images. But feel free to skip this argument. This way, Prefect will tag the image with the current date and time and will store each flow version with a different tag.
Last note: the extra argument python_dependencies=["pandas==1.1.0"] allows to define a list of Python packages that needs to be installed within the container. If you need a more finegrained control over your image, you can provide a path to a custom Dockerfile, ex: dockerfile='/Users/anna/my_flows/aws/Dockerfile'
We can now mark the previous flow run as Finished or Failed:
Finally, we can register a new version of the flow by simply rerunning the modified script, as shown in the second Gist:
If we visit the link from the output, we can see that a Version 2 has been created and if we run it, we can see that the flow will transfer to the state Submitted for execution (yellow), then Runnning (blue), and finally All reference tasks succeeded (green):
The logs display more information:
We can also inspect the pods by using kubectl:
To summarize what we did so far:
we created a Prefect Cloud account
we deployed a serverless Kubernetes cluster by using AWS EKS
we created an ECR repository and pushed our Docker image containing our flow code and all Python package dependencies to this ECR repository
we then registered our flow, ran it from the Prefect Cloud UI, and inspected its status in the UI and in the CLI using kubectl.
If you came that far, congratulations! 👏🏻
In the next section, we will create a second flow which will make use of a distributed Dask cluster.
Prefect Cloud works great with Dask Distributed. In order to run your Python code on Dask in a distributed fashion, you would typically have to deploy a Dask cluster with several worker nodes and maintain it. Prefect provides a great abstraction DaskKubernetesEnvironment which:
spins up on-demand Dask cluster across multiple pods and possibly also across multiple nodes (you can specify the min and max number of workers)
submits your flow to this on-demand cluster
cleans up the resources (i.e. terminates the cluster after the job is finished).
Here is an example flow based on the Prefect Docs that you can use to test your Dask setup. I saved this flow as dask-k8.py and provided the same name dask-k8 as the flow name and as the name of ECR repository:
We now register the flow and trigger it from the UI again:
➜  python3 dask-k8.py
Now we can observe the pods that are being created to distribute work across Dask workers:
We can now see several pods related to Dask. The UI shows the current progress of all tasks, running in parallel, and the final runtime:
If we analyze the logs, we can see that:
the flow was submitted for execution at 12:43
the flow run began at 12:47
the flow finished at 12:51.
This means that it took around 3 minutes until the flow has been picked up by the Kubernetes agent and the Dask cluster has been provisioned on our serverless AWS EKS cluster together with pulling all necessary images. After that, the flow needed 4 minutes to successfully run the flow.
If the latency related to a Dask cluster being created on-demand is not acceptable by your workloads, you can create and maintain your own Dask cluster and adapt the code as follows:
This way, you would use Executor rather than Environment abstraction. Also, you would have to adapt 255.255.255.255 to your Dask scheduler address and change port 8786 accordingly, if needed.
Before we wrap up, make sure to delete the AWS EKS cluster and the ECR repositories to avoid any charges:
In this article, we used AWS EKS on Fargate to create a serverless Kubernetes cluster on AWS. We connected it in a secure way as our execution layer with Prefect Cloud. Then, we dockerized our Prefect flows and pushed the images to ECR by using DockerStorage abstraction. Finally, we deployed both, a simple data pipeline running within a single pod, as well as distributed Dask flow, allowing for high-level parallelism of your ETL & ML code.
Along the way, we identified the differences, as well as the pros and cons of running data pipelines on already available resources that we need to maintain vs. running your containerized flows in a serverless way.
Hopefully, this setup will make it easier to start orchestrating your ETL & Data Science workflows. Regardless of whether you are a start-up, large enterprise, or a student running code for a thesis, the combination of Prefect & AWS EKS on Fargate allows you to move your data projects to production faster than ever before.
Thank you for reading!