Introducing DataJob — build and deploy a serverless data pipeline on AWS | by Vincent Claes | Towards Data Science

One of the core activities of a data engineer is to build, deploy, run and monitor data pipelines. While working in the field of data and ML engineering I lacked a tool that eased the process of deploying my data pipeline on AWS services like Glue and Sagemaker and how to easily orchestrate the steps of my data pipeline with Step Functions. This made me develop DataJob! 🚀
In this article I will show you how to install DataJob, guide you through a simple example, and showcase a couple of features of DataJob.
To continue my development i would like to get some feedback from the community to either continue and add additional services (lambda, ecs fargate, aws batch, ...), change direction or abbandon this open source project.
Let me know in a response! 🙏
github.com
You can install DataJob from PyPI. DataJob uses AWS CDK to provision AWS services, so make sure to install this as well. If you want to follow the example, you need an AWS account of course 🙂
pip install --upgrade pippip install datajob# take latest of v1, there is no support for v2 yetnpm install -g aws-cdk@1.134.0
We have a simple data pipeline composed of 2 tasks that print “Hello World” and these tasks need to be orchestrated sequentially. The tasks are deployed to Glue and orchestrated by Step Functions.
We add the above code in a file called datajob_stack.py in the root of the project. This file contains everything you need to configure the AWS services, deploy your code, and run your data pipeline.
To continue, clone the repo and navigate to the example.
git clone https://github.com/vincentclaes/datajob.gitcd datajob/examples/data_pipeline_simple
To configure CDK you need AWS credentials. If you don’t know how to configure your AWS credentials, follow the steps here.
export AWS_PROFILE=default# use the aws cli to get your account numberexport AWS_ACCOUNT=$(aws sts get-caller-identity --query Account --output text --profile $AWS_PROFILE)export AWS_DEFAULT_REGION=eu-west-1# bootstrap aws account for your regioncdk bootstrap aws://$AWS_ACCOUNT/$AWS_DEFAULT_REGION    ⏳  Bootstrapping environment aws://01234567890/eu-west-1...    CDKToolkit: creating CloudFormation changeset...    ✅  Environment aws://01234567890/eu-west-1 bootstrapped.
Create the DataJob stack with the Glue jobs that contain you code and the Step Functions state machine that will orchestrate the Glue jobs.
cdk deploy --app  "python datajob_stack.py" --require-approval never     data-pipeline-simple: deploying...     [0%] start: Publishing          [50%] success: Published      [100%] success: Published      data-pipeline-simple: creating CloudFormation changeset...     ✅  data-pipeline-simple
when cdk deploy successfully finishes, the services are configured and ready to be executed.
Trigger the Step Functions state machine that will orchestrate the data pipeline.
datajob execute --state-machine data-pipeline-simple-workflow    executing: data-pipeline-simple-workflow    status: RUNNING    view the execution on the AWS console:       <here will be a link to see the step functions workflow>
The terminal will show a link to the step functions web page to follow up on your pipeline run. If you click the link you should see something like the following:
Once your data pipeline has finished, remove it from AWS. This will leave you with a clean AWS account.
cdk destroy --app  "python datajob_stack.py"    data-pipeline-simple: destroying...    ✅  data-pipeline-simple: destroyed
Find out more in the examples.
Specify a stage as a context parameter in CDK to deploy an isolated pipeline. Typical examples are dev , prod, ...
cdk deploy --app "python datajob_stack.py" --context stage=dev
To speed up your data pipeline you might want to run tasks in parallel. This is possible with DataJob! I borrowed the concept of Airflow where you can use the operator >> to orchestrate the different tasks.
with StepfunctionsWorkflow(datajob_stack=datajob_stack, name="workflow") as sfn:    task1 >> task2    task3 >> task4    task2 >> task5    task4 >> task5
DataJob figures out what tasks can run in parallel to speed up the execution.
Find out more in the examples.
Provide the parameter notification with an email address in the constructor of a StepfunctionsWorkflow object. This will create an SNS Topic which will be triggered in case of failure or success. The email will receive a notification in its inbox.
with StepfunctionsWorkflow(datajob_stack=datajob_stack,                           name="workflow",                           notification="email@domain.com") as sfn:    task1 >> task2
Ship your project with all its dependencies to Glue. By specifying project_root in the constructor of DataJobStack, DataJob will look for a wheel (.whl file) in the dist/ folder in your project_root.
current_dir = str(pathlib.Path(__file__).parent.absolute())with DataJobStack(    scope=app, id="data-pipeline-pkg", project_root=current_dir) as datajob_stack:
Find out more in the examples
Check the new example of an End-to-end Machine Learning Pipeline on the GitHub repo with Glue, Sagemaker, and Step functions.
⚠️ Give me a response on what you like, didn't like, and what other services you would want to see next in DataJob! ⚠️
github.com
👋 Follow me on Medium, Linkedin, and Twitter if you want to read more on ML engineering and ML pipelines.