Package: datajob
Build and deploy a serverless data pipeline on AWS with no effort.Our goal is to let developers think about the business logic, datajob does the rest...Deploy code to python shell / pysparkAWS Glue jobs.UseAWS Sagemakerto create ML Models.Orchestrate the above jobs usingAWS Stepfunctionsas simple astask1 >> task2Let usknowwhat you want to see next.:rocket: :new: :rocket:Check our new example of an End-to-end Machine Learning Pipeline with Glue, Sagemaker and Stepfunctions:rocket: :new: :rocket:InstallationDatajob can be installed using pip.Beware that we depend onaws cdk cli!pip install datajob
npm install -g aws-cdk@1.109.0 # latest version of datajob depends this versionQuickstartYou can find the full example inexamples/data_pipeline_simple.We have a simple data pipeline composed of2 glue jobsorchestrated sequentially using step functions.fromaws_cdkimportcorefromdatajob.datajob_stackimportDataJobStackfromdatajob.glue.glue_jobimportGlueJobfromdatajob.stepfunctions.stepfunctions_workflowimportStepfunctionsWorkflowapp=core.App()# The datajob_stack is the instance that will result in a cloudformation stack.# We inject the datajob_stack object through all the resources that we want to add.withDataJobStack(scope=app,id="data-pipeline-simple")asdatajob_stack:# We define 2 glue jobs with the relative path to the source code.task1=GlueJob(datajob_stack=datajob_stack,name="task1",job_path="glue_jobs/task.py")task2=GlueJob(datajob_stack=datajob_stack,name="task2",job_path="glue_jobs/task2.py")# We instantiate a step functions workflow and orchestrate the glue jobs.withStepfunctionsWorkflow(datajob_stack=datajob_stack,name="workflow")assfn:task1>>task2app.synth()We add the above code in a file calleddatajob_stack.pyin theroot of the project.Configure CDKFollow the stepshereto configure your credentials.exportAWS_PROFILE=default# use the aws cli to get your account numberexportAWS_ACCOUNT=$(awsstsget-caller-identity--queryAccount--outputtext--profile$AWS_PROFILE)exportAWS_DEFAULT_REGION=eu-west-1# init cdkcdkbootstrapaws://$AWS_ACCOUNT/$AWS_DEFAULT_REGIONDeployDeploy the pipeline using CDK.cdexamples/data_pipeline_simple
cdkdeploy--app"python datajob_stack.py"--require-approvalneverExecutedatajobexecute--state-machinedata-pipeline-simple-workflowThe terminal will show a link to the step functions page to follow up on your pipeline run.Destroycdkdestroy--app"python datajob_stack.py"ExamplesData pipeline with parallel stepsData pipeline for processing big data using PySparkData pipeline where you package and ship your project as a wheelMachine Learning pipeline where we combine glue jobs with sagemakerAll our examples are in./examplesFunctionalityDeploy to a stageSpecify a stage to deploy an isolated pipeline.Typical examples would bedev,prod, ...cdkdeploy--app"python datajob_stack.py"--contextstage=my-stageUsing datajob's S3 data bucketDynamically reference thedatajob_stackdata bucket name to the arguments of your GlueJob by callingdatajob_stack.context.data_bucket_name.importpathlibfromaws_cdkimportcorefromdatajob.datajob_stackimportDataJobStackfromdatajob.glue.glue_jobimportGlueJobfromdatajob.stepfunctions.stepfunctions_workflowimportStepfunctionsWorkflowcurrent_dir=str(pathlib.Path(__file__).parent.absolute())app=core.App()withDataJobStack(scope=app,id="datajob-python-pyspark",project_root=current_dir)asdatajob_stack:pyspark_job=GlueJob(datajob_stack=datajob_stack,name="pyspark-job",job_path="glue_job/glue_pyspark_example.py",job_type="glueetl",glue_version="2.0",# we only support glue 2.0python_version="3",worker_type="Standard",# options are Standard / G.1X / G.2Xnumber_of_workers=1,arguments={"--source":f"s3://{datajob_stack.context.data_bucket_name}/raw/iris_dataset.csv","--destination":f"s3://{datajob_stack.context.data_bucket_name}/target/pyspark_job/iris_dataset.parquet",},)withStepfunctionsWorkflow(datajob_stack=datajob_stack,name="workflow")assfn:pyspark_job>>...you can find this examplehereDeploy files to the datajob's deployment bucketSpecify the path to the folder we would like to include in the deployment bucket.fromaws_cdkimportcorefromdatajob.datajob_stackimportDataJobStackapp=core.App()withDataJobStack(scope=app,id="some-stack-name",include_folder="path/to/folder/")asdatajob_stack:...Package your project as a wheel and ship it to AWSYou can find the examplehere# We add the path to the project root in the constructor of DataJobStack.# By specifying project_root, datajob will look for a .whl in# the dist/ folder in your project_root.withDataJobStack(scope=app,id="data-pipeline-pkg",project_root=current_dir)asdatajob_stack:Package you project usingpoetrypoetrybuild
cdkdeploy--app"python datajob_stack.py"Package you project usingsetup.pypythonsetup.pybdist_wheel
cdkdeploy--app"python datajob_stack.py"you can also use the datajob cli to do the two commands at once:# for poetrydatajobdeploy--configdatajob_stack.py--packagepoetry# for setup.pydatajobdeploy--configdatajob_stack.py--packagesetuppyProcessing big data using a Glue Pyspark jobimportpathlibfromaws_cdkimportcorefromdatajob.datajob_stackimportDataJobStackfromdatajob.glue.glue_jobimportGlueJobcurrent_dir=str(pathlib.Path(__file__).parent.absolute())app=core.App()withDataJobStack(scope=app,id="datajob-python-pyspark",project_root=current_dir)asdatajob_stack:pyspark_job=GlueJob(datajob_stack=datajob_stack,name="pyspark-job",job_path="glue_job/glue_pyspark_example.py",job_type="glueetl",glue_version="2.0",# we only support glue 2.0python_version="3",worker_type="Standard",# options are Standard / G.1X / G.2Xnumber_of_workers=1,arguments={"--source":f"s3://{datajob_stack.context.data_bucket_name}/raw/iris_dataset.csv","--destination":f"s3://{datajob_stack.context.data_bucket_name}/target/pyspark_job/iris_dataset.parquet",},)full example can be found inexamples/data_pipeline_pyspark.Orchestrate stepfunctions tasks in parallel# Task2 comes after task1. task4 comes after task3.# Task 5 depends on both task2 and task4 to be finished.# Therefore task1 and task2 can run in parallel,# as well as task3 and task4.withStepfunctionsWorkflow(datajob_stack=datajob_stack,name="workflow")assfn:task1>>task2task3>>task4task2>>task5task4>>task5More can be found inexamples/data_pipeline_parallelOrchestrate 1 stepfunction taskUse theEllipsisobject to be able to orchestrate 1 job via step functions.some_task>>...Notify in case of error/successProvide the parameternotificationin the constructor of aStepfunctionsWorkflowobject.
This will create an SNS Topic which will be triggered in case of failure or success.
The email will subscribe to the topic and receive the notification in its inbox.withStepfunctionsWorkflow(datajob_stack=datajob_stack,name="workflow",notification="email@domain.com")assfn:task1>>task2You can provide 1 email or a list of emails["email1@domain.com", "email2@domain.com"].Datajob in depthThedatajob_stackis the instance that will result in a cloudformation stack.
The path inproject_roothelpsdatajob_stacklocate the root of the project where
the setup.py/poetry pyproject.toml file can be found, as well as thedist/folder with the wheel of your project .importpathlibfromaws_cdkimportcorefromdatajob.datajob_stackimportDataJobStackcurrent_dir=pathlib.Path(__file__).parent.absolute()app=core.App()withDataJobStack(scope=app,id="data-pipeline-pkg",project_root=current_dir)asdatajob_stack:...Whenentering the contextmanagerof DataJobStack:ADataJobContextis initialized
to deploy and run a data pipeline on AWS.
The following resources are created:"data bucket"an S3 bucket that you can use to dump ingested data, dump intermediate results and the final output.you can access the data bucket as aBucketobject viadatajob_stack.context.data_bucketyou can access the data bucket name viadatajob_stack.context.data_bucket_name"deployment bucket"an s3 bucket to deploy code, artifacts, scripts, config, files, ...you can access the deployment bucket as aBucketobject viadatajob_stack.context.deployment_bucketyou can access the deployment bucket name viadatajob_stack.context.deployment_bucket_namewhenexiting the context managerall the resources of our DataJobStack object are created.We can write the above example more explicitly...importpathlibfromaws_cdkimportcorefromdatajob.datajob_stackimportDataJobStackfromdatajob.glue.glue_jobimportGlueJobfromdatajob.stepfunctions.stepfunctions_workflowimportStepfunctionsWorkflowcurrent_dir=pathlib.Path(__file__).parent.absolute()app=core.App()datajob_stack=DataJobStack(scope=app,id="data-pipeline-pkg",project_root=current_dir)datajob_stack.init_datajob_context()task1=GlueJob(datajob_stack=datajob_stack,name="task1",job_path="glue_jobs/task.py")task2=GlueJob(datajob_stack=datajob_stack,name="task2",job_path="glue_jobs/task2.py")withStepfunctionsWorkflow(datajob_stack=datajob_stack,name="workflow")asstep_functions_workflow:task1>>task2datajob_stack.create_resources()app.synth()IdeasAny suggestions can be shared by starting adiscussionThese are the ideas, we find interesting to implement;add a time based trigger to the step functions workflow.add an s3 event trigger to the step functions workflow.add a lambda that copies data from one s3 location to another.version your data pipeline.cli command to view the logs / glue jobs / s3 bucketimplement sagemaker servicesprocessing jobshyperparameter tuning jobstraining jobsimplement lambdaimplement ECS Fargatecreate a serverless UI that follows up on the different pipelines deployed on possibly different AWS accounts using DatajobFeedbackis much appreciated!