Package: cdk-databrew-cicd
cdk-databrew-cicdThis construct creates aCodePipelinepipeline where users can push aDataBrewrecipe into theCodeCommitrepository and the recipe will be pushed to a pre-production AWS account and a production AWS account by order automatically.npm (JS/TS)PyPI (Python)Maven (Java)GoNuGetLinkLinkLinkLinkLinkTable of ContentsServerless ArchitectureIntroductionExampleTypescriptPythonJavaC#GoSome Efforts after Stack CreationHow Successful Commits Look LikeServerless ArchitectureRomi B. and Gaurav W., 2021IntroductionThe architecture was introduced byRomi BoimerandGaurav Wadhawanand was posted on the AWS Blog asSet up CI/CD pipelines for AWS Glue DataBrew using AWS Developer Tools.
I converted the architecture into a CDK construct for 4 programming languages. Before applying the AWS construct, make sure you've set up a proper IAM role for the pre-production and production AWS accounts. You could achieve it either by creating manually or creating through a custom construct in this library.import{IamRole}from'cdk-databrew-cicd';newIamRole(this,'AccountIamRole',{environment:'preproduction',//or'production'accountID:'ACCOUNT_ID',//roleName:'OPTIONAL'});ExampleTypescriptYou could also refer tohere.$cdk--initlanguagetypescript
$yarnaddcdk-databrew-cicdimport*ascdkfrom'@aws-cdk/core';import{DataBrewCodePipeline}from'cdk-databrew-cicd';classTypescriptStackextendscdk.Stack{constructor(scope:cdk.Construct,id:string,props?:cdk.StackProps){super(scope,id,props);constpreproductionAccountId='PREPRODUCTION_ACCOUNT_ID';constproductionAccountId='PRODUCTION_ACCOUNT_ID';constdataBrewPipeline=newDataBrewCodePipeline(this,'DataBrewCicdPipeline',{preproductionIamRoleArn:`arn:${cdk.Aws.PARTITION}:iam::${preproductionAccountId}:role/preproduction-Databrew-Cicd-Role`,productionIamRoleArn:`arn:${cdk.Aws.PARTITION}:iam::${productionAccountId}:role/production-Databrew-Cicd-Role`,//bucketName:'OPTIONAL',//repoName:'OPTIONAL',//branchName:'OPTIONAL',//pipelineName:'OPTIONAL'});newcdk.CfnOutput(this,'OPreproductionLambdaArn',{value:dataBrewPipeline.preproductionFunctionArn});newcdk.CfnOutput(this,'OProductionLambdaArn',{value:dataBrewPipeline.productionFunctionArn});newcdk.CfnOutput(this,'OCodeCommitRepoArn',{value:dataBrewPipeline.codeCommitRepoArn});newcdk.CfnOutput(this,'OCodePipelineArn',{value:dataBrewPipeline.codePipelineArn});}}constapp=newcdk.App();newTypescriptStack(app,'TypescriptStack',{stackName:'DataBrew-CICD'});PythonYou could also refer tohere.# upgrading related Python packages$python-mensurepip--upgrade
$python-mpipinstall--upgradepip
$python-mpipinstall--upgradevirtualenv# initialize a CDK Python project$cdkinit--languagepython# make packages installed locally instead of globally$source.venv/bin/activate
$cat<<EOL > requirements.txtaws-cdk.corecdk-databrew-cicdEOL$python-mpipinstall-rrequirements.txtfromaws_cdkimportcoreascdkfromcdk_databrew_cicdimportDataBrewCodePipelineclassPythonStack(cdk.Stack):def__init__(self,scope:cdk.Construct,construct_id:str,**kwargs)->None:super().__init__(scope,construct_id,**kwargs)preproduction_account_id="PREPRODUCTION_ACCOUNT_ID"production_account_id="PRODUCTION_ACCOUNT_ID"databrew_pipeline=DataBrewCodePipeline(self,"DataBrewCicdPipeline",preproduction_iam_role_arn=f"arn:{cdk.Aws.PARTITION}:iam::{preproduction_account_id}:role/preproduction-Databrew-Cicd-Role",production_iam_role_arn=f"arn:{cdk.Aws.PARTITION}:iam::{production_account_id}:role/preproduction-Databrew-Cicd-Role",# bucket_name="OPTIONAL",# repo_name="OPTIONAL",# repo_name="OPTIONAL",# branch_namne="OPTIONAL",# pipeline_name="OPTIONAL")cdk.CfnOutput(self,'OPreproductionLambdaArn',value=databrew_pipeline.preproduction_function_arn)cdk.CfnOutput(self,'OProductionLambdaArn',value=databrew_pipeline.production_function_arn)cdk.CfnOutput(self,'OCodeCommitRepoArn',value=databrew_pipeline.code_commit_repo_arn)cdk.CfnOutput(self,'OCodePipelineArn',value=databrew_pipeline.code_pipeline_arn)$deactivateJavaYou could also refer tohere.$cdkinit--languagejava
$mvnpackage.
.<properties><project.build.sourceEncoding>UTF-8</project.build.sourceEncoding><cdk.version>2.87.0</cdk.version><constrcut.verion>2.0.196</constrcut.verion><junit.version>5.7.1</junit.version></properties>..<dependencies><!-- AWS Cloud Development Kit --><dependency><groupId>software.amazon.awscdk</groupId><artifactId>aws-cdk-lib</artifactId><version>${cdk.version}</version></dependency><dependency><groupId>io.github.hsiehshujeng</groupId><artifactId>cdk-databrew-cicd</artifactId><version>${constrcut.verion}</version></dependency>...</dependencies>packagecom.myorg;importsoftware.amazon.awscdk.core.CfnOutput;importsoftware.amazon.awscdk.core.CfnOutputProps;importsoftware.amazon.awscdk.core.Construct;importsoftware.amazon.awscdk.core.Stack;importsoftware.amazon.awscdk.core.StackProps;importio.github.hsiehshujeng.cdk.databrew.cicd.DataBrewCodePipeline;importio.github.hsiehshujeng.cdk.databrew.cicd.DataBrewCodePipelineProps;publicclassJavaStackextendsStack{publicJavaStack(finalConstructscope,finalStringid){this(scope,id,null);}publicJavaStack(finalConstructscope,finalStringid,finalStackPropsprops){super(scope,id,props);StringpreproductionAccountId="PREPRODUCTION_ACCOUNT_ID";StringproductionAccountId="PRODUCTION_ACCOUNT_ID";DataBrewCodePipelinedatabrewPipeline=newDataBrewCodePipeline(this,"DataBrewCicdPipeline",DataBrewCodePipelineProps.builder().preproductionIamRoleArn(preproductionAccountId).productionIamRoleArn(productionAccountId)// .bucketName("OPTIONAL")// .branchName("OPTIONAL")// .pipelineName("OPTIONAL").build());newCfnOutput(this,"OPreproductionLambdaArn",CfnOutputProps.builder().value(databrewPipeline.getPreproductionFunctionArn()).build());newCfnOutput(this,"OProductionLambdaArn",CfnOutputProps.builder().value(databrewPipeline.getProductionFunctionArn()).build());newCfnOutput(this,"OCodeCommitRepoArn",CfnOutputProps.builder().value(databrewPipeline.getCodeCommitRepoArn()).build());newCfnOutput(this,"OCodePipelineArn",CfnOutputProps.builder().value(databrewPipeline.getCodePipelineArn()).build());}}C#You could also refer tohere.$cdkinit--languagecsharp
$dotnetaddsrc/CsharppackageDatabrew.Cicd--version2.0.196usingAmazon.CDK;usingScottHsieh.Cdk;namespaceCsharp{publicclassCsharpStack:Stack{internalCsharpStack(Constructscope,stringid,IStackPropsprops=null):base(scope,id,props){varpreproductionAccountId="PREPRODUCTION_ACCOUNT_ID";varproductionAccountId="PRODUCTION_ACCOUNT_ID";vardataBrewPipeline=newDataBrewCodePipeline(this,"DataBrewCicdPipeline",newDataBrewCodePipelineProps{PreproductionIamRoleArn=$"arn:{Aws.PARTITION}:iam::{preproductionAccountId}:role/preproduction-Databrew-Cicd-Role",ProductionIamRoleArn=$"arn:{Aws.PARTITION}:iam::{productionAccountId}:role/preproduction-Databrew-Cicd-Role",// BucketName = "OPTIONAL",// RepoName = "OPTIONAL",// BranchName = "OPTIONAL",// PipelineName = "OPTIONAL"});newCfnOutput(this,"OPreproductionLambdaArn",newCfnOutputProps{Value=dataBrewPipeline.PreproductionFunctionArn});newCfnOutput(this,"OProductionLambdaArn",newCfnOutputProps{Value=dataBrewPipeline.ProductionFunctionArn});newCfnOutput(this,"OCodeCommitRepoArn",newCfnOutputProps{Value=dataBrewPipeline.CodeCommitRepoArn});newCfnOutput(this,"OCodePipelineArn",newCfnOutputProps{Value=dataBrewPipeline.CodeCommitRepoArn});}}}GoYou could also refer tohere.# Initialize a new AWS CDK application in the current directory with the Go programming languagecdkinitapp-lgo# Add this custom CDK construct to your projectgogetgithub.com/HsiehShuJeng/cdk-databrew-cicd-go/cdkdatabrewcicd/v2@v2.0.196# Ensure all dependencies are properly listed in the go.mod file and remove any unused onesgomodtidy# Upgrade all Go modules in your project to their latest minor or patch versionsgoget-u./...packagemainimport("fmt""github.com/aws/aws-cdk-go/awscdk/v2"// "github.com/aws/aws-cdk-go/awscdk/v2/awssqs""github.com/HsiehShuJeng/cdk-databrew-cicd-go/cdkdatabrewcicd/v2""github.com/aws/constructs-go/constructs/v10""github.com/aws/jsii-runtime-go")typeGoLangStackPropsstruct{awscdk.StackProps}funcNewGoLangStack(scopeconstructs.Construct,idstring,props*GoLangStackProps)awscdk.Stack{varspropsawscdk.StackPropsifprops!=nil{sprops=props.StackProps}stack:=awscdk.NewStack(scope,&id,&sprops)preproductionAccountId:="PREPRODUCTION_ACCOUNT_ID"productionAccountId:="PRODUCTION_ACCOUNT_ID"dataBrewPipeline:=cdkdatabrewcicd.NewDataBrewCodePipeline(stack,jsii.String("DataBrewCicdPipeline"),&cdkdatabrewcicd.DataBrewCodePipelineProps{PreproductionIamRoleArn:jsii.String(fmt.Sprintf("arn:%s:iam::%s:role/preproduction-Databrew-Cicd-Role",*awscdk.Aws_PARTITION(),preproductionAccountId)),ProductionIamRoleArn:jsii.String(fmt.Sprintf("arn:%s:iam::%s:role/production-Databrew-Cicd-Role",*awscdk.Aws_PARTITION(),productionAccountId)),// BucketName:   jsii.String("OPTIONAL"),// RepoName:     jsii.String("OPTIONAL"),// BranchName:   jsii.String("OPTIONAL"),// PipelineName: jsii.String("OPTIONAL"),})awscdk.NewCfnOutput(stack,jsii.String("OPreproductionLambdaArn"),&awscdk.CfnOutputProps{Value:dataBrewPipeline.PreproductionFunctionArn()})awscdk.NewCfnOutput(stack,jsii.String("OProductionLambdaArn"),&awscdk.CfnOutputProps{Value:dataBrewPipeline.ProductionFunctionArn()})awscdk.NewCfnOutput(stack,jsii.String("OCodeCommitRepoArn"),&awscdk.CfnOutputProps{Value:dataBrewPipeline.CodeCommitRepoArn()})awscdk.NewCfnOutput(stack,jsii.String("OCodePipelineArn"),&awscdk.CfnOutputProps{Value:dataBrewPipeline.CodePipelineArn()})returnstack}funcmain(){deferjsii.Close()app:=awscdk.NewApp(nil)NewGoLangStack(app,"GoLangStack",&GoLangStackProps{awscdk.StackProps{Env:env(),},})app.Synth(nil)}funcenv()*awscdk.Environment{returnnil}Some Efforts after Stack CreationCodeCommitCreate HTTPS Git credentials for AWS CodeCommit with an IAM user that you're going to use.Run through the steps noted on the README.md of the CodeCommit repository after finishing establishing the stack via CDK. The returned message with success should be looked like the following (assume you have installedgit-remote-codecommit):$gitclonecodecommit://scott.codecommit@DataBrew-Recipes-Repo
Cloninginto'DataBrew-Recipes-Repo'...
remote:Countingobjects:6,done.
Unpackingobjects:100%(6/6),2.03KiB|138.00KiB/s,done.Add a DataBrew recipe into the local repositroy (directory) and commit the change. (either directly on the main branch or merging another branch into the main branch)Glue DataBrewDownload any recipe either generated out by followingGetting started with AWS Glue DataBrewor made by yourself asJSON file.Move the recipe from the download directory to the local directory for the CodeCommit repository.$mv${DOWNLOAD_DIRECTORY}/chess-project-recipe.json${CODECOMMIT_LOCAL_DIRECTORY}/Commit the change to a branch with a name you prefer.$cd${{CODECOMMIT_LOCAL_DIRECTORY}}$gitcheckout-badd-recipemain
$gitadd.
$gitcommit-m"first recipe"$gitpush--set-upstreamoriginadd-recipeMerge the branch into the main branch. Just go to theAWS CodeCommitweb console to do the merge as its process is purely the same as you've already done thousands of times onGithubbut only with different UIs.How Successful Commits Look LikeIn the infrastructure account, the status of the CodePipeline DataBrew pipeline should be similar as the following:In thepre-productionaccount with the same region as where the CICD pipeline is deployed at the infrastructue account, you'll see this.In theproductionaccount with the same region as where the CICD pipeline is deployed at the infrastructue account, you'll see this.