Package: aws-insurancelake-etl
InsuranceLake ETL with CDK PipelineThe InsuranceLake solution is comprised of two codebases:InfrastructureandETL. This codebase is specific to the ETL features, but the documentation that follows applies to the solution as a whole. For documentation with specific details on the Infrastructure, refer to theInsuranceLake Infrastructure with CDK Pipeline README.This solution helps you deploy ETL processes and data storage resources to create InsuranceLake. It uses Amazon S3 buckets for storage,AWS Gluefor data transformation, andAWS CDK Pipelines. The solution is originally based on the AWS blogDeploy data lake ETL jobs using CDK Pipelines.CDK Pipelinesis a construct library module for painless continuous delivery of CDK applications. CDK stands for Cloud Development Kit. It is an open source software development framework to define your cloud application resources using familiar programming languages.Specifically, this solution helps you to:Deploy a 3 Cs (Collect, Cleanse, Consume) architecture InsuranceLakeDeploy ETL jobs needed make common insurance industry data souces available in a data lakeUse pySpark Glue jobs and supporting resoures to perform data transforms in a modular approachBuild and replicate the application in multiple environments quicklyDeploy ETL jobs from a central deployment account to multiple AWS environments such as Dev, Test, and ProdLeverage the benefit of self-mutating feature of CDK Pipelines; specifically, the pipeline itself is infrastructure as code and can be changed as part of the deploymentIncrease the speed of prototyping, testing, and deployment of new ETL jobsContentsQuickstartPython/CDK BasicsDeploy the ApplicationTry out the ETL ProcessQuickstart with CI/CDArchitectureInsuranceLakeETLPipeline UsageBucket LayoutTransformation ModulesCodebaseSource Code StructureSecurityUnit TestingIntegration TestingAdditional resourcesAuthorsLicense SummaryQuickstartIf you'd like to get started quickly transforming some sample raw insurance data and running SQL on the resulting dataset, and without worrying about CI/CD, follow this guide:Python/CDK BasicsSkip steps in this section as needed if you've worked with CDK and Python before.Open the AWS Console and navigate toAWS Cloud9Select the region to create the Cloud9 environment (should be the same region as the stack; us-east-2 by default)Select Create environmentEnter an environment name, for example, InsuranceLakeDemoSelect the t3.small instance size (CDK deployment requires more than 1 GB RAM)Leave the Platform selected as Amazon Linux 2023Adjust the timeout to your preferenceClick CreateOpen the environment you created and wait until it is available for useClone the repositoriesgitclonehttps://github.com/aws-samples/aws-insurancelake-infrastructure.git
gitclonehttps://github.com/aws-samples/aws-insurancelake-etl.gitUse a terminal or command prompt and change the working directory to the location of theinfrastructurecodecdaws-insurancelake-infrastructureCreate a Python virtual environmentpython3-mvenv.venvActivate the virtual environmentsource.venv/bin/activateInstall required Python librariesNOTE: You may see a warning stating that a newer version is available; it is safe to ignore this for the Quickstartpipinstall-rrequirements.txtBootstrap CDK in your AWS accountBy default the solution will deploy resources to theus-east-2regioncdkbootstrapDeploy the ApplicationEnsure you are still in theaws-insurancelake-infrastructuredirectoryDeploy infrastructure resources in the development environment (1 stack)cdkdeployDev-InsuranceLakeInfrastructurePipeline/Dev/InsuranceLakeInfrastructureS3BucketZonesReview and accept IAM credential creation for the S3 bucket stackWait for deployment to finish (approx. 5 mins)Copy the S3 bucket name for the Collect bucket to use laterBucket name will be in the form:dev-insurancelake-<AWS Account ID>-<Region>-collectSwitch the working directory to the location of theetlcodecd../aws-insurancelake-etlDeploy the ETL resources in the development environment (4 stacks)cdkdeployDev-InsuranceLakeEtlPipeline/Dev/InsuranceLakeEtlDynamoDbDev-InsuranceLakeEtlPipeline/Dev/InsuranceLakeEtlGlueDev-InsuranceLakeEtlPipeline/Dev/InsuranceLakeEtlStepFunctionsDev-InsuranceLakeEtlPipeline/Dev/InsuranceLakeEtlAthenaHelperWait for approximately 1 minute for DynamoDB deployment to finishReview and accept IAM credential creation for the Glue jobs stackWait approximately 3 minutes for deployment to finishReview and accept IAM credential creation for the Step Functions stackWait approximately 7 minutes for deployment of Step Functions and Athena Helper stacks to finishTry out the ETL ProcessPopulate the DynamoDB lookup table with sample lookup dataAWS_DEFAULT_REGION=us-east-2resources/load_dynamodb_lookup_table.pySyntheticGeneralDatadev-insurancelake-etl-value-lookupresources/syntheticgeneral_lookup_data.jsonTransfer the sample claim data to the Collect bucketawss3cpresources/syntheticgeneral-claim-data.csvs3://<CollectS3Bucket>/SyntheticGeneralData/ClaimData/Transfer the sample policy data to the Collect bucketawss3cpresources/syntheticgeneral-policy-data.csvs3://<CollectS3Bucket>/SyntheticGeneralData/PolicyData/OpenStep Functionsin the AWS Console and selectdev-insurancelake-etl-state-machineOpen the state machine execution in progress and monitor the status until completedOpenAthenain the AWS ConsoleSelect Launch Query Editor, and change the Workgroup toinsurancelakeRun the following query to view a sample of prepared data in the consume bucket:select*fromsyntheticgeneraldata_consume.policydatalimit100Quickstart with CI/CDIf you've determined the AWS CDK InsuranceLake is a good starting point for your own InsuranceLake, and would like to rapidly iterate through development cycles with one or more teams, we recommend deploying with a CI/CD pipeline. Follow this guide to create your CodePipeline stack and to use it to deploy the InsuranceLake resources:If this is your first time using the application, follow thePython/CDK BasicsstepsUse a terminal or command prompt and change the working directory to the location of the infrastruture codecdaws-cdk-insurancelake-infrastructureInlib/configuration.py, review thelocal_mappingstructure in theget_local_configuration()functionSpecifically, the regions and account IDs should make sense for your environments. These values, in the repository (not locally), will be used by CodeCommit and need to be maintained in the repository.The values for the Test and Production environments can be ommitted at this time, because we will only be deploying the Deployment and Development environments.We want to explicitly specify the account and region for each deployment environment so that the infrastructure VPCs get 3 Availability Zones (if the region has them available).ReferenceDeploy CodeCommit repository stackcdkdeployDeploy-InsuranceLakeInfrastructureMirrorRepositoryWhile this stack is designed for a mirror repository, it can also be used as a main repository for your InsuranceLake code. You can follow links to help setup other repository types here:GithubBitbucketGitlabIf you plan to use CodeCommit as the main repository,install the Git CodeCommit Helper:sudopipinstallgit-remote-codecommitInitialize git, create a develop branch, perform initial commit, and push to remoteWe are using the develop branch because the Dev environment deployment is triggered by commits to the develop branch.Edit the repository URL to correspond to your version control system if you are not using CodeCommitgitinit
gitbranch-Mdevelop
gitadd.
gitcommit-m'Initial commit'gitremoteaddorigincodecommit::us-east-2://aws-cdk-insurancelake-infrastructure
gitpush--set-upstreamorigindevelopDeploy Infrastructure CodePipeline resource in the development environment (1 stack)cdkdeployDevInsuranceLakeInfrastructurePipelineReview and accept IAM credential creation for the CodePipeline stackWait for deployment to finish (approx. 5 mins)Open CodePipeline in the AWS Console and select thedev-insurancelake-infrastructure-pipelinePipelineThe first run of the pipeline starts automatically after the Pipeline stack is deployed.Monitor the status of the pipeline until completedSwitch the working directory to the location of the etl codecd../aws-cdk-insurancelake-etlInlib/configuration.py, review thelocal_mappingstructure in theget_local_configuration()function, ensure this matches the Infrastructure configuration, or differs if specifically needed.Deploy CodeCommit repository stackcdkdeployDeploymentInsuranceLakeEtlMirrorRepositoryInitialize git, create a develop branch, perform initial commit, and push to remoteWe are using the develop branch because the Dev environment deployment is triggered by commits to the develop branch.Edit the repository URL to correspond to your version control system if you are not using CodeCommitgitinit
gitbranch-Mdevelop
gitadd.
gitcommit-m'Initial commit'gitremoteaddorigincodecommit::us-east-2://aws-cdk-insurancelake-etl
gitpush--set-upstreamorigindevelopDeploy ETL CodePipeline resource in the development environment (1 stack)cdkdeployDevInsuranceLakeEtlPipelineReview and accept IAM credential creation for the CodePipeline stackWait for deployment to finish (approx. 5 mins)Open CodePipeline in the AWS Console and select thedev-insurancelake-etl-pipelinePipelineThe first run of the pipeline starts automatically after the Pipeline stack is deployed.Monitor the status of the pipeline until completedArchitectureIn this section we talk about the overall InsuranceLake architecture and the ETL component.InsuranceLake 3 CsAs shown in the figure below, we use Amazon S3 for storage. We use three S3 buckets:Collect bucket to store raw data in its original formatCleanse/Curate bucket to store the data that meets the quality and consistency requirements of the lakeConsume bucket for data that is used by analysts and data consumers of the lake (for example, Amazon Quicksight, Amazon Sagemaker)InsuranceLake is designed to support a number of source systems with different file formats and data partitions. To demonstrate, we have provided a CSV parser and sample data files for a source system with two data tables, which are uploaded to the Collect bucket.We use AWS Lambda and AWS Step Functions for orchestration and scheduling of ETL workloads. We then use AWS Glue with pySpark for ETL and data cataloging, Amazon DynamoDB for transformation persistence, Amazon Athena for interactive queries and analysis. We use various AWS services for logging, monitoring, security, authentication, authorization, notification, build, and deployment.Note:AWS Lake Formationis a service that makes it easy to set up a secure data lake in days.Amazon QuickSightis a scalable, serverless, embeddable, machine learning-powered business intelligence (BI) service built for the cloud. These two services are not used in this solution but can be added.ETLThe figure below represents the ETL resources we provision for Data Lake.A file server uploads files to S3 collect bucket of InsuranceLake; file server is a data producer/source for the data lakeAmazon S3 triggers an event notification to AWS Lambda FunctionAWS Lambda function inserts job information in DynamoDB tableAWS Lambda function starts an execution of AWS Step Functions State machineRuns the first Glue job: initiates data processing from Collect to CleanseGlue job: Spark Glue job will process the data from Collect to Cleanse; source data is assumed to be in CSV format and will be converted to Parquet formatDynamoDB: Glue job tokenization will store original values, and lookup tables reside in databaseAfter creating Parquet data, update the Glue Data Catalog tableRuns the second Glue job: initiates data processing from Cleanse to ConsumeGlue job: Cleanse to Consume fetches data transformation rules from Glue scripts bucket, and runs transformationsStores the result in Parquet format in Consume bucketGlue job updates the Data Catalog tableUpdates DynamoDB table with job statusSends SNS notificationData engineers or analysts analyze data using Amazon AthenaPipeline UsageBucket LayoutIn order to allow transform specifications to be matched with source system data and organized in groups, each of the three ETL stage buckets (Collect, Cleanse, Consume) have similar directory layouts. The first level represents the source system name or the database that will group the underlying tables. The second layer represents the data set or table containing the uploaded data. In the Collect bucket, the source files are stored at the second layer. In the Cleanse bucket, data is converted to parquet files and stored in partitions at the second layer. In the Consume bucket database and table names may change if data is merged.Transformation ModulesFile / FolderDescriptiondatatransform_lookuppySpark logic to perform column value lookup operationsdatatransform_typeconversionpySpark logic to convert date columns, and other data types to standard formatdatatransform_dataprotectionpySpark logic to redact, hash, and tokenize sensitive data columnsdatatransform_regexpySpark logic to perform regex transforms, and Python formatting string operations on datadatatransform_premiumpySpark logic to perform common insurance industry data transformscustom_mappingpySpark logic to rename columns according to a map filedataquality_checkGlue logic to run Data Quality rules according to a rules filedatalineageCustom data lineage tracking class designed to work with InsuranceLake transformsCodebaseSource Code StructureTable below explains how this source code is structured:File / FolderDescriptionapp.pyApplication entry pointcode_commit_stackOptional stack to deploy an empty CodeCommit respository for mirroringpipeline_stackPipeline stack entry pointpipeline_deploy_stagePipeline deploy stage entry pointdynamodb_stackStack creates DynamoDB Tables for Job Auditing and ETL transformation rules.glue_stackStack creates Glue Jobs and supporting resources such as Connections, S3 Buckets - script and temporary - and an IAM execution Rolestep_functions_stackStack creates an ETL State machine which invokes Glue Jobs and supporting Lambdas - state machine trigger and status notification.athena_helper_stackStack creates an Athena workgroup with query results bucket ready for demonstration SQL queries.Collect-to-Cleanse Glue ScriptGlue pySpark job data processing logic for Collect bucket data, which stores results in the Cleanse bucketCleanse-to-Consume Glue ScriptGlue pySpark job data processing logic for Cleanse bucket data, which stores results in the Consume bucketEntity Match Glue ScriptGlue pySpark job data processing logic for Entity Matching, which stores results in the Consume bucketETL Job AuditorLambda script to update DynamoDB in case of glue job success or failureETL TriggerLambda script to trigger step function and initiate DynamoDBETL Transformation Mapping and SpecificationField mapping and transformation specification logic to be used for data processing from Collect to CleanseETL Transformation SQLTransformation SQL logic to be used for data processing from Cleanse to ConsumeETL Data Quality RulesGlue Data Quality rules for quality checks from Cleanse to ConsumetestThis folder contains pytest unit testsresourcesThis folder has architecture, process flow diagrams, and additional documentationSecurityInsuranceLake usesCDK-nagto ensure AWS resource security recommendations are followed. CDK-nag can generate warnings, which may need to be fixed depending on the context, and errors, which will interrupt the stack synthesis and prevent any deployment.To force synthesis of all stacks (including the CodePipeline deployed stacks), use the following command:cdksynth'**'When this operation is complete, you will also have access to the CDK-nag reports in CSV format in the cdk.out directory and assembly directories. By default, the AWS-Solutions Nag pack is used, but any Nag pack can be selected by adjustingInsuranceLake usesbanditto check all helper script, Lambda, and Glue Python code.To scan all application code using bandit, use the following command:bandit-r--ini.banditUnit TestingThe Python CDK unit tests use pytest, which will be installed as part of the solution requirements.
The pySpark Glue Job and Python Lambda function unit tests are still under development.Run tests with the following command (--covwill include a code coverage report):python-mpytest--covNote that without a AWS Glue Docker container, the Glue job tests will be skipped with a message like:test/test_custom_mapping.py::test_custommapping_renames_field SKIPPED (No pySpark environment found)                                    [ 17%]To setup your local environment with a Glue container, retrieve the container image fromthe AWS Glue Dockerhub repository. Ensure you use the right tag for the version of AWS Glue used in the stack (currently v4). Detailed instructions can be found onDeveloping AWS Glue ETL jobs locally using a containerIntegration TestingTODO: Automate manual stepsTransfer the sample claim data to the Collect bucket (Source system: SyntheticData, Table: ClaimData)awss3cpresources/syntheticgeneral-claim-data.csvs3://<CollectS3Bucket>/SyntheticGeneralData/ClaimData/Transfer the sample policy data to the Collect bucket (Source system: SyntheticData, Table: PolicyData)awss3cpresources/syntheticgeneral-policy-data.csvs3://<CollectS3Bucket>/SyntheticGeneralData/PolicyData/Upon successful load of file S3 event notification will trigger the lambdaLambda will insert record into the dynamodb table{environment}-{resource_name_prefix}-etl-job-auditto track job start statusLambda function will trigger the step function. Step function name will be<filename>-<YYYYMMDDHHMMSSxxxxxx>and provided the required metadata inputStep functions state machine will trigger the Glue job for Collect to Cleanse data processing.Glue job will load the data into conformed bucket using the provided metadata and data will be loaded tos3://{environment}-{resource_name_prefix}-{account}-{region}-cleanse/syntheticgeneraldata/claimdata/year=YYYY/month=MM/day=DDin parquet formatGlue job will create/update the catalog table using the tablename passed as parameter based on folder nameclaimdataAfter Collect to Cleanse job completion, Cleanse to Consume Glue job will get triggered in step functionCleanse to Consume Glue glue job will use the transformation logic being provided in Dynamodb as part of prerequisites for data transformationCleanse to Consume Glue job will store the result set in S3 bucket unders3://{environment}-{resource_name_prefix}-{account}-{region}-consume/syntheticgeneraldata/claimdata/year=YYYY/month=MM/day=DDCleanse to Consume Glue job will create/update the catalog tableAfter completion of Glue job, Lambda will get triggered in step function to update the Dynamodb table{environment}-{resource_name_prefix}-etl-job-auditwith latest statusSNS notification will be sent to the subscribed usersTo validate the data, please open Athena service and execute query:select*fromsyntheticgeneraldata_consume.policydatalimit100Additional ResourcesIn this section, we provide some additional resources.Detailed Collect-to-Cleanse transform referenceData quality rules with Glue Data Quality referenceAWS CDK Detailed InstructionsDeveloper Guidefor more information on working with this solutionFull Deployment Guidefor deploying to 3 environments from 3 code branchesGithub / CodePipeline Integration GuideDevelop and test AWS Glue version 3.0 and 4.0 jobs locally using a Docker containerGeneral Insurance dashboardon Quicksight's DemoCentral using Consume-ready-dataAuthorsThe following people are involved in the design, architecture, development, testing, and review of this solution:Cory Visi, Senior Solutions Architect, Amazon Web ServicesRatnadeep Bardhan Roy, Senior Solutions Architect, Amazon Web ServicesIsaiah Grant, Cloud Consultant, 2nd Watch, Inc.Muhammad Zahid Ali, Data Architect, Amazon Web ServicesRavi Itha, Senior Data Architect, Amazon Web ServicesJustiono Putro, Cloud Infrastructure Architect, Amazon Web ServicesMike Apted, Principal Solutions Architect, Amazon Web ServicesNikunj Vaidya, Senior DevOps Specialist, Amazon Web ServicesLicense SummaryThis sample code is made available under the MIT-0 license. See the LICENSE file.Copyright Amazon.com and its affiliates; all rights reserved. This file is Amazon Web Services Content and may not be duplicated or distributed without permission.