Package: cdklabs.cdk-appflow
Amazon AppFlow Construct LibraryNote:this library is currently in technical preview.IntroductionAmazon AppFlow is a service that enables creating managed, bi-directional data transfer integrations between various SaaS applications and AWS services.For more information, see theAmazon AppFlow User Guide.Examplefromaws_cdkimportSecretValuefromaws_cdk.aws_s3importBucketfromaws_cdk.aws_secretsmanagerimportISecretfromcdklabs.cdk_appflowimportISource,IDestination,Filter,FilterCondition,Mapping,OnDemandFlow,S3Destination,SalesforceConnectorProfile,SalesforceSource,Transform,Validation,ValidationAction,ValidationCondition# client_secret: ISecret# access_token: SecretValue# refresh_token: SecretValue# instance_url: strprofile=SalesforceConnectorProfile(self,"MyConnectorProfile",o_auth=SalesforceOAuthSettings(access_token=access_token,flow=SalesforceOAuthFlow(refresh_token_grant=SalesforceOAuthRefreshTokenGrantFlow(refresh_token=refresh_token,client=client_secret))),instance_url=instance_url,is_sandbox=False)source=SalesforceSource(profile=profile,object="Account")bucket=Bucket(self,"DestinationBucket")destination=S3Destination(location=S3Location(bucket=bucket))OnDemandFlow(self,"SfAccountToS3",source=source,destination=destination,mappings=[Mapping.map_all()],transforms=[Transform.mask(Field(name="Name"),"*")],validations=[Validation.when(ValidationCondition.is_null("Name"),ValidationAction.ignore_record())],filters=[Filter.when(FilterCondition.timestamp_less_than_equals(Field(name="LastModifiedDate",data_type="datetime"),Date(Date.parse("2022-02-02"))))])ConceptsAmazon AppFlow introduces several concepts that abstract away the technicalities of setting up and managing data integrations.AnApplicationis any SaaS data integration component that can be either asourceor adestinationfor Amazon AppFlow. A source is an application from which Amazon AppFlow will retrieve data, whereas a destination is an application to which Amazon AppFlow will send data.AFlowis Amazon AppFlow's integration between a source and a destination.AConnectorProfileis Amazon AppFlow's abstraction over authentication/authorization with a particular SaaS application. The per-SaaS application permissions given to a particularConnectorProfilewill determine whether the connector profile can support the application as a source or as a destination (see whether a particular application is supported as either a source or a destination inthe documentation).Types of FlowsThe library introduces three, separate types of flows:OnDemandFlow- a construct representing a flow that can be triggered programmatically with the use of aStartFlow API call.OnEventFlow- a construct representing a flow that is triggered by a SaaS application event published to AppFlow. At the time of writing only a Salesforce source is able to publish events that can be consumed by AppFlow flows.OnScheduleFlow- a construct representing a flow that is triggered on aScheduleTasksTasks are steps that can be taken upon fields. Tasks compose higher level objects that in this library are namedOperations. There are four operations identified:Transforms - 1-1 transforms on source fields, like truncation or maskingMappings - 1-1 or many-to-1 operations from source fields to a destination fieldFilters - operations that limit the source data on a particular conditionsValidations - operations that work on a per-record level and can have either a record-level consequence (i.e. dropping the record) or a global one (terminating the flow).Each flow exposes dedicated properties to each of the operation types that one can use like in the example below:fromcdklabs.cdk_appflowimportFilter,FilterCondition,IDestination,ISource,Mapping,OnDemandFlow,S3Destination,SalesforceConnectorProfile,SalesforceSource,Transform,Validation,ValidationAction,ValidationCondition# stack: Stack# source: ISource# destination: IDestinationflow=OnDemandFlow(stack,"OnDemandFlow",source=source,destination=destination,transforms=[Transform.mask(Field(name="Name"),"*")],mappings=[Mapping.map(Field(name="Name",data_type="String"),name="Name",data_type="string")],filters=[Filter.when(FilterCondition.timestamp_less_than_equals(Field(name="LastModifiedDate",data_type="datetime"),Date(Date.parse("2022-02-02"))))],validations=[Validation.when(ValidationCondition.is_null("Name"),ValidationAction.ignore_record())])MonitoringMetrcisEach flow allows to access metrics through the methods:metricFlowExecutionsStartedmetricFlowExecutionsFailedmetricFlowExecutionsSucceededmetricFlowExecutionTimemetricFlowExecutionRecordsProcessedFor detailed information about AppFlow metrics refer tothe documentation.It can be consume by CloudWatch alert using as in the example below:fromcdklabs.cdk_appflowimportIFlow# flow: IFlow# stack: Stackmetric=flow.metric_flow_executions_started()metric.create_alarm(stack,"FlowExecutionsStartedAlarm",threshold=1000,evaluation_periods=2)EventBridge notificationsEach flow publishes events to the default EventBridge bus:onRunStartedonRunCompletedonDeactivated(only for theOnEventFlowand theOnScheduleFlow)onStatus(only for theOnEventFlow)This way one can consume the notifications as in the example below:fromaws_cdk.aws_snsimportITopicfromaws_cdk.aws_events_targetsimportSnsTopicfromcdklabs.cdk_appflowimportIFlow# flow: IFlow# my_topic: ITopicflow.on_run_completed("OnRunCompleted",target=SnsTopic(my_topic))Notable distinctions from CloudFormation specificationOnScheduleFlowandincrementalPullConfigIn CloudFormation the definition of theincrementalPullConfig(which effectively gives a name of the field used for tracking the last pulled timestamp) is on theSourceFlowConfigproperty. In the library this has been moved to theOnScheduleFlowconstructor properties.S3Destinationand Glue CatalogAlthough in CloudFormation the Glue Catalog configuration is settable on the flow level - it works only when the destination is S3. That is why the library shifts the Glue Catalog properties definition to theS3Destination, which in turn requires using Lazy for populatingmetadataCatalogConfigin the flow.Security considerationsIt isrecommendedto followdata protection mechanisms for Amazon AppFlow.Confidential informationAmazon AppFlow application integration is done usingConnectionProfiles. AConnectionProfilerequires providing sensitive information in the form of e.g. access and refresh tokens. It isrecommendedthat such information is stored securely and passed to AWS CDK securely. All sensitive fields are effectivelyIResolvableand this means they can be resolved at deploy time. With that one should follow thebest practices for credentials with CloudFormation. In this library, the sensitive fields are typed asSecretValueto emphasize these should not be plain strings.An example of using a predefined AWS Secrets Manager secret for storing sensitive information can be found below:fromaws_cdk.aws_secretsmanagerimportSecretfromcdklabs.cdk_appflowimportGoogleAnalytics4ConnectorProfile# stack: Stacksecret=Secret.from_secret_name_v2(stack,"GA4Secret","appflow/ga4")profile=GoogleAnalytics4ConnectorProfile(stack,"GA4Connector",o_auth=GoogleAnalytics4OAuthSettings(flow=GoogleAnalytics4OAuthFlow(refresh_token_grant=GoogleAnalytics4RefreshTokenGrantFlow(refresh_token=secret.secret_value_from_json("refreshToken"),client_id=secret.secret_value_from_json("clientId"),client_secret=secret.secret_value_from_json("clientSecret")))))An approach to managing permissionsThis library relies on an internalAppFlowPermissionsManagerclass to automatically infer and apply appropriate resource policy statements to the S3 Bucket, KMS Key, and Secrets Manager Secret resources.AppFlowPermissionsManagerplaces the statements exactly once for theappflow.amazonaws.comprincipal no matter how many times a resource is reused in the code.Confused Deputy ProblemAmazon AppFlow is an account-bound and a regional service. With this it is invurlnerable to the confused deputy problem (see, e.g.here). However,AppFlowPermissionsManagerstill introduces theaws:SourceAccountcondtition to the resource policies as abest practice.