Package: aws-cdk.aws-stepfunctions-tasks
Tasks for AWS Step Functions---AWS CDK v1 has reached End-of-Support on 2023-06-01.
This package is no longer being updated, and users should migrate to AWS CDK v2.For more information on how to migrate, see theMigrating to AWS CDK v2guide.AWS Step Functionsis a web service that enables you to coordinate the
components of distributed applications and microservices using visual workflows.
You build applications from individual components that each perform a discrete
function, or task, allowing you to scale and change applications quickly.ATaskstate represents a single unit of work performed by a state machine.
All work in your state machine is performed by tasks.This module is part of theAWS Cloud Development Kitproject.Table Of ContentsTasks for AWS Step FunctionsTable Of ContentsTaskPathsInputPathOutputPathResultPathTask parameters from the state JSONEvaluate ExpressionAPI GatewayCall REST API EndpointCall HTTP API EndpointAWS SDKAthenaStartQueryExecutionGetQueryExecutionGetQueryResultsStopQueryExecutionBatchSubmitJobCodeBuildStartBuildDynamoDBGetItemPutItemDeleteItemUpdateItemECSRunTaskEC2FargateEMRCreate ClusterTermination ProtectionTerminate ClusterAdd StepCancel StepModify Instance FleetModify Instance GroupEMR on EKSCreate Virtual ClusterDelete Virtual ClusterStart Job RunEKSCallEventBridgePut EventsGlueGlue DataBrewLambdaSageMakerCreate Training JobCreate Transform JobCreate EndpointCreate Endpoint ConfigCreate ModelUpdate EndpointSNSStep FunctionsStart ExecutionInvoke ActivitySQSTaskA Task state represents a single unit of work performed by a state machine. In the
CDK, the exact work to be done is determined by a class that implementsIStepFunctionsTask.AWS Step Functionsintegrateswith some AWS services so that you can call API
actions, and coordinate executions directly from the Amazon States Language in
Step Functions. You can directly call and pass parameters to the APIs of those
services.PathsIn the Amazon States Language, apathis a string beginning with$that you
can use to identify components within JSON text.Learn more about input and output processing in Step FunctionshereInputPathBothInputPathandParametersfields provide a way to manipulate JSON as it
moves through your workflow. AWS Step Functions applies theInputPathfield first,
and then theParametersfield. You can first filter your raw input to a selection
you want using InputPath, and then apply Parameters to manipulate that input
further, or add new values. If you don't specify anInputPath, a default value
of$will be used.The following example provides the field namedinputas the input to theTaskstate that runs a Lambda function.# fn: lambda.Functionsubmit_job=tasks.LambdaInvoke(self,"Invoke Handler",lambda_function=fn,input_path="$.input")OutputPathTasks also allow you to select a portion of the state output to pass to the next
state. This enables you to filter out unwanted information, and pass only the
portion of the JSON that you care about. If you don't specify anOutputPath,
a default value of$will be used. This passes the entire JSON node to the next
state.Theresponsefrom a Lambda function includes the response from the function
as well as other metadata.The following example assigns the output from the Task to a field namedresult# fn: lambda.Functionsubmit_job=tasks.LambdaInvoke(self,"Invoke Handler",lambda_function=fn,output_path="$.Payload.result")ResultSelectorYou can useResultSelectorto manipulate the raw result of a Task, Map or Parallel state before it is
passed toResultPath. For service integrations, the raw
result contains metadata in addition to the response payload. You can use
ResultSelector to construct a JSON payload that becomes the effective result
using static values or references to the raw result or context object.The following example extracts the output payload of a Lambda function Task and combines
it with some static values and the state name from the context object.# fn: lambda.Functiontasks.LambdaInvoke(self,"Invoke Handler",lambda_function=fn,result_selector={"lambda_output":sfn.JsonPath.string_at("$.Payload"),"invoke_request_id":sfn.JsonPath.string_at("$.SdkResponseMetadata.RequestId"),"static_value":{"foo":"bar"},"state_name":sfn.JsonPath.string_at("$.State.Name")})ResultPathThe output of a state can be a copy of its input, the result it produces (for
example, output from a Task state’s Lambda function), or a combination of its
input and result. UseResultPathto control which combination of these is
passed to the state output. If you don't specify anResultPath, a default
value of$will be used.The following example adds the item from calling DynamoDB'sgetItemAPI to the state
input and passes it to the next state.# my_table: dynamodb.Tabletasks.DynamoPutItem(self,"PutItem",item={"MessageId":tasks.DynamoAttributeValue.from_string("message-id")},table=my_table,result_path="$.Item")⚠️ TheOutputPathis computed after applyingResultPath. All service integrations
return metadata as part of their response. When usingResultPath, it's not possible to
merge a subset of the task output to the input.Task parameters from the state JSONMost tasks take parameters. Parameter values can either be static, supplied directly
in the workflow definition (by specifying their values), or a value available at runtime
in the state machine's execution (either as its input or an output of a prior state).
Parameter values available at runtime can be specified via theJsonPathclass,
using methods such asJsonPath.stringAt().The following example provides the field namedinputas the input to the Lambda function
and invokes it asynchronously.# fn: lambda.Functionsubmit_job=tasks.LambdaInvoke(self,"Invoke Handler",lambda_function=fn,payload=sfn.TaskInput.from_json_path_at("$.input"),invocation_type=tasks.LambdaInvocationType.EVENT)You can also useintrinsic functionsavailable onJsonPath, for exampleJsonPath.format().
Here is an example of starting an Athena query that is dynamically created using the task input:start_query_execution_job=tasks.AthenaStartQueryExecution(self,"Athena Start Query",query_string=sfn.JsonPath.format("select contacts where year={};",sfn.JsonPath.string_at("$.year")),query_execution_context=tasks.QueryExecutionContext(database_name="interactions"),result_configuration=tasks.ResultConfiguration(encryption_configuration=tasks.EncryptionConfiguration(encryption_option=tasks.EncryptionOption.S3_MANAGED),output_location=s3.Location(bucket_name="mybucket",object_key="myprefix")),integration_pattern=sfn.IntegrationPattern.RUN_JOB)Each service integration has its own set of parameters that can be supplied.Evaluate ExpressionUse theEvaluateExpressionto perform simple operations referencing state paths. Theexpressionreferenced in the task will be evaluated in a Lambda function
(eval()). This allows you to not have to write Lambda code for simple operations.Example: convert a wait time from milliseconds to seconds, concat this in a message and wait:convert_to_seconds=tasks.EvaluateExpression(self,"Convert to seconds",expression="$.waitMilliseconds / 1000",result_path="$.waitSeconds")create_message=tasks.EvaluateExpression(self,"Create message",# Note: this is a string inside a string.expression="`Now waiting ${$.waitSeconds} seconds...`",runtime=lambda_.Runtime.NODEJS_14_X,result_path="$.message")publish_message=tasks.SnsPublish(self,"Publish message",topic=sns.Topic(self,"cool-topic"),message=sfn.TaskInput.from_json_path_at("$.message"),result_path="$.sns")wait=sfn.Wait(self,"Wait",time=sfn.WaitTime.seconds_path("$.waitSeconds"))sfn.StateMachine(self,"StateMachine",definition=convert_to_seconds.next(create_message).next(publish_message).next(wait))TheEvaluateExpressionsupports aruntimeprop to specify the Lambda
runtime to use to evaluate the expression. Currently, only runtimes
of the Node.js family are supported.API GatewayStep Functions supportsAPI Gatewaythrough the service integration pattern.HTTP APIs are designed for low-latency, cost-effective integrations with AWS services, including AWS Lambda, and HTTP endpoints.
HTTP APIs support OIDC and OAuth 2.0 authorization, and come with built-in support for CORS and automatic deployments.
Previous-generation REST APIs currently offer more features. More details can be foundhere.Call REST API EndpointTheCallApiGatewayRestApiEndpointcalls the REST API endpoint.importaws_cdk.aws_apigatewayasapigatewayrest_api=apigateway.RestApi(self,"MyRestApi")invoke_task=tasks.CallApiGatewayRestApiEndpoint(self,"Call REST API",api=rest_api,stage_name="prod",method=tasks.HttpMethod.GET)Be aware that the header values must be arrays. When passing the Task Token
in the headers fieldWAIT_FOR_TASK_TOKENintegration, useJsonPath.array()to wrap the token in an array:importaws_cdk.aws_apigatewayasapigateway# api: apigateway.RestApitasks.CallApiGatewayRestApiEndpoint(self,"Endpoint",api=api,stage_name="Stage",method=tasks.HttpMethod.PUT,integration_pattern=sfn.IntegrationPattern.WAIT_FOR_TASK_TOKEN,headers=sfn.TaskInput.from_object({"TaskToken":sfn.JsonPath.array(sfn.JsonPath.task_token)}))Call HTTP API EndpointTheCallApiGatewayHttpApiEndpointcalls the HTTP API endpoint.importaws_cdk.aws_apigatewayv2asapigatewayv2http_api=apigatewayv2.HttpApi(self,"MyHttpApi")invoke_task=tasks.CallApiGatewayHttpApiEndpoint(self,"Call HTTP API",api_id=http_api.api_id,api_stack=Stack.of(http_api),method=tasks.HttpMethod.GET)AWS SDKStep Functions supports callingAWS service's API actionsthrough the service integration pattern.You can use Step Functions' AWS SDK integrations to call any of the over two hundred AWS services
directly from your state machine, giving you access to over nine thousand API actions.# my_bucket: s3.Bucketget_object=tasks.CallAwsService(self,"GetObject",service="s3",action="getObject",parameters={"Bucket":my_bucket.bucket_name,"Key":sfn.JsonPath.string_at("$.key")},iam_resources=[my_bucket.arn_for_objects("*")])Use camelCase for actions and PascalCase for parameter names.The task automatically adds an IAM statement to the state machine role's policy based on the
service and action called. The resources for this statement must be specified iniamResources.Use theiamActionprop to manually specify the IAM action name in the case where the IAM
action name does not match with the API service/action name:list_buckets=tasks.CallAwsService(self,"ListBuckets",service="s3",action="listBuckets",iam_resources=["*"],iam_action="s3:ListAllMyBuckets")AthenaStep Functions supportsAthenathrough the service integration pattern.StartQueryExecutionTheStartQueryExecutionAPI runs the SQL query statement.start_query_execution_job=tasks.AthenaStartQueryExecution(self,"Start Athena Query",query_string=sfn.JsonPath.string_at("$.queryString"),query_execution_context=tasks.QueryExecutionContext(database_name="mydatabase"),result_configuration=tasks.ResultConfiguration(encryption_configuration=tasks.EncryptionConfiguration(encryption_option=tasks.EncryptionOption.S3_MANAGED),output_location=s3.Location(bucket_name="query-results-bucket",object_key="folder")))GetQueryExecutionTheGetQueryExecutionAPI gets information about a single execution of a query.get_query_execution_job=tasks.AthenaGetQueryExecution(self,"Get Query Execution",query_execution_id=sfn.JsonPath.string_at("$.QueryExecutionId"))GetQueryResultsTheGetQueryResultsAPI that streams the results of a single query execution specified by QueryExecutionId from S3.get_query_results_job=tasks.AthenaGetQueryResults(self,"Get Query Results",query_execution_id=sfn.JsonPath.string_at("$.QueryExecutionId"))StopQueryExecutionTheStopQueryExecutionAPI that stops a query execution.stop_query_execution_job=tasks.AthenaStopQueryExecution(self,"Stop Query Execution",query_execution_id=sfn.JsonPath.string_at("$.QueryExecutionId"))BatchStep Functions supportsBatchthrough the service integration pattern.SubmitJobTheSubmitJobAPI submits an AWS Batch job from a job definition.importaws_cdk.aws_batchasbatch# batch_job_definition: batch.JobDefinition# batch_queue: batch.JobQueuetask=tasks.BatchSubmitJob(self,"Submit Job",job_definition_arn=batch_job_definition.job_definition_arn,job_name="MyJob",job_queue_arn=batch_queue.job_queue_arn)CodeBuildStep Functions supportsCodeBuildthrough the service integration pattern.StartBuildStartBuildstarts a CodeBuild Project by Project Name.importaws_cdk.aws_codebuildascodebuildcodebuild_project=codebuild.Project(self,"Project",project_name="MyTestProject",build_spec=codebuild.BuildSpec.from_object({"version":"0.2","phases":{"build":{"commands":["echo\"Hello, CodeBuild!\""]}}}))task=tasks.CodeBuildStartBuild(self,"Task",project=codebuild_project,integration_pattern=sfn.IntegrationPattern.RUN_JOB,environment_variables_override={"ZONE":codebuild.BuildEnvironmentVariable(type=codebuild.BuildEnvironmentVariableType.PLAINTEXT,value=sfn.JsonPath.string_at("$.envVariables.zone"))})DynamoDBYou can call DynamoDB APIs from aTaskstate.
Read more about calling DynamoDB APIshereGetItemTheGetItemoperation returns a set of attributes for the item with the given primary key.# my_table: dynamodb.Tabletasks.DynamoGetItem(self,"Get Item",key={"message_id":tasks.DynamoAttributeValue.from_string("message-007")},table=my_table)PutItemThePutItemoperation creates a new item, or replaces an old item with a new item.# my_table: dynamodb.Tabletasks.DynamoPutItem(self,"PutItem",item={"MessageId":tasks.DynamoAttributeValue.from_string("message-007"),"Text":tasks.DynamoAttributeValue.from_string(sfn.JsonPath.string_at("$.bar")),"TotalCount":tasks.DynamoAttributeValue.from_number(10)},table=my_table)DeleteItemTheDeleteItemoperation deletes a single item in a table by primary key.# my_table: dynamodb.Tabletasks.DynamoDeleteItem(self,"DeleteItem",key={"MessageId":tasks.DynamoAttributeValue.from_string("message-007")},table=my_table,result_path=sfn.JsonPath.DISCARD)UpdateItemTheUpdateItemoperation edits an existing item's attributes, or adds a new item
to the table if it does not already exist.# my_table: dynamodb.Tabletasks.DynamoUpdateItem(self,"UpdateItem",key={"MessageId":tasks.DynamoAttributeValue.from_string("message-007")},table=my_table,expression_attribute_values={":val":tasks.DynamoAttributeValue.number_from_string(sfn.JsonPath.string_at("$.Item.TotalCount.N")),":rand":tasks.DynamoAttributeValue.from_number(20)},update_expression="SET TotalCount = :val + :rand")ECSStep Functions supportsECS/Fargatethrough the service integration pattern.RunTaskRunTaskstarts a new task using the specified task definition.EC2The EC2 launch type allows you to run your containerized applications on a cluster
of Amazon EC2 instances that you manage.When a task that uses the EC2 launch type is launched, Amazon ECS must determine where
to place the task based on the requirements specified in the task definition, such as
CPU and memory. Similarly, when you scale down the task count, Amazon ECS must determine
which tasks to terminate. You can apply task placement strategies and constraints to
customize how Amazon ECS places and terminates tasks. Learn more abouttask placementThe latest ACTIVE revision of the passed task definition is used for running the task.The following example runs a job from a task definition on EC2vpc=ec2.Vpc.from_lookup(self,"Vpc",is_default=True)cluster=ecs.Cluster(self,"Ec2Cluster",vpc=vpc)cluster.add_capacity("DefaultAutoScalingGroup",instance_type=ec2.InstanceType("t2.micro"),vpc_subnets=ec2.SubnetSelection(subnet_type=ec2.SubnetType.PUBLIC))task_definition=ecs.TaskDefinition(self,"TD",compatibility=ecs.Compatibility.EC2)task_definition.add_container("TheContainer",image=ecs.ContainerImage.from_registry("foo/bar"),memory_limit_mi_b=256)run_task=tasks.EcsRunTask(self,"Run",integration_pattern=sfn.IntegrationPattern.RUN_JOB,cluster=cluster,task_definition=task_definition,launch_target=tasks.EcsEc2LaunchTarget(placement_strategies=[ecs.PlacementStrategy.spread_across_instances(),ecs.PlacementStrategy.packed_by_cpu(),ecs.PlacementStrategy.randomly()],placement_constraints=[ecs.PlacementConstraint.member_of("blieptuut")]))FargateAWS Fargate is a serverless compute engine for containers that works with Amazon
Elastic Container Service (ECS). Fargate makes it easy for you to focus on building
your applications. Fargate removes the need to provision and manage servers, lets you
specify and pay for resources per application, and improves security through application
isolation by design. Learn more aboutFargateThe Fargate launch type allows you to run your containerized applications without the need
to provision and manage the backend infrastructure. Just register your task definition and
Fargate launches the container for you. The latest ACTIVE revision of the passed
task definition is used for running the task. Learn more aboutFargate VersioningThe following example runs a job from a task definition on Fargatevpc=ec2.Vpc.from_lookup(self,"Vpc",is_default=True)cluster=ecs.Cluster(self,"FargateCluster",vpc=vpc)task_definition=ecs.TaskDefinition(self,"TD",memory_mi_b="512",cpu="256",compatibility=ecs.Compatibility.FARGATE)container_definition=task_definition.add_container("TheContainer",image=ecs.ContainerImage.from_registry("foo/bar"),memory_limit_mi_b=256)run_task=tasks.EcsRunTask(self,"RunFargate",integration_pattern=sfn.IntegrationPattern.RUN_JOB,cluster=cluster,task_definition=task_definition,assign_public_ip=True,container_overrides=[tasks.ContainerOverride(container_definition=container_definition,environment=[tasks.TaskEnvironmentVariable(name="SOME_KEY",value=sfn.JsonPath.string_at("$.SomeKey"))])],launch_target=tasks.EcsFargateLaunchTarget())EMRStep Functions supports Amazon EMR through the service integration pattern.
The service integration APIs correspond to Amazon EMR APIs but differ in the
parameters that are used.Read moreabout the differences when using these service integrations.Create ClusterCreates and starts running a cluster (job flow).
Corresponds to therunJobFlowAPI in EMR.cluster_role=iam.Role(self,"ClusterRole",assumed_by=iam.ServicePrincipal("ec2.amazonaws.com"))service_role=iam.Role(self,"ServiceRole",assumed_by=iam.ServicePrincipal("elasticmapreduce.amazonaws.com"))auto_scaling_role=iam.Role(self,"AutoScalingRole",assumed_by=iam.ServicePrincipal("elasticmapreduce.amazonaws.com"))auto_scaling_role.assume_role_policy.add_statements(iam.PolicyStatement(effect=iam.Effect.ALLOW,principals=[iam.ServicePrincipal("application-autoscaling.amazonaws.com")],actions=["sts:AssumeRole"]))tasks.EmrCreateCluster(self,"Create Cluster",instances=tasks.EmrCreateCluster.InstancesConfigProperty(),cluster_role=cluster_role,name=sfn.TaskInput.from_json_path_at("$.ClusterName").value,service_role=service_role,auto_scaling_role=auto_scaling_role)If you want to run multiple steps inparallel,
you can specify thestepConcurrencyLevelproperty. The concurrency range is between 1
and 256 inclusive, where the default concurrency of 1 means no step concurrency is allowed.stepConcurrencyLevelrequires the EMR release label to be 5.28.0 or above.tasks.EmrCreateCluster(self,"Create Cluster",instances=tasks.EmrCreateCluster.InstancesConfigProperty(),name=sfn.TaskInput.from_json_path_at("$.ClusterName").value,step_concurrency_level=10)Termination ProtectionLocks a cluster (job flow) so the EC2 instances in the cluster cannot be
terminated by user intervention, an API call, or a job-flow error.Corresponds to thesetTerminationProtectionAPI in EMR.tasks.EmrSetClusterTerminationProtection(self,"Task",cluster_id="ClusterId",termination_protected=False)Terminate ClusterShuts down a cluster (job flow).
Corresponds to theterminateJobFlowsAPI in EMR.tasks.EmrTerminateCluster(self,"Task",cluster_id="ClusterId")Add StepAdds a new step to a running cluster.
Corresponds to theaddJobFlowStepsAPI in EMR.tasks.EmrAddStep(self,"Task",cluster_id="ClusterId",name="StepName",jar="Jar",action_on_failure=tasks.ActionOnFailure.CONTINUE)Cancel StepCancels a pending step in a running cluster.
Corresponds to thecancelStepsAPI in EMR.tasks.EmrCancelStep(self,"Task",cluster_id="ClusterId",step_id="StepId")Modify Instance FleetModifies the target On-Demand and target Spot capacities for the instance
fleet with the specified InstanceFleetName.Corresponds to themodifyInstanceFleetAPI in EMR.tasks.EmrModifyInstanceFleetByName(self,"Task",cluster_id="ClusterId",instance_fleet_name="InstanceFleetName",target_on_demand_capacity=2,target_spot_capacity=0)Modify Instance GroupModifies the number of nodes and configuration settings of an instance group.Corresponds to themodifyInstanceGroupsAPI in EMR.tasks.EmrModifyInstanceGroupByName(self,"Task",cluster_id="ClusterId",instance_group_name=sfn.JsonPath.string_at("$.InstanceGroupName"),instance_group=tasks.EmrModifyInstanceGroupByName.InstanceGroupModifyConfigProperty(instance_count=1))EMR on EKSStep Functions supports Amazon EMR on EKS through the service integration pattern.
The service integration APIs correspond to Amazon EMR on EKS APIs, but differ in the parameters that are used.Read moreabout the differences when using these service integrations.Setting upthe EKS cluster is required.Create Virtual ClusterTheCreateVirtualClusterAPI creates a single virtual cluster that's mapped to a single Kubernetes namespace.The EKS cluster containing the Kubernetes namespace where the virtual cluster will be mapped can be passed in from the task input.tasks.EmrContainersCreateVirtualCluster(self,"Create a Virtual Cluster",eks_cluster=tasks.EksClusterInput.from_task_input(sfn.TaskInput.from_text("clusterId")))The EKS cluster can also be passed in directly.importaws_cdk.aws_eksaseks# eks_cluster: eks.Clustertasks.EmrContainersCreateVirtualCluster(self,"Create a Virtual Cluster",eks_cluster=tasks.EksClusterInput.from_cluster(eks_cluster))By default, the Kubernetes namespace that a virtual cluster maps to is "default", but a specific namespace within an EKS cluster can be selected.tasks.EmrContainersCreateVirtualCluster(self,"Create a Virtual Cluster",eks_cluster=tasks.EksClusterInput.from_task_input(sfn.TaskInput.from_text("clusterId")),eks_namespace="specified-namespace")Delete Virtual ClusterTheDeleteVirtualClusterAPI deletes a virtual cluster.tasks.EmrContainersDeleteVirtualCluster(self,"Delete a Virtual Cluster",virtual_cluster_id=sfn.TaskInput.from_json_path_at("$.virtualCluster"))Start Job RunTheStartJobRunAPI starts a job run. A job is a unit of work that you submit to Amazon EMR on EKS for execution. The work performed by the job can be defined by a Spark jar, PySpark script, or SparkSQL query. A job run is an execution of the job on the virtual cluster.Required setup:If not done already, follow thestepsto setup EMR on EKS andcreate an EKS Cluster.EnableCluster accessEnableIAM Role accessThe following actions must be performed if the virtual cluster ID is supplied from the task input. Otherwise, if it is supplied statically in the state machine definition, these actions will be done automatically.Create anIAM roleUpdate theRole Trust Policyof the Job Execution Role.The job can be configured with spark submit parameters:tasks.EmrContainersStartJobRun(self,"EMR Containers Start Job Run",virtual_cluster=tasks.VirtualClusterInput.from_virtual_cluster_id("de92jdei2910fwedz"),release_label=tasks.ReleaseLabel.EMR_6_2_0,job_driver=tasks.JobDriver(spark_submit_job_driver=tasks.SparkSubmitJobDriver(entry_point=sfn.TaskInput.from_text("local:///usr/lib/spark/examples/src/main/python/pi.py"),spark_submit_parameters="--conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1")))Configuring the job can also be done via application configuration:tasks.EmrContainersStartJobRun(self,"EMR Containers Start Job Run",virtual_cluster=tasks.VirtualClusterInput.from_virtual_cluster_id("de92jdei2910fwedz"),release_label=tasks.ReleaseLabel.EMR_6_2_0,job_name="EMR-Containers-Job",job_driver=tasks.JobDriver(spark_submit_job_driver=tasks.SparkSubmitJobDriver(entry_point=sfn.TaskInput.from_text("local:///usr/lib/spark/examples/src/main/python/pi.py"))),application_config=[tasks.ApplicationConfiguration(classification=tasks.Classification.SPARK_DEFAULTS,properties={"spark.executor.instances":"1","spark.executor.memory":"512M"})])Job monitoring can be enabled ifmonitoring.loggingis set true. This automatically generates an S3 bucket and CloudWatch logs.tasks.EmrContainersStartJobRun(self,"EMR Containers Start Job Run",virtual_cluster=tasks.VirtualClusterInput.from_virtual_cluster_id("de92jdei2910fwedz"),release_label=tasks.ReleaseLabel.EMR_6_2_0,job_driver=tasks.JobDriver(spark_submit_job_driver=tasks.SparkSubmitJobDriver(entry_point=sfn.TaskInput.from_text("local:///usr/lib/spark/examples/src/main/python/pi.py"),spark_submit_parameters="--conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1")),monitoring=tasks.Monitoring(logging=True))Otherwise, providing monitoring for jobs with existing log groups and log buckets is also available.importaws_cdk.aws_logsaslogslog_group=logs.LogGroup(self,"Log Group")log_bucket=s3.Bucket(self,"S3 Bucket")tasks.EmrContainersStartJobRun(self,"EMR Containers Start Job Run",virtual_cluster=tasks.VirtualClusterInput.from_virtual_cluster_id("de92jdei2910fwedz"),release_label=tasks.ReleaseLabel.EMR_6_2_0,job_driver=tasks.JobDriver(spark_submit_job_driver=tasks.SparkSubmitJobDriver(entry_point=sfn.TaskInput.from_text("local:///usr/lib/spark/examples/src/main/python/pi.py"),spark_submit_parameters="--conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1")),monitoring=tasks.Monitoring(log_group=log_group,log_bucket=log_bucket))Users can provide their own existing Job Execution Role.tasks.EmrContainersStartJobRun(self,"EMR Containers Start Job Run",virtual_cluster=tasks.VirtualClusterInput.from_task_input(sfn.TaskInput.from_json_path_at("$.VirtualClusterId")),release_label=tasks.ReleaseLabel.EMR_6_2_0,job_name="EMR-Containers-Job",execution_role=iam.Role.from_role_arn(self,"Job-Execution-Role","arn:aws:iam::xxxxxxxxxxxx:role/JobExecutionRole"),job_driver=tasks.JobDriver(spark_submit_job_driver=tasks.SparkSubmitJobDriver(entry_point=sfn.TaskInput.from_text("local:///usr/lib/spark/examples/src/main/python/pi.py"),spark_submit_parameters="--conf spark.executor.instances=2 --conf spark.executor.memory=2G --conf spark.executor.cores=2 --conf spark.driver.cores=1")))EKSStep Functions supports Amazon EKS through the service integration pattern.
The service integration APIs correspond to Amazon EKS APIs.Read moreabout the differences when using these service integrations.CallRead and write Kubernetes resource objects via a Kubernetes API endpoint.
Corresponds to thecallAPI in Step Functions Connector.The following code snippet includes a Task state that uses eks:call to list the pods.importaws_cdk.aws_eksaseksmy_eks_cluster=eks.Cluster(self,"my sample cluster",version=eks.KubernetesVersion.V1_18,cluster_name="myEksCluster")tasks.EksCall(self,"Call a EKS Endpoint",cluster=my_eks_cluster,http_method=tasks.HttpMethods.GET,http_path="/api/v1/namespaces/default/pods")EventBridgeStep Functions supports Amazon EventBridge through the service integration pattern.
The service integration APIs correspond to Amazon EventBridge APIs.Read moreabout the differences when using these service integrations.Put EventsSend events to an EventBridge bus.
Corresponds to theput-eventsAPI in Step Functions Connector.The following code snippet includes a Task state that uses events:putevents to send an event to the default bus.importaws_cdk.aws_eventsaseventsmy_event_bus=events.EventBus(self,"EventBus",event_bus_name="MyEventBus1")tasks.EventBridgePutEvents(self,"Send an event to EventBridge",entries=[tasks.EventBridgePutEventsEntry(detail=sfn.TaskInput.from_object({"Message":"Hello from Step Functions!"}),event_bus=my_event_bus,detail_type="MessageFromStepFunctions",source="step.functions")])GlueStep Functions supportsAWS Gluethrough the service integration pattern.You can call theStartJobRunAPI from aTaskstate.tasks.GlueStartJobRun(self,"Task",glue_job_name="my-glue-job",arguments=sfn.TaskInput.from_object({"key":"value"}),timeout=Duration.minutes(30),notify_delay_after=Duration.minutes(5))Glue DataBrewStep Functions supportsAWS Glue DataBrewthrough the service integration pattern.You can call theStartJobRunAPI from aTaskstate.tasks.GlueDataBrewStartJobRun(self,"Task",name="databrew-job")LambdaInvokea Lambda function.You can specify the input to your Lambda function through thepayloadattribute.
By default, Step Functions invokes Lambda function with the state input (JSON path '$')
as the input.The following snippet invokes a Lambda Function with the state input as the payload
by referencing the$path.# fn: lambda.Functiontasks.LambdaInvoke(self,"Invoke with state input",lambda_function=fn)When a function is invoked, the Lambda service sendsthese response
elementsback.⚠️ The response from the Lambda function is in an attribute calledPayloadThe following snippet invokes a Lambda Function by referencing the$.Payloadpath
to reference the output of a Lambda executed before it.# fn: lambda.Functiontasks.LambdaInvoke(self,"Invoke with empty object as payload",lambda_function=fn,payload=sfn.TaskInput.from_object({}))# use the output of fn as inputtasks.LambdaInvoke(self,"Invoke with payload field in the state input",lambda_function=fn,payload=sfn.TaskInput.from_json_path_at("$.Payload"))The following snippet invokes a Lambda and sets the task output to only include
the Lambda function response.# fn: lambda.Functiontasks.LambdaInvoke(self,"Invoke and set function response as task output",lambda_function=fn,output_path="$.Payload")If you want to combine the input and the Lambda function response you can use
thepayloadResponseOnlyproperty and specify theresultPath. This will put the
Lambda function ARN directly in the "Resource" string, but it conflicts with the
integrationPattern, invocationType, clientContext, and qualifier properties.# fn: lambda.Functiontasks.LambdaInvoke(self,"Invoke and combine function response with task input",lambda_function=fn,payload_response_only=True,result_path="$.fn")You can have Step Functions pause a task, and wait for an external process to
return a task token. Read more about thecallback patternTo use the callback pattern, set thetokenproperty on the task. Call the Step
FunctionsSendTaskSuccessorSendTaskFailureAPIs with the token to
indicate that the task has completed and the state machine should resume execution.The following snippet invokes a Lambda with the task token as part of the input
to the Lambda.# fn: lambda.Functiontasks.LambdaInvoke(self,"Invoke with callback",lambda_function=fn,integration_pattern=sfn.IntegrationPattern.WAIT_FOR_TASK_TOKEN,payload=sfn.TaskInput.from_object({"token":sfn.JsonPath.task_token,"input":sfn.JsonPath.string_at("$.someField")}))⚠️ The task will pause until it receives that task token back with aSendTaskSuccessorSendTaskFailurecall. Learn more aboutCallback with the Task
Token.AWS Lambda can occasionally experience transient service errors. In this case, invoking Lambda
results in a 500 error, such asServiceException,AWSLambdaException, orSdkClientException.
As a best practice, theLambdaInvoketask will retry on those errors with an interval of 2 seconds,
a back-off rate of 2 and 6 maximum attempts. Set theretryOnServiceExceptionsprop tofalseto
disable this behavior.SageMakerStep Functions supportsAWS SageMakerthrough the service integration pattern.If your training job or model uses resources from AWS Marketplace,network isolation is required.
To do so, set theenableNetworkIsolationproperty totrueforSageMakerCreateModelorSageMakerCreateTrainingJob.To set environment variables for the Docker container use theenvironmentproperty.Create Training JobYou can call theCreateTrainingJobAPI from aTaskstate.tasks.SageMakerCreateTrainingJob(self,"TrainSagemaker",training_job_name=sfn.JsonPath.string_at("$.JobName"),algorithm_specification=tasks.AlgorithmSpecification(algorithm_name="BlazingText",training_input_mode=tasks.InputMode.FILE),input_data_config=[tasks.Channel(channel_name="train",data_source=tasks.DataSource(s3_data_source=tasks.S3DataSource(s3_data_type=tasks.S3DataType.S3_PREFIX,s3_location=tasks.S3Location.from_json_expression("$.S3Bucket"))))],output_data_config=tasks.OutputDataConfig(s3_output_location=tasks.S3Location.from_bucket(s3.Bucket.from_bucket_name(self,"Bucket","mybucket"),"myoutputpath")),resource_config=tasks.ResourceConfig(instance_count=1,instance_type=ec2.InstanceType(sfn.JsonPath.string_at("$.InstanceType")),volume_size=Size.gibibytes(50)),# optional: default is 1 instance of EC2 `M4.XLarge` with `10GB` volumestopping_condition=tasks.StoppingCondition(max_runtime=Duration.hours(2)))Create Transform JobYou can call theCreateTransformJobAPI from aTaskstate.tasks.SageMakerCreateTransformJob(self,"Batch Inference",transform_job_name="MyTransformJob",model_name="MyModelName",model_client_options=tasks.ModelClientOptions(invocations_max_retries=3,# default is 0invocations_timeout=Duration.minutes(5)),transform_input=tasks.TransformInput(transform_data_source=tasks.TransformDataSource(s3_data_source=tasks.TransformS3DataSource(s3_uri="s3://inputbucket/train",s3_data_type=tasks.S3DataType.S3_PREFIX))),transform_output=tasks.TransformOutput(s3_output_path="s3://outputbucket/TransformJobOutputPath"),transform_resources=tasks.TransformResources(instance_count=1,instance_type=ec2.InstanceType.of(ec2.InstanceClass.M4,ec2.InstanceSize.XLARGE)))Create EndpointYou can call theCreateEndpointAPI from aTaskstate.tasks.SageMakerCreateEndpoint(self,"SagemakerEndpoint",endpoint_name=sfn.JsonPath.string_at("$.EndpointName"),endpoint_config_name=sfn.JsonPath.string_at("$.EndpointConfigName"))Create Endpoint ConfigYou can call theCreateEndpointConfigAPI from aTaskstate.tasks.SageMakerCreateEndpointConfig(self,"SagemakerEndpointConfig",endpoint_config_name="MyEndpointConfig",production_variants=[tasks.ProductionVariant(initial_instance_count=2,instance_type=ec2.InstanceType.of(ec2.InstanceClass.M5,ec2.InstanceSize.XLARGE),model_name="MyModel",variant_name="awesome-variant")])Create ModelYou can call theCreateModelAPI from aTaskstate.tasks.SageMakerCreateModel(self,"Sagemaker",model_name="MyModel",primary_container=tasks.ContainerDefinition(image=tasks.DockerImage.from_json_expression(sfn.JsonPath.string_at("$.Model.imageName")),mode=tasks.Mode.SINGLE_MODEL,model_s3_location=tasks.S3Location.from_json_expression("$.TrainingJob.ModelArtifacts.S3ModelArtifacts")))Update EndpointYou can call theUpdateEndpointAPI from aTaskstate.tasks.SageMakerUpdateEndpoint(self,"SagemakerEndpoint",endpoint_name=sfn.JsonPath.string_at("$.Endpoint.Name"),endpoint_config_name=sfn.JsonPath.string_at("$.Endpoint.EndpointConfig"))SNSStep Functions supportsAmazon SNSthrough the service integration pattern.You can call thePublishAPI from aTaskstate to publish to an SNS topic.topic=sns.Topic(self,"Topic")# Use a field from the execution data as message.task1=tasks.SnsPublish(self,"Publish1",topic=topic,integration_pattern=sfn.IntegrationPattern.REQUEST_RESPONSE,message=sfn.TaskInput.from_data_at("$.state.message"),message_attributes={"place":tasks.MessageAttribute(value=sfn.JsonPath.string_at("$.place")),"pic":tasks.MessageAttribute(# BINARY must be explicitly setdata_type=tasks.MessageAttributeDataType.BINARY,value=sfn.JsonPath.string_at("$.pic")),"people":tasks.MessageAttribute(value=4),"handles":tasks.MessageAttribute(value=["@kslater","@jjf",null,"@mfanning"])})# Combine a field from the execution data with# a literal object.task2=tasks.SnsPublish(self,"Publish2",topic=topic,message=sfn.TaskInput.from_object({"field1":"somedata","field2":sfn.JsonPath.string_at("$.field2")}))Step FunctionsStart ExecutionYou can manageAWS Step Functionsexecutions.AWS Step Functions supports it's ownStartExecutionAPI as a service integration.# Define a state machine with one Pass statechild=sfn.StateMachine(self,"ChildStateMachine",definition=sfn.Chain.start(sfn.Pass(self,"PassState")))# Include the state machine in a Task state with callback patterntask=tasks.StepFunctionsStartExecution(self,"ChildTask",state_machine=child,integration_pattern=sfn.IntegrationPattern.WAIT_FOR_TASK_TOKEN,input=sfn.TaskInput.from_object({"token":sfn.JsonPath.task_token,"foo":"bar"}),name="MyExecutionName")# Define a second state machine with the Task state abovesfn.StateMachine(self,"ParentStateMachine",definition=task)You can utilizeAssociate Workflow Executionsvia theassociateWithParentproperty. This allows the Step Functions UI to link child
executions from parent executions, making it easier to trace execution flow across state machines.# child: sfn.StateMachinetask=tasks.StepFunctionsStartExecution(self,"ChildTask",state_machine=child,associate_with_parent=True)This will add the payloadAWS_STEP_FUNCTIONS_STARTED_BY_EXECUTION_ID.$: $$.Execution.Idto theinputproperty for you, which will pass the execution ID from the context object to the
execution input. It requiresinputto be an object or not be set at all.Invoke ActivityYou can invoke aStep Functions Activitywhich enables you to have
a task in your state machine where the work is performed by aworkerthat can
be hosted on Amazon EC2, Amazon ECS, AWS Lambda, basically anywhere. Activities
are a way to associate code running somewhere (known as an activity worker) with
a specific task in a state machine.When Step Functions reaches an activity task state, the workflow waits for an
activity worker to poll for a task. An activity worker polls Step Functions by
using GetActivityTask, and sending the ARN for the related activity.After the activity worker completes its work, it can provide a report of its
success or failure by usingSendTaskSuccessorSendTaskFailure. These two
calls use the taskToken provided by GetActivityTask to associate the result
with that task.The following example creates an activity and creates a task that invokes the activity.submit_job_activity=sfn.Activity(self,"SubmitJob")tasks.StepFunctionsInvokeActivity(self,"Submit Job",activity=submit_job_activity)SQSStep Functions supportsAmazon SQSYou can call theSendMessageAPI from aTaskstate
to send a message to an SQS queue.queue=sqs.Queue(self,"Queue")# Use a field from the execution data as message.task1=tasks.SqsSendMessage(self,"Send1",queue=queue,message_body=sfn.TaskInput.from_json_path_at("$.message"))# Combine a field from the execution data with# a literal object.task2=tasks.SqsSendMessage(self,"Send2",queue=queue,message_body=sfn.TaskInput.from_object({"field1":"somedata","field2":sfn.JsonPath.string_at("$.field2")}))