Package: aws-cdk.aws-kinesisfirehose
Amazon Kinesis Data Firehose Construct Library---AWS CDK v1 has reached End-of-Support on 2023-06-01.
This package is no longer being updated, and users should migrate to AWS CDK v2.For more information on how to migrate, see theMigrating to AWS CDK v2guide.Amazon Kinesis Data Firehoseis a service for fully-managed delivery of real-time streaming data to storage services
such as Amazon S3, Amazon Redshift, Amazon Elasticsearch, Splunk, or any custom HTTP
endpoint or third-party services such as Datadog, Dynatrace, LogicMonitor, MongoDB, New
Relic, and Sumo Logic.Kinesis Data Firehose delivery streams are distinguished from Kinesis data streams in
their models of consumtpion. Whereas consumers read from a data stream by actively pulling
data from the stream, a delivery stream pushes data to its destination on a regular
cadence. This means that data streams are intended to have consumers that do on-demand
processing, like AWS Lambda or Amazon EC2. On the other hand, delivery streams are
intended to have destinations that are sources for offline processing and analytics, such
as Amazon S3 and Amazon Redshift.This module is part of theAWS Cloud Development Kitproject. It allows you to define Kinesis Data Firehose delivery streams.Defining a Delivery StreamIn order to define a Delivery Stream, you must specify a destination. An S3 bucket can be
used as a destination. More supported destinations are coveredbelow.bucket=s3.Bucket(self,"Bucket")firehose.DeliveryStream(self,"Delivery Stream",destinations=[destinations.S3Bucket(bucket)])The above example defines the following resources:An S3 bucketA Kinesis Data Firehose delivery stream with Direct PUT as the source and CloudWatch
error logging turned on.An IAM role which gives the delivery stream permission to write to the S3 bucket.SourcesThere are two main methods of sourcing input data: Kinesis Data Streams and via a "direct
put".See:Sending Data to a Delivery Streamin theKinesis Data Firehose Developer Guide.Kinesis Data StreamA delivery stream can read directly from a Kinesis data stream as a consumer of the data
stream. Configure this behaviour by providing a data stream in thesourceStreamproperty when constructing a delivery stream:# destination: firehose.IDestinationsource_stream=kinesis.Stream(self,"Source Stream")firehose.DeliveryStream(self,"Delivery Stream",source_stream=source_stream,destinations=[destination])Direct PutData must be provided via "direct put", ie., by using aPutRecordorPutRecordBatchAPI call. There are a number of ways of doing
so, such as:Kinesis Agent: a standalone Java application that monitors and delivers files while
handling file rotation, checkpointing, and retries. See:Writing to Kinesis Data Firehose Using Kinesis Agentin theKinesis Data Firehose Developer Guide.AWS SDK: a general purpose solution that allows you to deliver data to a delivery stream
from anywhere using Java, .NET, Node.js, Python, or Ruby. See:Writing to Kinesis Data Firehose Using the AWS SDKin theKinesis Data Firehose Developer Guide.CloudWatch Logs: subscribe to a log group and receive filtered log events directly into
a delivery stream. See:logs-destinations.Eventbridge: add an event rule target to send events to a delivery stream based on the
rule filtering. See:events-targets.SNS: add a subscription to send all notifications from the topic to a delivery
stream. See:sns-subscriptions.IoT: add an action to an IoT rule to send various IoT information to a delivery streamDestinationsThe following destinations are supported. Seekinesisfirehose-destinationsfor the implementations of these destinations.S3Defining a delivery stream with an S3 bucket destination:# bucket: s3.Buckets3_destination=destinations.S3Bucket(bucket)firehose.DeliveryStream(self,"Delivery Stream",destinations=[s3_destination])The S3 destination also supports custom dynamic prefixes.prefixwill be used for files
successfully delivered to S3.errorOutputPrefixwill be added to failed records before
writing them to S3.# bucket: s3.Buckets3_destination=destinations.S3Bucket(bucket,data_output_prefix="myFirehose/DeliveredYear=!{timestamp:yyyy}/anyMonth/rand=!{firehose:random-string}",error_output_prefix="myFirehoseFailures/!{firehose:error-output-type}/!{timestamp:yyyy}/anyMonth/!{timestamp:dd}")See:Custom S3 Prefixesin theKinesis Data Firehose Developer Guide.Server-side EncryptionEnabling server-side encryption (SSE) requires Kinesis Data Firehose to encrypt all data
sent to delivery stream when it is stored at rest. This means that data is encrypted
before being written to the service's internal storage layer and decrypted after it is
received from the internal storage layer. The service manages keys and cryptographic
operations so that sources and destinations do not need to, as the data is encrypted and
decrypted at the boundaries of the service (ie., before the data is delivered to a
destination). By default, delivery streams do not have SSE enabled.The Key Management Service (KMS) Customer Managed Key (CMK) used for SSE can either be
AWS-owned or customer-managed. AWS-owned CMKs are keys that an AWS service (in this case
Kinesis Data Firehose) owns and manages for use in multiple AWS accounts. As a customer,
you cannot view, use, track, or manage these keys, and you are not charged for their
use. On the other hand, customer-managed CMKs are keys that are created and owned within
your account and managed entirely by you. As a customer, you are responsible for managing
access, rotation, aliases, and deletion for these keys, and you are changed for their
use. See:Customer master keysin theKMS Developer Guide.# destination: firehose.IDestination# SSE with an customer-managed CMK that is explicitly specified# key: kms.Key# SSE with an AWS-owned CMKfirehose.DeliveryStream(self,"Delivery Stream AWS Owned",encryption=firehose.StreamEncryption.AWS_OWNED,destinations=[destination])# SSE with an customer-managed CMK that is created automatically by the CDKfirehose.DeliveryStream(self,"Delivery Stream Implicit Customer Managed",encryption=firehose.StreamEncryption.CUSTOMER_MANAGED,destinations=[destination])firehose.DeliveryStream(self,"Delivery Stream Explicit Customer Managed",encryption_key=key,destinations=[destination])See:Data Protectionin
theKinesis Data Firehose Developer Guide.MonitoringKinesis Data Firehose is integrated with CloudWatch, so you can monitor the performance of
your delivery streams via logs and metrics.LogsKinesis Data Firehose will send logs to CloudWatch when data transformation or data
delivery fails. The CDK will enable logging by default and create a CloudWatch LogGroup
and LogStream for your Delivery Stream.You can provide a specific log group to specify where the CDK will create the log streams
where log events will be sent:importaws_cdk.aws_logsaslogs# bucket: s3.Bucket# destination: firehose.IDestinationlog_group=logs.LogGroup(self,"Log Group")destination=destinations.S3Bucket(bucket,log_group=log_group)firehose.DeliveryStream(self,"Delivery Stream",destinations=[destination])Logging can also be disabled:# bucket: s3.Bucketdestination=destinations.S3Bucket(bucket,logging=False)firehose.DeliveryStream(self,"Delivery Stream",destinations=[destination])See:Monitoring using CloudWatch Logsin theKinesis Data Firehose Developer Guide.MetricsKinesis Data Firehose sends metrics to CloudWatch so that you can collect and analyze the
performance of the delivery stream, including data delivery, data ingestion, data
transformation, format conversion, API usage, encryption, and resource usage. You can then
use CloudWatch alarms to alert you, for example, when data freshness (the age of the
oldest record in the delivery stream) exceeds the buffering limit (indicating that data is
not being delivered to your destination), or when the rate of incoming records exceeds the
limit of records per second (indicating data is flowing into your delivery stream faster
than it is configured to process).CDK provides methods for accessing delivery stream metrics with default configuration,
such asmetricIncomingBytes, andmetricIncomingRecords(seeIDeliveryStreamfor a full list). CDK also provides a genericmetricmethod that can be used to produce
metric configurations for any metric provided by Kinesis Data Firehose; the configurations
are pre-populated with the correct dimensions for the delivery stream.importaws_cdk.aws_cloudwatchascloudwatch# delivery_stream: firehose.DeliveryStream# Alarm that triggers when the per-second average of incoming bytes exceeds 90% of the current service limitincoming_bytes_percent_of_limit=cloudwatch.MathExpression(expression="incomingBytes / 300 / bytePerSecLimit",using_metrics={"incoming_bytes":delivery_stream.metric_incoming_bytes(statistic=cloudwatch.Statistic.SUM),"byte_per_sec_limit":delivery_stream.metric("BytesPerSecondLimit")})cloudwatch.Alarm(self,"Alarm",metric=incoming_bytes_percent_of_limit,threshold=0.9,evaluation_periods=3)See:Monitoring Using CloudWatch Metricsin theKinesis Data Firehose Developer Guide.CompressionYour data can automatically be compressed when it is delivered to S3 as either a final or
an intermediary/backup destination. Supported compression formats are: gzip, Snappy,
Hadoop-compatible Snappy, and ZIP, except for Redshift destinations, where Snappy
(regardless of Hadoop-compatibility) and ZIP are not supported. By default, data is
delivered to S3 without compression.# Compress data delivered to S3 using Snappy# bucket: s3.Buckets3_destination=destinations.S3Bucket(bucket,compression=destinations.Compression.SNAPPY)firehose.DeliveryStream(self,"Delivery Stream",destinations=[s3_destination])BufferingIncoming data is buffered before it is delivered to the specified destination. The
delivery stream will wait until the amount of incoming data has exceeded some threshold
(the "buffer size") or until the time since the last data delivery occurred exceeds some
threshold (the "buffer interval"), whichever happens first. You can configure these
thresholds based on the capabilities of the destination and your use-case. By default, the
buffer size is 5 MiB and the buffer interval is 5 minutes.# Increase the buffer interval and size to 10 minutes and 8 MiB, respectively# bucket: s3.Bucketdestination=destinations.S3Bucket(bucket,buffering_interval=Duration.minutes(10),buffering_size=Size.mebibytes(8))firehose.DeliveryStream(self,"Delivery Stream",destinations=[destination])See:Data Delivery Frequencyin theKinesis Data Firehose Developer Guide.Destination EncryptionYour data can be automatically encrypted when it is delivered to S3 as a final or
an intermediary/backup destination. Kinesis Data Firehose supports Amazon S3 server-side
encryption with AWS Key Management Service (AWS KMS) for encrypting delivered data
in Amazon S3. You can choose to not encrypt the data or to encrypt with a key from
the list of AWS KMS keys that you own. For more information, seeProtecting Data
Using Server-Side Encryption with AWS KMS–Managed Keys (SSE-KMS). Data is not encrypted by default.# bucket: s3.Bucket# key: kms.Keydestination=destinations.S3Bucket(bucket,encryption_key=key)firehose.DeliveryStream(self,"Delivery Stream",destinations=[destination])BackupA delivery stream can be configured to backup data to S3 that it attempted to deliver to
the configured destination. Backed up data can be all the data that the delivery stream
attempted to deliver or just data that it failed to deliver (Redshift and S3 destinations
can only backup all data). CDK can create a new S3 bucket where it will back up data or
you can provide a bucket where data will be backed up. You can also provide a prefix under
which your backed-up data will be placed within the bucket. By default, source data is not
backed up to S3.# Enable backup of all source records (to an S3 bucket created by CDK).# bucket: s3.Bucket# Explicitly provide an S3 bucket to which all source records will be backed up.# backup_bucket: s3.Bucketfirehose.DeliveryStream(self,"Delivery Stream Backup All",destinations=[destinations.S3Bucket(bucket,s3_backup=destinations.DestinationS3BackupProps(mode=destinations.BackupMode.ALL))])firehose.DeliveryStream(self,"Delivery Stream Backup All Explicit Bucket",destinations=[destinations.S3Bucket(bucket,s3_backup=destinations.DestinationS3BackupProps(bucket=backup_bucket))])# Explicitly provide an S3 prefix under which all source records will be backed up.firehose.DeliveryStream(self,"Delivery Stream Backup All Explicit Prefix",destinations=[destinations.S3Bucket(bucket,s3_backup=destinations.DestinationS3BackupProps(mode=destinations.BackupMode.ALL,data_output_prefix="mybackup"))])If any Data Processing or Transformation is configured on your Delivery Stream, the source
records will be backed up in their original format.Data Processing/TransformationData can be transformed before being delivered to destinations. There are two types of
data processing for delivery streams: record transformation with AWS Lambda, and record
format conversion using a schema stored in an AWS Glue table. If both types of data
processing are configured, then the Lambda transformation is performed first. By default,
no data processing occurs. This construct library currently only supports data
transformation with AWS Lambda. See#15501to track the status of adding support for record format conversion.Data transformation with AWS LambdaTo transform the data, Kinesis Data Firehose will call a Lambda function that you provide
and deliver the data returned in place of the source record. The function must return a
result that contains records in a specific format, including the following fields:recordId-- the ID of the input record that corresponds the results.result-- the status of the transformation of the record: "Ok" (success), "Dropped"
(not processed intentionally), or "ProcessingFailed" (not processed due to an error).data-- the transformed data, Base64-encoded.The data is buffered up to 1 minute and up to 3 MiB by default before being sent to the
function, but can be configured usingbufferIntervalandbufferSizein the processor
configuration (see:Buffering). If the function invocation fails due to a
network timeout or because of hitting an invocation limit, the invocation is retried 3
times by default, but can be configured usingretriesin the processor configuration.# bucket: s3.Bucket# Provide a Lambda function that will transform records before delivery, with custom# buffering and retry configurationlambda_function=lambda_.Function(self,"Processor",runtime=lambda_.Runtime.NODEJS_14_X,handler="index.handler",code=lambda_.Code.from_asset(path.join(__dirname,"process-records")))lambda_processor=firehose.LambdaFunctionProcessor(lambda_function,buffer_interval=Duration.minutes(5),buffer_size=Size.mebibytes(5),retries=5)s3_destination=destinations.S3Bucket(bucket,processor=lambda_processor)firehose.DeliveryStream(self,"Delivery Stream",destinations=[s3_destination])importpathaspathimportaws_cdk.aws_kinesisfirehoseasfirehoseimportaws_cdk.aws_kmsaskmsimportaws_cdk.aws_lambda_nodejsaslambdanodejsimportaws_cdk.aws_logsaslogsimportaws_cdk.aws_s3ass3importaws_cdk.coreascdkimportaws_cdk.aws_kinesisfirehose_destinationsasdestinationsapp=cdk.App()stack=cdk.Stack(app,"aws-cdk-firehose-delivery-stream-s3-all-properties")bucket=s3.Bucket(stack,"Bucket",removal_policy=cdk.RemovalPolicy.DESTROY,auto_delete_objects=True)backup_bucket=s3.Bucket(stack,"BackupBucket",removal_policy=cdk.RemovalPolicy.DESTROY,auto_delete_objects=True)log_group=logs.LogGroup(stack,"LogGroup",removal_policy=cdk.RemovalPolicy.DESTROY)data_processor_function=lambdanodejs.NodejsFunction(stack,"DataProcessorFunction",entry=path.join(__dirname,"lambda-data-processor.js"),timeout=cdk.Duration.minutes(1))processor=firehose.LambdaFunctionProcessor(data_processor_function,buffer_interval=cdk.Duration.seconds(60),buffer_size=cdk.Size.mebibytes(1),retries=1)key=kms.Key(stack,"Key",removal_policy=cdk.RemovalPolicy.DESTROY)backup_key=kms.Key(stack,"BackupKey",removal_policy=cdk.RemovalPolicy.DESTROY)firehose.DeliveryStream(stack,"Delivery Stream",destinations=[destinations.S3Bucket(bucket,logging=True,log_group=log_group,processor=processor,compression=destinations.Compression.GZIP,data_output_prefix="regularPrefix",error_output_prefix="errorPrefix",buffering_interval=cdk.Duration.seconds(60),buffering_size=cdk.Size.mebibytes(1),encryption_key=key,s3_backup=destinations.DestinationS3BackupProps(mode=destinations.BackupMode.ALL,bucket=backup_bucket,compression=destinations.Compression.ZIP,data_output_prefix="backupPrefix",error_output_prefix="backupErrorPrefix",buffering_interval=cdk.Duration.seconds(60),buffering_size=cdk.Size.mebibytes(1),encryption_key=backup_key))])app.synth()!cdk-integ pragma:ignore-assetsimportpathaspathimportaws_cdk.aws_kinesisfirehoseasfirehoseimportaws_cdk.aws_kmsaskmsimportaws_cdk.aws_lambda_nodejsaslambdanodejsimportaws_cdk.aws_logsaslogsimportaws_cdk.aws_s3ass3importaws_cdk.coreascdkimportaws_cdk.aws_kinesisfirehose_destinationsasdestinationsapp=cdk.App()stack=cdk.Stack(app,"aws-cdk-firehose-delivery-stream-s3-all-properties")bucket=s3.Bucket(stack,"Bucket",removal_policy=cdk.RemovalPolicy.DESTROY,auto_delete_objects=True)backup_bucket=s3.Bucket(stack,"BackupBucket",removal_policy=cdk.RemovalPolicy.DESTROY,auto_delete_objects=True)log_group=logs.LogGroup(stack,"LogGroup",removal_policy=cdk.RemovalPolicy.DESTROY)data_processor_function=lambdanodejs.NodejsFunction(stack,"DataProcessorFunction",entry=path.join(__dirname,"lambda-data-processor.js"),timeout=cdk.Duration.minutes(1))processor=firehose.LambdaFunctionProcessor(data_processor_function,buffer_interval=cdk.Duration.seconds(60),buffer_size=cdk.Size.mebibytes(1),retries=1)key=kms.Key(stack,"Key",removal_policy=cdk.RemovalPolicy.DESTROY)backup_key=kms.Key(stack,"BackupKey",removal_policy=cdk.RemovalPolicy.DESTROY)firehose.DeliveryStream(stack,"Delivery Stream",destinations=[destinations.S3Bucket(bucket,logging=True,log_group=log_group,processor=processor,compression=destinations.Compression.GZIP,data_output_prefix="regularPrefix",error_output_prefix="errorPrefix",buffering_interval=cdk.Duration.seconds(60),buffering_size=cdk.Size.mebibytes(1),encryption_key=key,s3_backup=destinations.DestinationS3BackupProps(mode=destinations.BackupMode.ALL,bucket=backup_bucket,compression=destinations.Compression.ZIP,data_output_prefix="backupPrefix",error_output_prefix="backupErrorPrefix",buffering_interval=cdk.Duration.seconds(60),buffering_size=cdk.Size.mebibytes(1),encryption_key=backup_key))])app.synth()See:Data Transformationin theKinesis Data Firehose Developer Guide.Specifying an IAM roleThe DeliveryStream class automatically creates IAM service roles with all the minimum
necessary permissions for Kinesis Data Firehose to access the resources referenced by your
delivery stream. One service role is created for the delivery stream that allows Kinesis
Data Firehose to read from a Kinesis data stream (if one is configured as the delivery
stream source) and for server-side encryption. Another service role is created for each
destination, which gives Kinesis Data Firehose write access to the destination resource,
as well as the ability to invoke data transformers and read schemas for record format
conversion. If you wish, you may specify your own IAM role for either the delivery stream
or the destination service role, or both. It must have the correct trust policy (it must
allow Kinesis Data Firehose to assume it) or delivery stream creation or data delivery
will fail. Other required permissions to destination resources, encryption keys, etc.,
will be provided automatically.# Specify the roles created above when defining the destination and delivery stream.# bucket: s3.Bucket# Create service roles for the delivery stream and destination.# These can be used for other purposes and granted access to different resources.# They must include the Kinesis Data Firehose service principal in their trust policies.# Two separate roles are shown below, but the same role can be used for both purposes.delivery_stream_role=iam.Role(self,"Delivery Stream Role",assumed_by=iam.ServicePrincipal("firehose.amazonaws.com"))destination_role=iam.Role(self,"Destination Role",assumed_by=iam.ServicePrincipal("firehose.amazonaws.com"))destination=destinations.S3Bucket(bucket,role=destination_role)firehose.DeliveryStream(self,"Delivery Stream",destinations=[destination],role=delivery_stream_role)SeeControlling Accessin theKinesis Data Firehose Developer Guide.Granting application access to a delivery streamIAM roles, users or groups which need to be able to work with delivery streams should be
granted IAM permissions.Any object that implements theIGrantableinterface (ie., has an associated principal)
can be granted permissions to a delivery stream by calling:grantPutRecords(principal)- grants the principal the ability to put records onto the
delivery streamgrant(principal, ...actions)- grants the principal permission to a custom set of
actions# Give the role permissions to write data to the delivery stream# delivery_stream: firehose.DeliveryStreamlambda_role=iam.Role(self,"Role",assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"))delivery_stream.grant_put_records(lambda_role)The following write permissions are provided to a service principal by thegrantPutRecords()method:firehose:PutRecordfirehose:PutRecordBatchGranting a delivery stream access to a resourceConversely to the above, Kinesis Data Firehose requires permissions in order for delivery
streams to interact with resources that you own. For example, if an S3 bucket is specified
as a destination of a delivery stream, the delivery stream must be granted permissions to
put and get objects from the bucket. When using the built-in AWS service destinations
found in the@aws-cdk/aws-kinesisfirehose-destinationsmodule, the CDK grants the
permissions automatically. However, custom or third-party destinations may require custom
permissions. In this case, use the delivery stream as anIGrantable, as follows:# delivery_stream: firehose.DeliveryStreamfn=lambda_.Function(self,"Function",code=lambda_.Code.from_inline("exports.handler = (event) =>{}"),runtime=lambda_.Runtime.NODEJS_14_X,handler="index.handler")fn.grant_invoke(delivery_stream)Multiple destinationsThough the delivery stream allows specifying an array of destinations, only one
destination per delivery stream is currently allowed. This limitation is enforced at CDK
synthesis time and will throw an error.