Question: How can I push Amazon CloudWatch Logs cross-account to Kinesis Data Firehose?

I want to stream Amazon CloudWatch Logs from Amazon Kinesis Data Firehose into another account in a different AWS Region.

Short descriptionTo send Amazon CloudWatch Logs to a Kinesis Data Firehose stream in a different Region, the Region must support Kinesis Data Firehose. Therefore, make sure that your Region supports Kinesis Data Firehose.To establish cross-account and cross-Region streaming using Kinesis Data Firehose in a supported Region, perform the following steps:1.    Create an Amazon Simple Storage Service (Amazon S3) bucket in the destination account. Create an AWS Identity and Access Management (IAM) role. Then, attach the required permission for Kinesis Data Firehose to push data to Amazon S3.2.    Create a destination for Kinesis Data Firehose in the destination account. Create an IAM role for Amazon CloudWatch Logs service to push data to Kinesis Data Firehose service. Then, create a destination delivery stream where the logs will be pushed to.3.    Turn on VPC Flow Logs and push the logs to Amazon CloudWatch for the source account.4.    Create a subscription filter in the source account that points to the destination account.5.    Validate the flow of log events in the S3 bucket in the destination account.

ResolutionNote: This resolution uses the following example values that you must replace with your own values:Destination account: 111111111111Kinesis Data Firehose Region: us-east-1S3 bucket Region: us-west-2Destination Region (receiving logs from source account): us-east-2Source account: 222222222222Amazon CloudWatch log group Region: us-east2VPC Flow Logs Region: us-east-2Set up the destination account1.    Create an S3 bucket:aws s3api create-bucket --bucket my-bucket --create-bucket-configuration LocationConstraint=us-west-2 --region us-west-2The location constraint indicates that the bucket will be created in the us-west-2 Region.2.    Create the IAM role and trust policy that grant Kinesis Data Firehose the required permissions:{  "Statement": {    "Effect": "Allow",    "Principal": {      "Service": "firehose.amazonaws.com"    },    "Action": "sts:AssumeRole",    "Condition": {      "StringEquals": {        "sts:ExternalId": "111111111111"      }    }  }}The permission settings must allow Kinesis Data Firehose to put data into the S3 bucket that you created. Replace 111111111111 with your AWS account ID.3.    Create the IAM role and specify the trust policy file:aws iam create-role \    --role-name FirehosetoS3Role \    --assume-role-policy-document file://~/TrustPolicyForFirehose.jsonNote: You must use the Role_Arn value in a later step.4.    Create a permissions policy in a JSON file to define the actions that Kinesis Data Firehose can perform in the destination account:{  "Statement": [    {      "Effect": "Allow",      "Action": [        "s3:AbortMultipartUpload",        "s3:GetBucketLocation",        "s3:GetObject",        "s3:ListBucket",        "s3:ListBucketMultipartUploads",        "s3:PutObjectAcl",        "s3:PutObject"      ],      "Resource": [        "arn:aws:s3:::my-bucket",        "arn:aws:s3:::my-bucket/*"      ]    }  ]}5.    Associate the permissions policy with the IAM role:aws iam put-role-policy --role-name FirehosetoS3Role --policy-name Permissions-Policy-For-Firehose --policy-document file://~/PermissionsForFirehose.json6.    Create a destination delivery stream for Kinesis Data Firehose:aws firehose create-delivery-stream --delivery-stream-name 'my-delivery-stream' --s3-destination-configuration RoleARN='arn:aws:iam::111111111111:role/FirehosetoS3Role',BucketARN='arn:aws:s3:::my-bucket' --region us-east-1Replace RoleARN and BucketARN with the role and bucket Amazon Resource Names (ARNs) that you created.Note: When you successfully deliver an S3 object to Kinesis Data Firehose, a custom prefix is used in the timestamp namespace expression. You can add and specify an extra prefix at the front of the time format prefix (yyyy/MM/dd/HH/). If the prefix ends with a forward slash (/), then it appears as a folder in the S3 bucket.7.    Use the describe-delivery-stream command to check the DeliveryStreamDescription.DeliveryStreamStatus property:aws firehose describe-delivery-stream --delivery-stream-name "my-delivery-stream" --region us-east-1To confirm that the stream is active, check the describe-delivery-stream command output:{  "DeliveryStreamDescription": {    "DeliveryStreamType": "DirectPut",    "HasMoreDestinations": false,    "DeliveryStreamEncryptionConfiguration": {      "Status": "DISABLED"    },    "VersionId": "1",    "CreateTimestamp": 1604484348.804,    "DeliveryStreamARN": "arn:aws:firehose:us-east-1:111111111111:deliverystream/my-delivery-stream",    "DeliveryStreamStatus": "ACTIVE",    "DeliveryStreamName": "my-delivery-stream",    "Destinations": [      {        "DestinationId": "destinationId-000000000001",        "ExtendedS3DestinationDescription": {          "RoleARN": "arn:aws:iam::111111111111:role/FirehosetoS3Role2test",          "BufferingHints": {            "IntervalInSeconds": 300,            "SizeInMBs": 5          },          "EncryptionConfiguration": {            "NoEncryptionConfig": "NoEncryption"          },          "CompressionFormat": "UNCOMPRESSED",          "S3BackupMode": "Disabled",          "CloudWatchLoggingOptions": {            "Enabled": false          },          "BucketARN": "arn:aws:s3:::my-bucket"        },        "S3DestinationDescription": {          "RoleARN": "arn:aws:iam::111111111111:role/FirehosetoS3Role2test",          "BufferingHints": {            "IntervalInSeconds": 300,            "SizeInMBs": 5          },          "EncryptionConfiguration": {            "NoEncryptionConfig": "NoEncryption"          },          "CompressionFormat": "UNCOMPRESSED",          "CloudWatchLoggingOptions": {            "Enabled": false          },          "BucketARN": "arn:aws:s3:::my-bucket"        }      }    ]  }}Note: You must use the DeliveryStreamDescription.DeliveryStreamARN value in a later step.8.    Create the IAM role and trust policy that grant CloudWatch Logs the permission to put data into the Kinesis Data Firehose stream. Make sure to add the Regions where the logs are pushed:{  "Statement": {    "Effect": "Allow",    "Principal": {      "Service": [        "logs.us-east-1.amazonaws.com",        "logs.us-east-2.amazonaws.com"      ]    },    "Action": "sts:AssumeRole"  }}9.    To create the IAM role and specify the trust policy file, use the create-role command:aws iam create-role \    --role-name CWLtoKinesisFirehoseRole \    --assume-role-policy-document file://~/TrustPolicyForCWL.jsonNote: You must use the returned Role_Arn value in a later step.10.    Create a permissions policy to define the actions that CloudWatch Logs can perform in destination account:{  "Statement": [    {      "Effect": "Allow",      "Action": [        "firehose:*"      ],      "Resource": [        "arn:aws:firehose:us-east-1:111111111111:*"      ]    },    {      "Effect": "Allow",      "Action": [        "iam:PassRole"      ],      "Resource": [        "arn:aws:iam::111111111111:role/CWLtoKinesisFirehoseRole"      ]    }  ]}Use the DeliveryStreamDescription.DeliveryStreamARN value from step 7 and the Role_Arn value from step 9.11.    Associate the permissions policy with the role using the put-role-policy command:aws iam put-role-policy --role-name CWLtoKinesisFirehoseRole --policy-name Permissions-Policy-For-CWL --policy-document file://~/PermissionsForCWL.json12.    Create a destination in the destination account using put-destination API call. This is the destination that the source account is sending all the logs to:aws logs put-destination --destination-name "myDestination" --target-arn "arn:aws:firehose:us-east-1:111111111111:deliverystream/my-delivery-stream"--role-arn "arn:aws:iam::111111111111:role/CWLtoKinesisFirehoseRole" --region us-east-2Note: You can create a destination for the delivery stream in any Region where Kinesis Data Firehose is supported. The Region where you create the destination must be same as the log source Region.13.    Create an access policy for the Amazon CloudWatch destination:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "",      "Effect": "Allow",      "Principal": {        "AWS": "222222222222"      },      "Action": "logs:PutSubscriptionFilter",      "Resource": "arn:aws:logs:us-east-2:111111111111:destination:myDestination"    }  ]}Replace 222222222222 with the AWS source account where the Amazon Virtual Private Cloud (Amazon VPC) logs that you created are located.14.    Associate the access policy with the Amazon CloudWatch destination:aws logs put-destination-policy --destination-name "myDestination" --access-policy file://~/AccessPolicy.json --region us-east-215.    To verify the destination, run the following command:aws logs describe-destinations --region us-east-2Setting up the source accountNote: You must be the IAM admin user or root user of the source account.1.    Create an IAM role and trust policy to grant VPC Flow Logs the permissions to send data to the CloudWatch Logs log group:{  "Version": "2012-10-17",  "Statement": [    {      "Sid": "",      "Effect": "Allow",      "Principal": {        "Service": "vpc-flow-logs.amazonaws.com"      },      "Action": "sts:AssumeRole"    }  ]}2.    To create the IAM role and specify the trust policy file you created, use the following command:aws iam create-role --role-name PublishFlowLogs --assume-role-policy-document file://~/TrustPolicyForVPCFlowLogs.jsonNote: You'll use the returned ARN value to be passed on to VPC Flow Logs in a later step.3.    Create a permissions policy to define the actions that VPC Flow Logs can perform in the source account:{  "Version": "2012-10-17",  "Statement": [    {      "Action": [        "logs:CreateLogGroup",        "logs:CreateLogStream",        "logs:PutLogEvents",        "logs:DescribeLogGroups",        "logs:DescribeLogStreams"      ],      "Effect": "Allow",      "Resource": "*"    }  ]}4.    To associate the permissions policy with the IAM role, run the following command:aws iam put-role-policy --role-name PublishFlowLogs --policy-name Permissions-Policy-For-VPCFlowLogs --policy-document file://~/PermissionsForVPCFlowLogs.json5.    Create a CloudWatch Logs log group to configure the destination for the VPC Flow Logs:aws logs create-log-group --log-group-name vpc-flow-logs --region us-east-26.    To turn on VPC Flow Logs, run the following command:aws ec2 create-flow-logs --resource-type VPC --resource-ids vpc-12345678 --traffic-type ALL --log-group-name vpc-flow-logs --deliver-logs-permission-arn arn:aws:iam::222222222222:role/PublishFlowLogs --region us-east-2Note: Replace the --resource-ids and --deliver-logs-permission-arn placeholder values with your VPC ID and VPC Flow Logs role.7.    Subscribe the CloudWatch Logs log group to Kinesis Data Firehose in the destination account:aws logs put-subscription-filter --log-group-name "vpc-flow-logs" --filter-name "AllTraffic" --filter-pattern "" --destination-arn"arn:aws:logs:us-east-2:111111111111:destination:myDestination" --region us-east-2Update the --destination ARN value and replace "111111111111" with the destination account number.8.    Check the S3 bucket to confirm that the logs are published.

Related informationDeliveryStreamDescription