Repository: localstack/localstack

Issue:
bug: Multiple firehose delivery streams subscribing to kinesis stream doesn't fire both. 
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

1. Create kinesis stream with `awslocal kinesis create-stream`
2. Create two firehose delivery stream with `awslocal firehose create-delivery-stream`, both of which have `--kinesis-stream-source-configuration` to the stream created above
3. Put record to the stream
4. Only one of the delivery stream receives the message

### Expected Behavior

At the 4th step, both of the delivery stream should receive the message.
(which was true in real AWS environment.)

### How are you starting LocalStack?

With a `docker run` command

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

    docker run localstack/localstack:2.3.2

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

    docker exec -it admiring_chatelet bash
    (in the localstack container)
    awslocal opensearch create-domain --domain-name my-domain
    awslocal kinesis create-stream --stream-name samplestream --shard-count 2 --region dummy
    awslocal firehose create-delivery-stream --delivery-stream-name s1 --delivery-stream-type KinesisStreamAsSource --kinesis-stream-source-configuration "KinesisStreamARN=arn:aws:kinesis:us-east-1:000000000000:stream/samplestream,RoleARN=arn:aws:kinesis:us-east-1:000000000000:role/stream-role" --amazonopensearchservice-destination-configuration "ClusterEndpoint=http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566,IndexName=index1,TypeName=_doc,RoleARN=arn:aws:kinesis:us-east-1:000000000000:role/es-role,S3Configuration={RoleARN=arn:aws:iam::000000000000:role/Firehose-Reader-Role,BucketARN=arn:aws:s3:::kinesis-activity-backup-local}"
    awslocal firehose create-delivery-stream --delivery-stream-name s2 --delivery-stream-type KinesisStreamAsSource --kinesis-stream-source-configuration "KinesisStreamARN=arn:aws:kinesis:us-east-1:000000000000:stream/samplestream,RoleARN=arn:aws:kinesis:us-east-1:000000000000:role/stream-role" --amazonopensearchservice-destination-configuration "ClusterEndpoint=http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566,IndexName=index2,TypeName=_doc,RoleARN=arn:aws:kinesis:us-east-1:000000000000:role/es-role,S3Configuration={RoleARN=arn:aws:iam::000000000000:role/Firehose-Reader-Role,BucketARN=arn:aws:s3:::kinesis-activity-backup-local}"
    awslocal kinesis put-record --stream-name samplestream --data '{"test":"value"}' --partition-key 1
    curl http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566/index1/_search
    curl http://my-domain.us-east-1.opensearch.localhost.localstack.cloud:4566/index2/_search

by the last two `curl`s, only one of the results shows the given data.

### Environment

```markdown
- OS: MacOS 13.2.1
- LocalStack: 2.3.2
```


### Anything else?

_No response_


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [contributing guide](https://docs.localstack.cloud/contributing/).

Resolving patch:
diff --git a/localstack/services/firehose/provider.py b/localstack/services/firehose/provider.py
--- a/localstack/services/firehose/provider.py
+++ b/localstack/services/firehose/provider.py
@@ -320,7 +320,7 @@ def _startup():
                         region_name=context.region,
                         listener_func=listener_function,
                         wait_until_started=True,
-                        ddb_lease_table_suffix="-firehose",
+                        ddb_lease_table_suffix=f"-firehose-{delivery_stream_name}",
                     )
 
                     self.kinesis_listeners[delivery_stream_arn] = process


Test patch:
diff --git a/localstack/testing/pytest/fixtures.py b/localstack/testing/pytest/fixtures.py
--- a/localstack/testing/pytest/fixtures.py
+++ b/localstack/testing/pytest/fixtures.py
@@ -1590,13 +1590,18 @@ def lambda_su_role(aws_client):
 def create_iam_role_with_policy(aws_client):
     roles = {}
 
-    def _create_role_and_policy(**kwargs):
+    def _create_role_and_policy(**kwargs: dict[str, any]) -> str:
+        if "RoleName" not in kwargs:
+            kwargs["RoleName"] = f"test-role-{short_uid()}"
         role = kwargs["RoleName"]
+        if "PolicyName" not in kwargs:
+            kwargs["PolicyName"] = f"test-policy-{short_uid()}"
         policy = kwargs["PolicyName"]
         role_policy = json.dumps(kwargs["RoleDefinition"])
 
         result = aws_client.iam.create_role(RoleName=role, AssumeRolePolicyDocument=role_policy)
         role_arn = result["Role"]["Arn"]
+
         policy_document = json.dumps(kwargs["PolicyDefinition"])
         aws_client.iam.put_role_policy(
             RoleName=role, PolicyName=policy, PolicyDocument=policy_document
@@ -1613,21 +1618,24 @@ def _create_role_and_policy(**kwargs):
 
 @pytest.fixture
 def firehose_create_delivery_stream(wait_for_delivery_stream_ready, aws_client):
-    delivery_streams = {}
+    delivery_stream_names = []
 
     def _create_delivery_stream(**kwargs):
         if "DeliveryStreamName" not in kwargs:
             kwargs["DeliveryStreamName"] = f"test-delivery-stream-{short_uid()}"
-
-        delivery_stream = aws_client.firehose.create_delivery_stream(**kwargs)
-        delivery_streams.update({kwargs["DeliveryStreamName"]: delivery_stream})
-        wait_for_delivery_stream_ready(kwargs["DeliveryStreamName"])
-        return delivery_stream
+        delivery_stream_name = kwargs["DeliveryStreamName"]
+        response = aws_client.firehose.create_delivery_stream(**kwargs)
+        delivery_stream_names.append(delivery_stream_name)
+        wait_for_delivery_stream_ready(delivery_stream_name)
+        return response
 
     yield _create_delivery_stream
 
-    for delivery_stream_name in delivery_streams.keys():
-        aws_client.firehose.delete_delivery_stream(DeliveryStreamName=delivery_stream_name)
+    for delivery_stream_name in delivery_stream_names:
+        try:
+            aws_client.firehose.delete_delivery_stream(DeliveryStreamName=delivery_stream_name)
+        except Exception:
+            LOG.info("Failed to delete delivery stream %s", delivery_stream_name)
 
 
 @pytest.fixture
diff --git a/tests/aws/services/firehose/conftest.py b/tests/aws/services/firehose/conftest.py
new file mode 100644
--- /dev/null
+++ b/tests/aws/services/firehose/conftest.py
@@ -0,0 +1,95 @@
+import logging
+from typing import List, Literal, Union
+
+import pytest
+
+from localstack.utils.sync import retry
+
+StreamType = Literal["DirectPut", "KinesisStreamAsSource", "MSKAsSource"]
+
+LOG = logging.getLogger(__name__)
+
+
+@pytest.fixture
+def read_s3_data(aws_client):
+    s3 = aws_client.s3
+
+    def _read_s3_data(bucket_name: str, timeout: int = 10) -> dict[str, str]:
+        def _get_data():
+            response = s3.list_objects(Bucket=bucket_name)
+            if response.get("Contents") is None:
+                raise Exception("No data in bucket yet")
+
+            keys = [obj.get("Key") for obj in response.get("Contents")]
+
+            bucket_data = dict()
+            for key in keys:
+                response = s3.get_object(Bucket=bucket_name, Key=key)
+                data = response["Body"].read().decode("utf-8")
+                bucket_data[key] = data
+            return bucket_data
+
+        bucket_data = retry(_get_data, sleep=1, retries=timeout)
+
+        return bucket_data
+
+    return _read_s3_data
+
+
+def get_firehose_iam_documents(
+    bucket_arns: Union[List[str], str], stream_arns: Union[List[str], str]
+) -> tuple[dict, dict]:
+    """
+    Generate the required IAM role and policy documents for Firehose.
+    """
+    if isinstance(bucket_arns, str):
+        bucket_arns = [bucket_arns]
+    bucket_arns.extend([f"{bucket_arn}/*" for bucket_arn in bucket_arns])
+    if isinstance(stream_arns, str):
+        stream_arns = [stream_arns]
+
+    role_document = {
+        "Version": "2012-10-17",
+        "Statement": [
+            {
+                "Sid": "FirehoseAssumeRole",
+                "Effect": "Allow",
+                "Principal": {"Service": "firehose.amazonaws.com"},
+                "Action": "sts:AssumeRole",
+            }
+        ],
+    }
+
+    policy_document = {
+        "Version": "2012-10-17",
+        "Statement": [
+            {
+                "Effect": "Allow",
+                "Action": [
+                    "s3:AbortMultipartUpload",
+                    "s3:GetBucketLocation",
+                    "s3:GetObject",
+                    "s3:ListBucket",
+                    "s3:ListBucketMultipartUploads",
+                    "s3:PutObject",
+                ],
+                "Resource": bucket_arns,
+            },
+            {
+                "Effect": "Allow",
+                "Action": [
+                    "kinesis:DescribeStream",
+                    "kinesis:GetShardIterator",
+                    "kinesis:GetRecords",
+                    "kinesis:ListShards",
+                ],
+                "Resource": stream_arns,
+            },
+            {
+                "Effect": "Allow",
+                "Action": ["logs:PutLogEvents", "logs:CreateLogStream"],
+                "Resource": "arn:aws:logs:*:*:*",
+            },
+        ],
+    }
+    return role_document, policy_document
diff --git a/tests/aws/services/firehose/test_firehose.py b/tests/aws/services/firehose/test_firehose.py
--- a/tests/aws/services/firehose/test_firehose.py
+++ b/tests/aws/services/firehose/test_firehose.py
@@ -1,5 +1,6 @@
 import base64
 import json
+import time
 
 import pytest as pytest
 import requests
@@ -7,11 +8,14 @@
 
 from localstack import config
 from localstack.constants import TEST_AWS_ACCOUNT_ID
+from localstack.testing.aws.util import is_aws_cloud
 from localstack.testing.pytest import markers
 from localstack.utils.aws import arns
 from localstack.utils.strings import short_uid, to_bytes, to_str
 from localstack.utils.sync import poll_condition, retry
 
+from .conftest import get_firehose_iam_documents
+
 PROCESSOR_LAMBDA = """
 def handler(event, context):
     import base64
@@ -31,11 +35,17 @@ def handler(event, context):
     return {"records": records}
 """
 
+TEST_MESSAGE = "Test-message-2948294kdlsie"
+
 
 @pytest.mark.parametrize("lambda_processor_enabled", [True, False])
 @markers.aws.unknown
-def test_firehose_http(
-    aws_client, lambda_processor_enabled: bool, create_lambda_function, httpserver: HTTPServer
+def test_kinesis_firehose_http(
+    aws_client,
+    lambda_processor_enabled: bool,
+    create_lambda_function,
+    httpserver: HTTPServer,
+    cleanups,
 ):
     httpserver.expect_request("").respond_with_data(b"", 200)
     http_endpoint = httpserver.url_for("/")
@@ -47,7 +57,6 @@ def test_firehose_http(
         ]["FunctionArn"]
 
     # define firehose configs
-    # records = []
     http_destination_update = {
         "EndpointConfiguration": {"Url": http_endpoint, "Name": "test_update"}
     }
@@ -87,6 +96,8 @@ def test_firehose_http(
         HttpEndpointDestinationConfiguration=http_destination,
     )
     assert stream
+    cleanups.append(lambda: firehose.delete_delivery_stream(DeliveryStreamName=stream_name))
+
     stream_description = firehose.describe_delivery_stream(DeliveryStreamName=stream_name)
     stream_description = stream_description["DeliveryStreamDescription"]
     destination_description = stream_description["Destinations"][0][
@@ -123,10 +134,6 @@ def test_firehose_http(
     ]
     assert destination_description["EndpointConfiguration"]["Name"] == "test_update"
 
-    # delete stream
-    stream = firehose.delete_delivery_stream(DeliveryStreamName=stream_name)
-    assert stream["ResponseMetadata"]["HTTPStatusCode"] == 200
-
 
 class TestFirehoseIntegration:
     @markers.skip_offline
@@ -347,7 +354,7 @@ def assert_s3_contents():
             aws_client.opensearch.delete_domain(DomainName=domain_name)
 
     @markers.aws.unknown
-    def test_delivery_stream_with_kinesis_as_source(
+    def test_kinesis_firehose_kinesis_as_source(
         self, s3_bucket, kinesis_create_stream, cleanups, aws_client
     ):
         bucket_arn = arns.s3_bucket_arn(s3_bucket)
@@ -413,3 +420,110 @@ def check_stream_state():
             return stream["DeliveryStreamDescription"]["DeliveryStreamStatus"] == "ACTIVE"
 
         assert poll_condition(check_stream_state, 45, 1)
+
+    @markers.aws.validated
+    def test_kinesis_firehose_kinesis_as_source_multiple_delivery_streams(
+        self,
+        s3_create_bucket,
+        kinesis_create_stream,
+        create_iam_role_with_policy,
+        wait_for_stream_ready,
+        firehose_create_delivery_stream,
+        read_s3_data,
+        snapshot,
+        aws_client,
+    ):
+        # create s3 bucket a and b
+        bucket_a_name = f"test-bucket-a-{short_uid()}"
+        s3_create_bucket(Bucket=bucket_a_name)
+        bucket_a_arn = arns.s3_bucket_arn(bucket_a_name)
+        bucket_b_name = f"test-bucket-b-{short_uid()}"
+        s3_create_bucket(Bucket=bucket_b_name)
+        bucket_b_arn = arns.s3_bucket_arn(bucket_b_name)
+
+        # create kinesis stream
+        stream_name = f"test-stream-{short_uid()}"
+        kinesis_create_stream(
+            StreamName=stream_name,
+            ShardCount=1,
+            StreamModeDetails={"StreamMode": "PROVISIONED"},
+        )
+        stream_arn = aws_client.kinesis.describe_stream(StreamName=stream_name)[
+            "StreamDescription"
+        ]["StreamARN"]
+
+        # create IAM role and policy
+        role_document, policy_document = get_firehose_iam_documents(
+            [bucket_a_arn, bucket_b_arn], stream_arn
+        )
+        role_arn = create_iam_role_with_policy(
+            RoleDefinition=role_document, PolicyDefinition=policy_document
+        )
+        # required for role propagation delay on aws
+        if is_aws_cloud():
+            time.sleep(10)
+        wait_for_stream_ready(stream_name)
+
+        # create log groupe for firehose delivery stream error logging
+        log_group_name = f"group-{short_uid()}"
+        aws_client.logs.create_log_group(
+            logGroupName=log_group_name,
+        )
+
+        # create firehose streams & subscribe to kinesis
+        delivery_stream_a_name = f"test-delivery-stream-a-{short_uid()}"
+        delivery_stream_b_name = f"test-delivery-stream-b-{short_uid()}"
+
+        for bucket_arn, delivery_stream_name in [
+            (bucket_a_arn, delivery_stream_a_name),
+            (bucket_b_arn, delivery_stream_b_name),
+        ]:
+            extended_s3_destination_configuration = {
+                "RoleARN": role_arn,
+                "BucketARN": bucket_arn,
+                "Prefix": "firehoseTest",
+                "ErrorOutputPrefix": "firehoseTest-errors/!{firehose:error-output-type}/",
+                "BufferingHints": {"SizeInMBs": 1, "IntervalInSeconds": 1},
+                "CompressionFormat": "UNCOMPRESSED",
+                "EncryptionConfiguration": {"NoEncryptionConfig": "NoEncryption"},
+                "CloudWatchLoggingOptions": {
+                    "Enabled": True,
+                    "LogGroupName": log_group_name,
+                    "LogStreamName": f"stream-{short_uid()}",
+                },
+            }
+
+            firehose_create_delivery_stream(
+                DeliveryStreamName=delivery_stream_name,
+                DeliveryStreamType="KinesisStreamAsSource",
+                KinesisStreamSourceConfiguration={
+                    "KinesisStreamARN": stream_arn,
+                    "RoleARN": role_arn,
+                },
+                ExtendedS3DestinationConfiguration=extended_s3_destination_configuration,
+            )
+
+        # put message to kinesis event stream
+        record_data = TEST_MESSAGE
+        aws_client.kinesis.put_record(
+            StreamName=stream_name,
+            Data=record_data,
+            PartitionKey="1",
+        )
+
+        # poll file from s3 buckets
+        s3_data = dict()
+        for bucket_name in [bucket_a_name, bucket_b_name]:
+            s3_data_bucket = read_s3_data(bucket_name, timeout=300)
+            assert len(s3_data_bucket.keys()) == 1
+            assert record_data == next(iter(s3_data_bucket.values()))
+            s3_data_bucket = {"folder-name": s3_data_bucket.popitem()[1]}
+            s3_data[bucket_name] = s3_data_bucket
+
+        snapshot.add_transformer(
+            [
+                snapshot.transform.regex(bucket_a_name, "<bucket-a>"),
+                snapshot.transform.regex(bucket_b_name, "<bucket-b>"),
+            ]
+        )
+        snapshot.match("kinesis-event-stream-multiple-delivery-streams", s3_data)
diff --git a/tests/aws/services/firehose/test_firehose.snapshot.json b/tests/aws/services/firehose/test_firehose.snapshot.json
new file mode 100644
--- /dev/null
+++ b/tests/aws/services/firehose/test_firehose.snapshot.json
@@ -0,0 +1,15 @@
+{
+  "tests/aws/services/firehose/test_firehose.py::TestFirehoseIntegration::test_kinesis_firehose_kinesis_as_source_multiple_delivery_streams": {
+    "recorded-date": "25-01-2024, 10:54:39",
+    "recorded-content": {
+      "kinesis-event-stream-multiple-delivery-streams": {
+        "<bucket-a>": {
+          "folder-name": "Test-message-2948294kdlsie"
+        },
+        "<bucket-b>": {
+          "folder-name": "Test-message-2948294kdlsie"
+        }
+      }
+    }
+  }
+}
diff --git a/tests/aws/services/firehose/test_firehose.validation.json b/tests/aws/services/firehose/test_firehose.validation.json
new file mode 100644
--- /dev/null
+++ b/tests/aws/services/firehose/test_firehose.validation.json
@@ -0,0 +1,5 @@
+{
+  "tests/aws/services/firehose/test_firehose.py::TestFirehoseIntegration::test_multiple_delivery_streams_with_kinesis_as_source": {
+    "last_validated_date": "2024-01-25T10:54:39+00:00"
+  }
+}