Repository: aws-powertools/powertools-lambda-python

Issue:
Feature request: KinesisDataStreamEnvelope().parse() method seems to be missing data decompression step
### Expected Behaviour

When provided with an event and a Pydantic model, the envelope's `parse()` method should correctly parse the data, including decompression of it (as as far as I am aware, it is compressed by the Kinesis itself).

### Current Behaviour

My Lambda function receives CloudWatch Logs via a Kinesis Data Stream. When using event parser with the `KinesisDataStreamEnvelope` either as a annotation above `lambda_handler` or explicitly calling envelope's `parse` method, I keep getting a `UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte`.

After debugging it appears as if there is a decompression step missing in the envelope's `parse` method between casting to bytes and decoding using 'utf-8'.


### Code snippet

```python
# Pydantic models for CloudWatch Logs representation
class CloudWatchLogEvent(BaseModel):
  model_config = ConfigDict(alias_generator=to_camel)

 id: str
 timestamp: str | int
 message: str


class CloudWatchLog(BaseModel):
  model_config = ConfigDict(alias_generator=to_camel)

  message_type: str
  owner: str = Field(pattern=r"^\d{12}$")  # 12 digits pattern
  log_group: str
  log_stream: str
  subscription_filter: list[str]
  log_events: list[CloudWatchLogEvent] = Field(min_length=1)


# Version 1 - not working
@event_parser(model=CloudWatchLog, envelope=envelopes.KinesisDataStreamEnvelope)
def lambda_handler(event: list[CloudWatchLog], context: LambdaContext) -> dict:
  ...


# Version 2 - also not working
def lambda_handler(event: dict, context: LambdaContext) -> dict:
log: list[MyLogModel] = envelopes.KinesisDataStreamEnvelope().parse(
  event, CloudWatchLog
)
...
```

### Possible Solution

When manually parsing Kinesis Data Stream input in a Lambda function and locally (using a saved event), the following code works:
```
record_data: bytes = base64.b64decode(record['kinesis']['data'])
decompressed_data: bytes = gzip.decompress(record_data)
decoded_data: str = record_data_decode('utf-8')
```

### Fixing the envelope
To fix the envelope itself, importing `gzip` and adding a line:
`data = gzip.decompress(data)`
in **_utilities/parser/envelopes/kinesis.py_** file between casting to bytes and decoding using 'utf-8' fixes the issue.

Since this could be an edge case, it should also be fine to wrap the `models.append(self._parse(data=data.decode('utf-8')...` line in a `try`/`except` clause catching the `UnicodeDecodeError` exception and then performing the decompression before trying again to decode using 'utf-8'.

### Steps to Reproduce

1. Have a Kinesis Data Stream moving CloudWatch Logs to a Lambda function (or have a saved event and use it locally)
2. Use the CloudWatchLog Pydantic model provided in the code snippet
3. Use the KinesisDataStream envelope with an event parser either as an annotation above the Lambda handler, or explicitly call the `parse` method

Thank you

### Powertools for AWS Lambda (Python) version

3.9.0

### AWS Lambda function runtime

3.12

### Packaging format used

PyPi

### Debugging logs

```python
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
```


Discussion:
Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: [Invite link](https://discord.gg/B8zZKbbyET)

Hi @Artur-T-Malas, thanks for opening this issue and reporting this! While I recognize that one of the possible use cases for Kinesis is ingesting CloudWatch logs, this is not the only use case. I say this because if we assume that we always need to base64 decode + gzip decompress and then return the value, we can break all use cases where customers use Kinesis to not parse CloudWatch logs and using plain text records.

That said, I agree that we should solve this problem and improve this experience, but we need a solution that is able to handle both cases: gziped and plain text.

If you have any solution, I'm more than happy to accept a PR, if not, I'll work this on the next week.

Thanks
Hi @leandrodamascena. I've created a [PR #6656](https://github.com/aws-powertools/powertools-lambda-python/pull/6656). To be honest it's my first PR to a project this big, so I'll greatly appreciate tips on what could be better. 

I hope that my solution with wrapping it in a `try`/`except` block and decompressing only after encountering a `UnicodeDecodeError` is acceptable.



Patch:
diff --git a/aws_lambda_powertools/utilities/parser/envelopes/kinesis.py b/aws_lambda_powertools/utilities/parser/envelopes/kinesis.py
index a4d484931df..41527e03930 100644
--- a/aws_lambda_powertools/utilities/parser/envelopes/kinesis.py
+++ b/aws_lambda_powertools/utilities/parser/envelopes/kinesis.py
@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import logging
+import zlib
 from typing import TYPE_CHECKING, Any, cast
 
 from aws_lambda_powertools.utilities.parser.envelopes.base import BaseEnvelope
@@ -45,5 +46,17 @@ def parse(self, data: dict[str, Any] | Any | None, model: type[Model]) -> list[M
         for record in parsed_envelope.Records:
             # We allow either AWS expected contract (bytes) or a custom Model, see #943
             data = cast(bytes, record.kinesis.data)
-            models.append(self._parse(data=data.decode("utf-8"), model=model))
+            try:
+                decoded_data = data.decode("utf-8")
+            except UnicodeDecodeError as ude:
+                try:
+                    logger.debug(
+                        f"{type(ude).__name__}: {str(ude)} encountered. "
+                        "Data will be decompressed with zlib.decompress().",
+                    )
+                    decompressed_data = zlib.decompress(data, zlib.MAX_WBITS | 32)
+                    decoded_data = decompressed_data.decode("utf-8")
+                except Exception as e:
+                    raise ValueError("Unable to decode and/or decompress data.") from e
+            models.append(self._parse(data=decoded_data, model=model))
         return models


Test patch:
diff --git a/tests/unit/parser/_pydantic/test_kinesis.py b/tests/unit/parser/_pydantic/test_kinesis.py
index 9da19ed3e0b..9c3e365c5c5 100644
--- a/tests/unit/parser/_pydantic/test_kinesis.py
+++ b/tests/unit/parser/_pydantic/test_kinesis.py
@@ -105,3 +105,28 @@ class DummyModel(BaseModel): ...
         for record in stream_data.Records:
             record.kinesis.data = DummyModel()
             record.decompress_zlib_record_data_as_json()
+
+
+def test_kinesis_stream_event_with_cloud_watch_logs_data_using_envelope():
+    # GIVEN Kinesis Data Stream event with compressed data
+    # such as CloudWatch Logs
+    raw_event = load_event("kinesisStreamCloudWatchLogsEvent.json")
+
+    # WHEN parsing using KinesisDataStreamEvelope to CloudWatchLogsDecode
+    logs = envelopes.KinesisDataStreamEnvelope().parse(raw_event, CloudWatchLogsDecode)
+
+    # THEN logs should be extracted as CloudWatchLogsDecode objects
+    assert isinstance(logs[0], CloudWatchLogsDecode)
+
+
+def test_kinesis_stream_event_with_cloud_watch_logs_data_fails_using_envelope():
+    # GIVEN Kinesis Data Stream event with corrupted compressed data
+    # such as CloudWatch Logs
+    raw_event = load_event("kinesisStreamCloudWatchLogsEvent.json")
+
+    # WHEN parsing using KinesisDataStreamEvelope to CloudWatchLogsDecode
+    # and the data is corrupted
+    raw_event["Records"][0]["kinesis"]["data"] = "eyJ4eXoiOiAiYWJjIn0KH4sIAK25JWgAA6tWqqisUrJSUEpMSlaq5QIAbdJPfw8AAAA="
+    # THEN a ValueError should be thrown
+    with pytest.raises(ValueError):
+        envelopes.KinesisDataStreamEnvelope().parse(raw_event, CloudWatchLogsDecode)