Repository: localstack/localstack

Issue:
Kinesis stream records do not offer AWS-defined optional values
<!-- Love localstack? Please consider supporting our collective:
👉  https://opencollective.com/localstack/donate -->

# Type of request: This is a ...

[ ] bug report
[x] feature request

# Detailed description

Kinesis does not populate/offer to populate all potential values in a record. I have a lambda being triggered by kinesis events. The records being passed in are only those required by AWS, but there are no optional values being used.

## Expected behavior

An event's kinesis object should be configurable to look as follows
```
{ 
    "ApproximateArrivalTimestamp": number,
    "Data": blob,
    "EncryptionType": "string",
    "PartitionKey": "string",
    "SequenceNumber": "string"
}
```

## Actual behavior

In its current state, only required values are present.
```
{ 
    "Data": blob,
    "PartitionKey": "string",
    "SequenceNumber": "string"
}
```

# Steps to reproduce

## Command used to start LocalStack

`docker-compose up` with given `docker-compose.yml`

## Client code (AWS SDK code snippet, or sequence of "awslocal" commands)

The source of this issue/feature request lies in the kinesis listener. AWS documentation of a kinesis stream record lists all values here: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_Record.html

We are only returning the required values.
https://github.com/localstack/localstack/blob/78031dd65da9394f8b1b020be01ef02c63c433ee/localstack/services/kinesis/kinesis_listener.py#L85-L94

If there is no implicit reason for not offering optional values, I'm more than happy to submit a PR.



Discussion:
Thanks for the detailed report @payton - would be great if you could submit a PR for the missing fields. Thanks, much appreciated.

Resolving patch:
diff --git a/localstack/services/kinesis/kinesis_listener.py b/localstack/services/kinesis/kinesis_listener.py
--- a/localstack/services/kinesis/kinesis_listener.py
+++ b/localstack/services/kinesis/kinesis_listener.py
@@ -1,9 +1,8 @@
 import json
 import random
-from datetime import datetime
 from requests.models import Response
 from localstack import config
-from localstack.utils.common import to_str, json_safe, clone
+from localstack.utils.common import to_str, json_safe, clone, timestamp_millis
 from localstack.utils.analytics import event_publisher
 from localstack.services.awslambda import lambda_api
 from localstack.services.generic_proxy import ProxyListener
@@ -32,7 +31,7 @@ def forward_request(self, method, path, data, headers):
             consumer = clone(data)
             consumer['ConsumerStatus'] = 'ACTIVE'
             consumer['ConsumerARN'] = '%s/consumer/%s' % (data['StreamARN'], data['ConsumerName'])
-            consumer['ConsumerCreationTimestamp'] = datetime.now()
+            consumer['ConsumerCreationTimestamp'] = timestamp_millis()
             consumer = json_safe(consumer)
             STREAM_CONSUMERS.append(consumer)
             return {'Consumer': consumer}
@@ -85,7 +84,9 @@ def return_response(self, method, path, data, headers, response):
         elif action == ACTION_PUT_RECORD:
             response_body = json.loads(to_str(response.content))
             event_record = {
+                'approximateArrivalTimestamp': timestamp_millis(),
                 'data': data['Data'],
+                'encryptionType': 'NONE',
                 'partitionKey': data['PartitionKey'],
                 'sequenceNumber': response_body.get('SequenceNumber')
             }
@@ -101,7 +102,9 @@ def return_response(self, method, path, data, headers, response):
                 for index in range(0, len(records)):
                     record = records[index]
                     event_record = {
+                        'approximateArrivalTimestamp': timestamp_millis(),
                         'data': record['Data'],
+                        'encryptionType': 'NONE',
                         'partitionKey': record['PartitionKey'],
                         'sequenceNumber': response_records[index].get('SequenceNumber')
                     }


Test patch: