Repository: localstack/localstack

Issue:
bug: dynamodb batch_writer no longer supports binary/bytes data
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Docker container releases _after_ localstack/localstack:0.14.2 no longer appear to be support Binary or bytes data in dynamodb objects using batch_writer fails with botocore error. Possibly related to this change https://github.com/localstack/localstack/commit/3c6446584192beb1c96468c44690f421dcb784b8

The issue appears to be because the new dynamo provider is trying to use json.dumps on the record with bytes causing a serialization error.

The same operations work in the localstack:0.14.2 docker image and prior without issue.

### Expected Behavior

_No response_

### How are you starting LocalStack?

With the `localstack` script

### Steps To Reproduce

localstack start

```
        import boto3
        import os
        from boto3.dynamodb.types import Binary

        os.environ["AWS_ACCESS_KEY_ID"] = "test"
        os.environ["AWS_SECRET_ACCESS_KEY"] = "test"

        dynamodb = boto3.client('dynamodb', endpoint_url='http://localhost:4566')

        dynamodb.create_table(
            TableName='BatchWriteTest',
            AttributeDefinitions=[
                {'AttributeName': 'PK', 'AttributeType': 'S'},
                {'AttributeName': 'SK', 'AttributeType': 'S'},
            ],
            KeySchema=[
                {'AttributeName': 'PK', 'KeyType': 'HASH'},
                {'AttributeName': 'SK', 'KeyType': 'RANGE'},
            ],
            BillingMode='PAY_PER_REQUEST',
        )
        dynamodb.get_waiter('table_exists').wait(TableName='BatchWriteTest')
        # This works
        dynamodb.put_item(TableName="BatchWriteTest", Item={'PK': {"S": 'hello'}, 'SK': {"S": 'user'}, 'data': {"B": b"test"}})
        print(dynamodb.get_item(TableName="BatchWriteTest", Key={"PK": {"S": "hello"}, "SK": {"S": "user"}}))

        batch_write_test_table = boto3.resource('dynamodb', endpoint_url='http://localhost:4566').Table('BatchWriteTest')

        with batch_write_test_table.batch_writer() as batch:
            for no in range(1, 6):
                # Fails with botocore.exceptions.ClientError: An error occurred (500) when calling the BatchWriteItem operation (reached max retries: 9
                batch.put_item(Item={'PK': f'hello{no}', 'SK': f'user-{no}', 'data': b"test"})

        print(dynamodb.get_item(TableName="BatchWriteTest", Key={"PK": {"S": "hello1"}, "SK": {"S": "user-1"}}))
```


### Environment

```markdown
- OS: Mac 12.3
- LocalStack: latest
```


### Anything else?

Traceback in localstack console

```
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack_ext/services/edge.py.enc", line 128, in do_forward_request
File "/opt/code/localstack/localstack/services/edge.py", line 225, in do_forward_request
result = do_forward_request_inmem(api, method, path, data, headers, port=port)
File "/opt/code/localstack/localstack/services/edge.py", line 249, in do_forward_request_inmem
response = modify_and_forward(
File "/opt/code/localstack/localstack/services/generic_proxy.py", line 514, in wrapper
value = func(*args, **kwargs)
File "/opt/code/localstack/localstack/services/generic_proxy.py", line 594, in modify_and_forward
listener_result = listener.forward_request(
File "/opt/code/localstack/localstack/services/dynamodb/provider.py", line 296, in forward_request
return super().forward_request(method, path, data, headers)
File "/opt/code/localstack/localstack/http/adapters.py", line 51, in forward_request
response = self.request(request)
File "/opt/code/localstack/localstack/aws/proxy.py", line 42, in request
response = self.skeleton.invoke(context)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 153, in invoke
return self.dispatch_request(context, instance)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 165, in dispatch_request
result = handler(context, instance) or {}
File "/opt/code/localstack/localstack/aws/forwarder.py", line 57, in _call
return handler(context, req)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 117, in __call__
return self.fn(*args, **kwargs)
File "/opt/code/localstack/localstack/aws/api/core.py", line 95, in operation_marker
return fn(*args, **kwargs)
File "/opt/code/localstack/localstack/services/dynamodb/provider.py", line 700, in batch_write_item
records, unprocessed_items = self.prepare_batch_write_item_records(
File "/opt/code/localstack/localstack/services/dynamodb/provider.py", line 1179, in prepare_batch_write_item_records
new_record["dynamodb"]["SizeBytes"] = len(json.dumps(put_request["Item"]))
File "/usr/local/lib/python3.10/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/local/lib/python3.10/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.10/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.10/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable
```
bug: dynamodb batch_writer no longer supports binary/bytes data
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Docker container releases _after_ localstack/localstack:0.14.2 no longer appear to be support Binary or bytes data in dynamodb objects using batch_writer fails with botocore error. Possibly related to this change https://github.com/localstack/localstack/commit/3c6446584192beb1c96468c44690f421dcb784b8

The issue appears to be because the new dynamo provider is trying to use json.dumps on the record with bytes causing a serialization error.

The same operations work in the localstack:0.14.2 docker image and prior without issue.

### Expected Behavior

_No response_

### How are you starting LocalStack?

With the `localstack` script

### Steps To Reproduce

localstack start

```
        import boto3
        import os
        from boto3.dynamodb.types import Binary

        os.environ["AWS_ACCESS_KEY_ID"] = "test"
        os.environ["AWS_SECRET_ACCESS_KEY"] = "test"

        dynamodb = boto3.client('dynamodb', endpoint_url='http://localhost:4566')

        dynamodb.create_table(
            TableName='BatchWriteTest',
            AttributeDefinitions=[
                {'AttributeName': 'PK', 'AttributeType': 'S'},
                {'AttributeName': 'SK', 'AttributeType': 'S'},
            ],
            KeySchema=[
                {'AttributeName': 'PK', 'KeyType': 'HASH'},
                {'AttributeName': 'SK', 'KeyType': 'RANGE'},
            ],
            BillingMode='PAY_PER_REQUEST',
        )
        dynamodb.get_waiter('table_exists').wait(TableName='BatchWriteTest')
        # This works
        dynamodb.put_item(TableName="BatchWriteTest", Item={'PK': {"S": 'hello'}, 'SK': {"S": 'user'}, 'data': {"B": b"test"}})
        print(dynamodb.get_item(TableName="BatchWriteTest", Key={"PK": {"S": "hello"}, "SK": {"S": "user"}}))

        batch_write_test_table = boto3.resource('dynamodb', endpoint_url='http://localhost:4566').Table('BatchWriteTest')

        with batch_write_test_table.batch_writer() as batch:
            for no in range(1, 6):
                # Fails with botocore.exceptions.ClientError: An error occurred (500) when calling the BatchWriteItem operation (reached max retries: 9
                batch.put_item(Item={'PK': f'hello{no}', 'SK': f'user-{no}', 'data': b"test"})

        print(dynamodb.get_item(TableName="BatchWriteTest", Key={"PK": {"S": "hello1"}, "SK": {"S": "user-1"}}))
```


### Environment

```markdown
- OS: Mac 12.3
- LocalStack: latest
```


### Anything else?

Traceback in localstack console

```
File "/opt/code/localstack/.venv/lib/python3.10/site-packages/localstack_ext/services/edge.py.enc", line 128, in do_forward_request
File "/opt/code/localstack/localstack/services/edge.py", line 225, in do_forward_request
result = do_forward_request_inmem(api, method, path, data, headers, port=port)
File "/opt/code/localstack/localstack/services/edge.py", line 249, in do_forward_request_inmem
response = modify_and_forward(
File "/opt/code/localstack/localstack/services/generic_proxy.py", line 514, in wrapper
value = func(*args, **kwargs)
File "/opt/code/localstack/localstack/services/generic_proxy.py", line 594, in modify_and_forward
listener_result = listener.forward_request(
File "/opt/code/localstack/localstack/services/dynamodb/provider.py", line 296, in forward_request
return super().forward_request(method, path, data, headers)
File "/opt/code/localstack/localstack/http/adapters.py", line 51, in forward_request
response = self.request(request)
File "/opt/code/localstack/localstack/aws/proxy.py", line 42, in request
response = self.skeleton.invoke(context)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 153, in invoke
return self.dispatch_request(context, instance)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 165, in dispatch_request
result = handler(context, instance) or {}
File "/opt/code/localstack/localstack/aws/forwarder.py", line 57, in _call
return handler(context, req)
File "/opt/code/localstack/localstack/aws/skeleton.py", line 117, in __call__
return self.fn(*args, **kwargs)
File "/opt/code/localstack/localstack/aws/api/core.py", line 95, in operation_marker
return fn(*args, **kwargs)
File "/opt/code/localstack/localstack/services/dynamodb/provider.py", line 700, in batch_write_item
records, unprocessed_items = self.prepare_batch_write_item_records(
File "/opt/code/localstack/localstack/services/dynamodb/provider.py", line 1179, in prepare_batch_write_item_records
new_record["dynamodb"]["SizeBytes"] = len(json.dumps(put_request["Item"]))
File "/usr/local/lib/python3.10/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/local/lib/python3.10/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.10/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.10/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable
```


Discussion:

Resolving patch:
diff --git a/localstack/services/dynamodb/provider.py b/localstack/services/dynamodb/provider.py
--- a/localstack/services/dynamodb/provider.py
+++ b/localstack/services/dynamodb/provider.py
@@ -1176,7 +1176,7 @@ def prepare_batch_write_item_records(
                         )
                         new_record = copy.deepcopy(record)
                         new_record["eventID"] = short_uid()
-                        new_record["dynamodb"]["SizeBytes"] = len(json.dumps(put_request["Item"]))
+                        new_record["dynamodb"]["SizeBytes"] = _get_size_bytes(put_request["Item"])
                         new_record["eventName"] = "INSERT" if not existing_item else "MODIFY"
                         new_record["dynamodb"]["Keys"] = keys
                         new_record["dynamodb"]["NewImage"] = put_request["Item"]
@@ -1199,7 +1199,7 @@ def prepare_batch_write_item_records(
                         new_record["eventName"] = "REMOVE"
                         new_record["dynamodb"]["Keys"] = keys
                         new_record["dynamodb"]["OldImage"] = existing_items[i]
-                        new_record["dynamodb"]["SizeBytes"] = len(json.dumps(existing_items[i]))
+                        new_record["dynamodb"]["SizeBytes"] = _get_size_bytes(existing_items[i])
                         new_record["eventSourceARN"] = aws_stack.dynamodb_table_arn(table_name)
                         records.append(new_record)
                     if unprocessed_delete_items and len(unprocessed_delete_items) > i:
@@ -1269,6 +1269,12 @@ def should_throttle(self, action):
 # ---
 # Misc. util functions
 # ---
+def _get_size_bytes(item) -> int:
+    try:
+        size_bytes = len(json.dumps(item))
+    except TypeError:
+        size_bytes = len(str(item))
+    return size_bytes
 
 
 def get_global_secondary_index(table_name, index_name):
@@ -1368,7 +1374,7 @@ def _add_record(item, comparison_set: ItemSet):
             "dynamodb": {
                 "Keys": keys,
                 "NewImage": new_image,
-                "SizeBytes": len(json.dumps(item)),
+                "SizeBytes": _get_size_bytes(item),
             },
         }
         if stream_spec:


Test patch:
diff --git a/tests/integration/test_dynamodb.py b/tests/integration/test_dynamodb.py
--- a/tests/integration/test_dynamodb.py
+++ b/tests/integration/test_dynamodb.py
@@ -462,6 +462,45 @@ def test_empty_and_binary_values(self, dynamodb):
         response = table.put_item(Item=item2)
         assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
 
+    def test_batch_write_binary(self, dynamodb_client):
+        table_name = "table_batch_binary_%s" % short_uid()
+        dynamodb_client.create_table(
+            TableName=table_name,
+            AttributeDefinitions=[
+                {"AttributeName": "PK", "AttributeType": "S"},
+                {"AttributeName": "SK", "AttributeType": "S"},
+            ],
+            KeySchema=[
+                {"AttributeName": "PK", "KeyType": "HASH"},
+                {"AttributeName": "SK", "KeyType": "RANGE"},
+            ],
+            BillingMode="PAY_PER_REQUEST",
+        )
+        dynamodb_client.put_item(
+            TableName=table_name,
+            Item={"PK": {"S": "hello"}, "SK": {"S": "user"}, "data": {"B": b"test"}},
+        )
+
+        item = {
+            "Item": {
+                "PK": {"S": "hello-1"},
+                "SK": {"S": "user-1"},
+                "data": {"B": b"test-1"},
+            }
+        }
+        item_non_decodable = {
+            "Item": {
+                "PK": {"S": "hello-2"},
+                "SK": {"S": "user-2"},
+                "data": {"B": b"test \xc0 \xed"},
+            }
+        }
+        response = dynamodb_client.batch_write_item(
+            RequestItems={table_name: [{"PutRequest": item}, {"PutRequest": item_non_decodable}]}
+        )
+        assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
+        dynamodb_client.delete_table(TableName=table_name)
+
     def test_dynamodb_stream_shard_iterator(self):
         def wait_for_stream_created(table_name):
             stream_name = get_kinesis_stream_name(table_name)