Repository: getmoto/moto

Issue:
Moto doesn't implement default limits on dynamoDB return
According to the [AWS docs](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html) a DynamoDB scan will return data up to the maximum data size limit (1MB) or to the limit if it's set. Currently Moto doesn't do this and returns everything or the limit if set. There are two primary solutions for adding this feature.
1) A quick and dirty fix is to set a default limit. 
2) The correct way is to check the size and return up to 1MB but this can get quite complicated but doable. We could check the sizes of the items as they are being inserted and store the size with the item somehow or check the sizes of the items within the trim method and once we reach the 1 MB limit return.

I have done 1 already but have a few questions and still need to write tests. I could probably do the second option but it would take a big longer.

[Here](https://github.com/shade34321/moto/blob/implement_scan_limits/moto/dynamodb2/models.py#L674-L676) is my rudimentary implementation of the first option. It sets a default value of 1K for the limit and if you try and set the limit to something greater than that it will set it back to 1K. This will mock AWS's behavior of returning a max of "1 MB" regardless of limit. If you're ok with this I can add the tests and put up the PR for this.

I did have some questions around what some of these other scan functions are doing specifically the two below.
https://github.com/shade34321/moto/blob/implement_scan_limits/moto/dynamodb2/models.py#L895-L910
https://github.com/shade34321/moto/blob/implement_scan_limits/moto/dynamodb2/responses.py#L487-L537 

Thanks!


Discussion:
Thanks; setting as a feature request.

Resolving patch:
diff --git a/moto/dynamodb2/models/__init__.py b/moto/dynamodb2/models/__init__.py
--- a/moto/dynamodb2/models/__init__.py
+++ b/moto/dynamodb2/models/__init__.py
@@ -74,6 +74,9 @@ def __init__(self, hash_key, hash_key_type, range_key, range_key_type, attrs):
     def __repr__(self):
         return "Item: {0}".format(self.to_json())
 
+    def size(self):
+        return sum(bytesize(key) + value.size() for key, value in self.attrs.items())
+
     def to_json(self):
         attributes = {}
         for attribute_key, attribute in self.attrs.items():
@@ -921,6 +924,14 @@ def _trim_results(self, results, limit, exclusive_start_key, scanned_index=None)
                     break
 
         last_evaluated_key = None
+        size_limit = 1000000  # DynamoDB has a 1MB size limit
+        item_size = sum(res.size() for res in results)
+        if item_size > size_limit:
+            item_size = idx = 0
+            while item_size + results[idx].size() < size_limit:
+                item_size += results[idx].size()
+                idx += 1
+            limit = min(limit, idx) if limit else idx
         if limit and len(results) > limit:
             results = results[:limit]
             last_evaluated_key = {self.hash_key_attr: results[-1].hash_key}


Test patch:
diff --git a/tests/test_dynamodb2/test_dynamodb.py b/tests/test_dynamodb2/test_dynamodb.py
--- a/tests/test_dynamodb2/test_dynamodb.py
+++ b/tests/test_dynamodb2/test_dynamodb.py
@@ -4219,6 +4219,44 @@ def test_gsi_verify_negative_number_order():
     )
 
 
+@mock_dynamodb2
+def test_dynamodb_max_1mb_limit():
+    ddb = boto3.resource("dynamodb", region_name="eu-west-1")
+
+    table_name = "populated-mock-table"
+    table = ddb.create_table(
+        TableName=table_name,
+        KeySchema=[
+            {"AttributeName": "partition_key", "KeyType": "HASH"},
+            {"AttributeName": "sort_key", "KeyType": "RANGE"},
+        ],
+        AttributeDefinitions=[
+            {"AttributeName": "partition_key", "AttributeType": "S"},
+            {"AttributeName": "sort_key", "AttributeType": "S"},
+        ],
+        BillingMode="PAY_PER_REQUEST",
+    )
+
+    # Populate the table
+    items = [
+        {
+            "partition_key": "partition_key_val",  # size=30
+            "sort_key": "sort_key_value____" + str(i),  # size=30
+        }
+        for i in range(10000, 29999)
+    ]
+    with table.batch_writer() as batch:
+        for item in items:
+            batch.put_item(Item=item)
+
+    response = table.query(
+        KeyConditionExpression=Key("partition_key").eq("partition_key_val")
+    )
+    # We shouldn't get everything back - the total result set is well over 1MB
+    len(items).should.be.greater_than(response["Count"])
+    response["LastEvaluatedKey"].shouldnt.be(None)
+
+
 def assert_raise_syntax_error(client_error, token, near):
     """
     Assert whether a client_error is as expected Syntax error. Syntax error looks like: `syntax_error_template`