Repository: getmoto/moto
Pull request #8298 (2024-11-08 11:38:33)

DynamoDB: Validate duplicate path in projection expression
Hi, this is a proposed fix for issue #8297 

In case of duplicate paths in PorjectionExpression AWS raises an error below. Moto - does not.

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the Query operation: Invalid ProjectionExpression: Two document paths overlap with each other; must remove or rewrite one of these paths; path one: [body], path two: [body]

Appreciate your feedback a lot!

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/getmoto/moto/pull/8298?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 94.56%. Comparing base [(`60828f4`)](https://app.codecov.io/gh/getmoto/moto/commit/60828f4befc1da863dc4ad630409f6443145fc71?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto) to head [(`19c92fc`)](https://app.codecov.io/gh/getmoto/moto/commit/19c92fc992500c3c7a6961d8d1210f76949480b3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto).

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #8298      +/-   ##
==========================================
- Coverage   94.57%   94.56%   -0.01%     
==========================================
  Files        1158     1159       +1     
  Lines      100841   100854      +13     
==========================================
+ Hits        95366    95376      +10     
- Misses       5475     5478       +3     
```

| [Flag](https://app.codecov.io/gh/getmoto/moto/pull/8298/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto) | Coverage Δ | |
|---|---|---|
| [servertests](https://app.codecov.io/gh/getmoto/moto/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto) | `28.87% <46.66%> (+<0.01%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/getmoto/moto/pull/8298/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto) | `94.54% <100.00%> (-0.01%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/getmoto/moto/pull/8298?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto).


Patch:
diff --git a/moto/dynamodb/exceptions.py b/moto/dynamodb/exceptions.py
index 4d4d327e0..dc344f10d 100644
--- a/moto/dynamodb/exceptions.py
+++ b/moto/dynamodb/exceptions.py
@@ -305,6 +305,17 @@ class DuplicateUpdateExpression(InvalidUpdateExpression):
         )
 
 
+class InvalidProjectionExpression(MockValidationException):
+    msg = (
+        "Invalid ProjectionExpression: "
+        "Two document paths overlap with each other; must remove or rewrite one of these paths; "
+        "path one: [{paths[0]}], path two: [{paths[1]}]"
+    )
+
+    def __init__(self, paths: List[str]):
+        super().__init__(self.msg.format(paths=paths))
+
+
 class TooManyClauses(InvalidUpdateExpression):
     def __init__(self, _type: str) -> None:
         super().__init__(
diff --git a/moto/dynamodb/parsing/ast_nodes.py b/moto/dynamodb/parsing/ast_nodes.py
index 28985bf1a..75971f8ac 100644
--- a/moto/dynamodb/parsing/ast_nodes.py
+++ b/moto/dynamodb/parsing/ast_nodes.py
@@ -10,6 +10,7 @@ from ..exceptions import (
     MockValidationException,
     TooManyClauses,
 )
+from ..utils import extract_duplicates
 
 
 class Node(metaclass=abc.ABCMeta):
@@ -38,8 +39,11 @@ class Node(metaclass=abc.ABCMeta):
             set_attributes = [s.children[0].to_str() for s in set_actions]
             # We currently only check for duplicates
             # We should also check for partial duplicates, i.e. [attr, attr.sub] is also invalid
-            if len(set_attributes) != len(set(set_attributes)):
-                raise DuplicateUpdateExpression(set_attributes)
+            duplicates = extract_duplicates(set_attributes)
+            if duplicates:
+                # There might be more than one attribute duplicated:
+                # they may get mixed up in the Error Message which is inline with actual boto3 Error Messages
+                raise DuplicateUpdateExpression(duplicates)
 
             set_clauses = self.find_clauses([UpdateExpressionSetClause])
             if limit_set_actions and len(set_clauses) > 1:
diff --git a/moto/dynamodb/responses.py b/moto/dynamodb/responses.py
index a2c9c02f0..cb1f777f2 100644
--- a/moto/dynamodb/responses.py
+++ b/moto/dynamodb/responses.py
@@ -14,12 +14,14 @@ from moto.dynamodb.parsing.reserved_keywords import ReservedKeywords
 from moto.utilities.aws_headers import amz_crc32
 
 from .exceptions import (
+    InvalidProjectionExpression,
     KeyIsEmptyStringException,
     MockValidationException,
     ProvidedKeyDoesNotExist,
     ResourceNotFoundException,
     UnknownKeyType,
 )
+from .utils import extract_duplicates
 
 TRANSACTION_MAX_ITEMS = 25
 
@@ -794,6 +796,9 @@ class DynamoHandler(BaseResponse):
 
         if projection_expression:
             expressions = [x.strip() for x in projection_expression.split(",")]
+            duplicates = extract_duplicates(expressions)
+            if duplicates:
+                raise InvalidProjectionExpression(duplicates)
             for expression in expressions:
                 check_projection_expression(expression)
             return [
diff --git a/moto/dynamodb/utils.py b/moto/dynamodb/utils.py
new file mode 100644
index 000000000..f16a1e6f1
--- /dev/null
+++ b/moto/dynamodb/utils.py
@@ -0,0 +1,5 @@
+from typing import List
+
+
+def extract_duplicates(input_list: List[str]) -> List[str]:
+    return [item for item in input_list if input_list.count(item) > 1]
diff --git a/setup.cfg b/setup.cfg
index cef6c2d57..93f64f237 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -255,9 +255,6 @@ console_scripts =
     moto_server = moto.server:main
     moto_proxy = moto.proxy:main
 
-[bdist_wheel]
-universal=1
-
 [tool:pytest]
 markers =
     network: requires network connection


Test patch:
diff --git a/tests/test_dynamodb/exceptions/test_dynamodb_exceptions.py b/tests/test_dynamodb/exceptions/test_dynamodb_exceptions.py
index 0757efa11..15bfe899a 100644
--- a/tests/test_dynamodb/exceptions/test_dynamodb_exceptions.py
+++ b/tests/test_dynamodb/exceptions/test_dynamodb_exceptions.py
@@ -664,6 +664,148 @@ def test_update_item_with_duplicate_expressions(expression):
     assert item == {"pk": "example_id", "example_column": "example"}
 
 
+@mock_aws
+def test_query_item_with_duplicate_path_in_projection_expressions():
+    dynamodb = boto3.resource("dynamodb", region_name="us-east-1")
+
+    # Create the DynamoDB table.
+    dynamodb.create_table(
+        TableName="users",
+        KeySchema=[
+            {"AttributeName": "forum_name", "KeyType": "HASH"},
+            {"AttributeName": "subject", "KeyType": "RANGE"},
+        ],
+        AttributeDefinitions=[
+            {"AttributeName": "forum_name", "AttributeType": "S"},
+            {"AttributeName": "subject", "AttributeType": "S"},
+        ],
+        ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5},
+    )
+    table = dynamodb.Table("users")
+    table.put_item(
+        Item={"forum_name": "the-key", "subject": "123", "body": "some test message"}
+    )
+
+    with pytest.raises(ClientError) as exc:
+        _result = table.query(
+            KeyConditionExpression=Key("forum_name").eq("the-key"),
+            ProjectionExpression="body, subject, body",  # duplicate body
+        )["Items"][0]
+
+    err = exc.value.response["Error"]
+    assert err["Code"] == "ValidationException"
+    assert err["Message"] == (
+        "Invalid ProjectionExpression: "
+        "Two document paths overlap with each other; must remove or rewrite one of these paths; "
+        "path one: [body], path two: [body]"
+    )
+
+
+@mock_aws
+def test_get_item_with_duplicate_path_in_projection_expressions():
+    dynamodb = boto3.resource("dynamodb", region_name="us-east-1")
+
+    # Create the DynamoDB table.
+    table = dynamodb.create_table(
+        TableName="users",
+        KeySchema=[
+            {"AttributeName": "forum_name", "KeyType": "HASH"},
+            {"AttributeName": "subject", "KeyType": "RANGE"},
+        ],
+        AttributeDefinitions=[
+            {"AttributeName": "forum_name", "AttributeType": "S"},
+            {"AttributeName": "subject", "AttributeType": "S"},
+        ],
+        ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5},
+    )
+    table = dynamodb.Table("users")
+
+    table.put_item(
+        Item={
+            "forum_name": "the-key",
+            "subject": "123",
+            "body": "some test message",
+            "attachment": "something",
+        }
+    )
+
+    table.put_item(
+        Item={
+            "forum_name": "not-the-key",
+            "subject": "123",
+            "body": "some other test message",
+            "attachment": "something",
+        }
+    )
+    with pytest.raises(ClientError) as exc:
+        _result = table.get_item(
+            Key={"forum_name": "the-key", "subject": "123"},
+            ProjectionExpression="#rl, #rt, subject, subject",  # duplicate subject
+            ExpressionAttributeNames={"#rl": "body", "#rt": "attachment"},
+        )
+
+    err = exc.value.response["Error"]
+    assert err["Code"] == "ValidationException"
+    assert err["Message"] == (
+        "Invalid ProjectionExpression: "
+        "Two document paths overlap with each other; must remove or rewrite one of these paths; "
+        "path one: [subject], path two: [subject]"
+    )
+
+
+@mock_aws
+def test_scan_with_duplicate_path_in_projection_expressions():
+    dynamodb = boto3.resource("dynamodb", region_name="us-east-1")
+
+    # Create the DynamoDB table.
+    table = dynamodb.create_table(
+        TableName="users",
+        KeySchema=[
+            {"AttributeName": "forum_name", "KeyType": "HASH"},
+            {"AttributeName": "subject", "KeyType": "RANGE"},
+        ],
+        AttributeDefinitions=[
+            {"AttributeName": "forum_name", "AttributeType": "S"},
+            {"AttributeName": "subject", "AttributeType": "S"},
+        ],
+        ProvisionedThroughput={"ReadCapacityUnits": 5, "WriteCapacityUnits": 5},
+    )
+    table = dynamodb.Table("users")
+
+    table.put_item(
+        Item={
+            "forum_name": "the-key",
+            "subject": "123",
+            "body": "some test message",
+            "attachment": "something",
+        }
+    )
+
+    table.put_item(
+        Item={
+            "forum_name": "not-the-key",
+            "subject": "123",
+            "body": "some other test message",
+            "attachment": "something",
+        }
+    )
+
+    with pytest.raises(ClientError) as exc:
+        _results = table.scan(
+            FilterExpression=Key("forum_name").eq("the-key"),
+            ProjectionExpression="#rl, #rt, subject, subject",  # duplicate subject
+            ExpressionAttributeNames={"#rl": "body", "#rt": "attachment"},
+        )
+
+    err = exc.value.response["Error"]
+    assert err["Code"] == "ValidationException"
+    assert err["Message"] == (
+        "Invalid ProjectionExpression: "
+        "Two document paths overlap with each other; must remove or rewrite one of these paths; "
+        "path one: [subject], path two: [subject]"
+    )
+
+
 @mock_aws
 def test_put_item_wrong_datatype():
     if settings.TEST_SERVER_MODE: