Repository: getmoto/moto
Pull request #7789 (2024-06-24 20:40:26)

EC2: Enabled regex matching in filters
There could be filters which uses regex instead of full string for filter value. I added the ability to filter using regex, when only one filter value is provided. 

This was talked about in a closed issue: https://github.com/getmoto/moto/issues/7735#issuecomment-2144092920

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/getmoto/moto/pull/7789?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 94.34%. Comparing base [(`0606ffa`)](https://app.codecov.io/gh/getmoto/moto/commit/0606ffa23d3573afd54317468f61f1267211058c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto) to head [(`1dade81`)](https://app.codecov.io/gh/getmoto/moto/commit/1dade81206382dc01ca61433e0b61aaffa201f76?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto).
> Report is 1 commits behind head on master.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #7789      +/-   ##
==========================================
- Coverage   94.34%   94.34%   -0.01%     
==========================================
  Files        1107     1107              
  Lines       93973    93975       +2     
==========================================
+ Hits        88660    88661       +1     
- Misses       5313     5314       +1     
```

| [Flag](https://app.codecov.io/gh/getmoto/moto/pull/7789/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto) | Coverage Δ | |
|---|---|---|
| [servertests](https://app.codecov.io/gh/getmoto/moto/pull/7789/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto) | `29.32% <0.00%> (-0.01%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/getmoto/moto/pull/7789/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto) | `94.31% <100.00%> (-0.01%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/getmoto/moto/pull/7789?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=getmoto).


Patch:
diff --git a/docs/docs/getting_started.rst b/docs/docs/getting_started.rst
index e0933f88f..498237fbe 100644
--- a/docs/docs/getting_started.rst
+++ b/docs/docs/getting_started.rst
@@ -286,7 +286,7 @@ Moto will delete any data after the mock ends, so the state is not shared across
 
 What about those pesky imports
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-As mentioned earlier, mocks should be established __BEFORE__ the clients are set up.
+As mentioned earlier, mocks should be established *BEFORE* the clients are set up.
 
 Some background on why this is necessary:  :raw-html:`<br />`
 Moto intercepts HTTP requests using a custom event handler that hooks into botocore's event-system.  :raw-html:`<br />`
@@ -303,7 +303,7 @@ Example:
 .. sourcecode:: python
 
     def test_something(aws):
-        # s3 is a fixture defined above that yields a boto3 s3 client.
+        # aws is a fixture defined above that yields a boto3 s3 client.
         
         from some.package.that.does.something.with.s3 import some_func # <-- Local import for unit test
         # ^^ Importing here ensures that the mock has been established.
diff --git a/moto/dynamodb/models/table.py b/moto/dynamodb/models/table.py
index 06ff8721b..2696608fe 100644
--- a/moto/dynamodb/models/table.py
+++ b/moto/dynamodb/models/table.py
@@ -713,30 +713,30 @@ class Table(CloudFormationModel):
                         f"Range Key comparison but no range key found for index: {index_name}"
                     )
 
-            actual_hash_attr = index_hash_key["AttributeName"]
+            hash_attrs = [index_hash_key["AttributeName"], self.hash_key_attr]
             if index_range_key:
-                actual_range_attrs = [
+                range_attrs = [
                     index_range_key["AttributeName"],
                     self.range_key_attr,
                 ]
             else:
-                actual_range_attrs = [self.range_key_attr]
+                range_attrs = [self.range_key_attr]
 
             possible_results = []
             for item in self.all_items():
                 if not isinstance(item, Item):
                     continue
-                item_hash_key = item.attrs.get(actual_hash_attr)
-                if len(actual_range_attrs) == 1:
+                item_hash_key = item.attrs.get(hash_attrs[0])
+                if len(range_attrs) == 1:
                     if item_hash_key and item_hash_key == hash_key:
                         possible_results.append(item)
                 else:
-                    item_range_key = item.attrs.get(actual_range_attrs[0])  # type: ignore
+                    item_range_key = item.attrs.get(range_attrs[0])  # type: ignore
                     if item_hash_key and item_hash_key == hash_key and item_range_key:
                         possible_results.append(item)
         else:
-            actual_hash_attr = self.hash_key_attr
-            actual_range_attrs = [self.range_key_attr]
+            hash_attrs = [self.hash_key_attr]
+            range_attrs = [self.range_key_attr]
 
             possible_results = [
                 item
@@ -746,15 +746,15 @@ class Table(CloudFormationModel):
 
         # SORT
         if index_name:
-            if len(actual_range_attrs) == 2:
+            if len(range_attrs) == 2:
                 # Convert to float if necessary to ensure proper ordering
                 def conv(x: DynamoType) -> Any:
                     return float(x.value) if x.type == "N" else x.value
 
                 possible_results.sort(
                     key=lambda item: (  # type: ignore
-                        conv(item.attrs[actual_range_attrs[0]])  # type: ignore
-                        if item.attrs.get(actual_range_attrs[0])  # type: ignore
+                        conv(item.attrs[range_attrs[0]])  # type: ignore
+                        if item.attrs.get(range_attrs[0])  # type: ignore
                         else None
                     )
                 )
@@ -777,8 +777,8 @@ class Table(CloudFormationModel):
                 if self._item_comes_before_dct(
                     result,
                     exclusive_start_key,
-                    actual_hash_attr,
-                    actual_range_attrs,
+                    hash_attrs,
+                    range_attrs,
                     scan_index_forward,
                 ):
                     continue
@@ -926,19 +926,16 @@ class Table(CloudFormationModel):
             except IndexError:
                 index_range_key = None
 
-            actual_hash_attr = index_hash_key["AttributeName"]
+            hash_attrs = [index_hash_key["AttributeName"], self.hash_key_attr]
             if index_range_key:
-                actual_range_attrs = [
-                    index_range_key["AttributeName"],
-                    self.range_key_attr,
-                ]
+                range_attrs = [index_range_key["AttributeName"], self.range_key_attr]
             else:
-                actual_range_attrs = [self.range_key_attr]
+                range_attrs = [self.range_key_attr]
 
             items = self.has_idx_items(index_name)
         else:
-            actual_hash_attr = self.hash_key_attr
-            actual_range_attrs = [self.range_key_attr]
+            hash_attrs = [self.hash_key_attr]
+            range_attrs = [self.range_key_attr]
 
             items = self.all_items()
 
@@ -951,8 +948,8 @@ class Table(CloudFormationModel):
                 if self._item_comes_before_dct(
                     item,
                     exclusive_start_key,
-                    actual_hash_attr,
-                    actual_range_attrs,
+                    hash_attrs,
+                    range_attrs,
                     True,
                 ):
                     continue
@@ -1012,13 +1009,16 @@ class Table(CloudFormationModel):
         self,
         item: Item,
         dct: Dict[str, Any],
-        hash_key_attr: str,
+        hash_key_attrs: List[str],
         range_key_attrs: List[Optional[str]],
         scan_index_forward: bool,
     ) -> bool:
         """
         Does item appear before or at dct relative to sort options?
 
+        hash_key_attrs: The list of hash keys.
+        Includes the key of the GSI (first, if it exists) and the key of the main table.
+
         range_key_attrs: A list of range keys (RK).
         Includes the RK-name of the GSI (first, if it exists) and the RK-name of the main table (second - can be None).
         When sorting a GSI, we'll try to sort by the GSI RK first.
@@ -1026,11 +1026,21 @@ class Table(CloudFormationModel):
         If that is the case, we compare by the RK of the main table instead
         Related: https://github.com/getmoto/moto/issues/7761
         """
-        dict_hash_val = DynamoType(dct.get(hash_key_attr))  # type: ignore[arg-type]
-        item_hash_val = item.attrs[hash_key_attr]
-        if item_hash_val != dict_hash_val:
+        hash_comes_before = any(
+            item.attrs[hash_key_attr] != DynamoType(dct.get(hash_key_attr))  # type: ignore
+            for hash_key_attr in hash_key_attrs
+            if hash_key_attr in item.attrs
+        )
+        if hash_comes_before:
             # If hash keys are different, order immediately
-            return bool(item_hash_val < dict_hash_val) == scan_index_forward
+            return (
+                any(
+                    item.attrs[hash_key_attr] < DynamoType(dct.get(hash_key_attr))  # type: ignore
+                    for hash_key_attr in hash_key_attrs
+                    if hash_key_attr in item.attrs
+                )
+                == scan_index_forward
+            )
         if not any(range_key_attrs):
             # If hash keys match and no range key, items are identical
             return True
diff --git a/moto/ec2/utils.py b/moto/ec2/utils.py
index 6387be29d..e3beeb1fd 100644
--- a/moto/ec2/utils.py
+++ b/moto/ec2/utils.py
@@ -517,13 +517,13 @@ def filter_internet_gateways(
 
 def is_filter_matching(obj: Any, _filter: str, filter_value: Any) -> bool:
     value = obj.get_filter_value(_filter)
-
     if filter_value is None:
         return False
 
     if isinstance(value, str):
         if not isinstance(filter_value, list):
             filter_value = [filter_value]
+
         if any(fnmatch.fnmatch(value, pattern) for pattern in filter_value):
             return True
         return False
@@ -534,6 +534,9 @@ def is_filter_matching(obj: Any, _filter: str, filter_value: Any) -> bool:
 
     try:
         value = set(value)
+        if isinstance(filter_value, list) and len(filter_value) == 1:
+            return any(fnmatch.fnmatch(element, filter_value[0]) for element in value)
+
         return (value and value.issubset(filter_value)) or value.issuperset(
             filter_value
         )
diff --git a/moto/elbv2/exceptions.py b/moto/elbv2/exceptions.py
index 97fcf48ca..cd4cc6e47 100644
--- a/moto/elbv2/exceptions.py
+++ b/moto/elbv2/exceptions.py
@@ -47,6 +47,14 @@ class TooManyTagsError(ELBClientError):
         )
 
 
+class TooManyCertificatesError(ELBClientError):
+    def __init__(self) -> None:
+        super().__init__(
+            "TooManyCertificates",
+            "You've reached the limit on the number of certificates per load balancer",
+        )
+
+
 class BadHealthCheckDefinition(ELBClientError):
     def __init__(self) -> None:
         super().__init__(
diff --git a/moto/elbv2/models.py b/moto/elbv2/models.py
index ca92ac9bb..73bfdb468 100644
--- a/moto/elbv2/models.py
+++ b/moto/elbv2/models.py
@@ -37,6 +37,7 @@ from .exceptions import (
     RuleNotFoundError,
     SubnetNotFoundError,
     TargetGroupNotFoundError,
+    TooManyCertificatesError,
     TooManyTagsError,
     ValidationError,
 )
@@ -1942,6 +1943,9 @@ Member must satisfy regular expression pattern: {expression}"
         listener = self.describe_listeners(load_balancer_arn=None, listener_arns=[arn])[
             0
         ]
+        # https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-limits.html
+        if len(certificates) + len(listener.certificates) > 25:
+            raise TooManyCertificatesError()
         listener.certificates.extend([c["certificate_arn"] for c in certificates])
         return listener.certificates
 
diff --git a/moto/elbv2/responses.py b/moto/elbv2/responses.py
index 74dc6938c..a744415a2 100644
--- a/moto/elbv2/responses.py
+++ b/moto/elbv2/responses.py
@@ -680,6 +680,7 @@ class ELBV2Response(BaseResponse):
             "network-load-balancers": 20,
             "targets-per-network-load-balancer": 200,
             "listeners-per-network-load-balancer": 50,
+            "certificates-per-application-load-balancer": 25,
         }
 
         template = self.response_template(DESCRIBE_LIMITS_TEMPLATE)
diff --git a/moto/resourcegroupstaggingapi/models.py b/moto/resourcegroupstaggingapi/models.py
index 55d5e5f55..c3548be44 100644
--- a/moto/resourcegroupstaggingapi/models.py
+++ b/moto/resourcegroupstaggingapi/models.py
@@ -211,7 +211,11 @@ class ResourceGroupsTaggingAPIBackend(BaseBackend):
                 yield {"ResourceARN": f"{vault.backup_vault_arn}", "Tags": tags}
 
         # S3
-        if not resource_type_filters or "s3" in resource_type_filters:
+        if (
+            not resource_type_filters
+            or "s3" in resource_type_filters
+            or "s3:bucket" in resource_type_filters
+        ):
             for bucket in self.s3_backend.buckets.values():
                 tags = self.s3_backend.tagger.list_tags_for_resource(bucket.arn)["Tags"]
                 if not tags or not tag_filter(


Test patch:
diff --git a/tests/test_dynamodb/test_dynamodb_query.py b/tests/test_dynamodb/test_dynamodb_query.py
index 5968013af..4770efd3e 100644
--- a/tests/test_dynamodb/test_dynamodb_query.py
+++ b/tests/test_dynamodb/test_dynamodb_query.py
@@ -770,3 +770,43 @@ class TestFilterExpression:
             "app": "app1",
             "nested": {"version": "version1", "contents": ["value1", "value2"]},
         }
+
+
+@pytest.mark.aws_verified
+@dynamodb_aws_verified(add_range=False, add_gsi_range=True)
+def test_query_gsi_pagination_with_string_gsi_range_no_sk(table_name=None):
+    dynamodb = boto3.resource("dynamodb", region_name="us-east-1")
+    table = dynamodb.Table(table_name)
+
+    for i in range(3, 7):
+        table.put_item(Item={"pk": f"{i}", "gsi_pk": "john", "gsi_sk": "jane"})
+
+    for i in range(9, 6, -1):
+        table.put_item(Item={"pk": f"{i}", "gsi_pk": "john", "gsi_sk": "jane"})
+
+    for i in range(3):
+        table.put_item(Item={"pk": f"{i}", "gsi_pk": "john", "gsi_sk": "jane"})
+
+    page1 = table.query(
+        KeyConditionExpression=Key("gsi_pk").eq("john"),
+        IndexName="test_gsi",
+        Limit=6,
+    )
+    assert page1["Count"] == 6
+    assert page1["ScannedCount"] == 6
+    assert len(page1["Items"]) == 6
+
+    page2 = table.query(
+        KeyConditionExpression=Key("gsi_pk").eq("john"),
+        IndexName="test_gsi",
+        Limit=6,
+        ExclusiveStartKey=page1["LastEvaluatedKey"],
+    )
+    assert page2["Count"] == 4
+    assert page2["ScannedCount"] == 4
+    assert len(page2["Items"]) == 4
+    assert "LastEvaluatedKey" not in page2
+
+    results = page1["Items"] + page2["Items"]
+    subjects = set([int(r["pk"]) for r in results])
+    assert subjects == set(range(10))
diff --git a/tests/test_dynamodb/test_dynamodb_scan.py b/tests/test_dynamodb/test_dynamodb_scan.py
index 91318a748..1f070a9b4 100644
--- a/tests/test_dynamodb/test_dynamodb_scan.py
+++ b/tests/test_dynamodb/test_dynamodb_scan.py
@@ -464,6 +464,41 @@ def test_scan_gsi_pagination_with_string_gsi_range_and_empty_gsi_sk(table_name=N
     assert set([r["sk"] for r in results]) == {"0", "1", "2", "7", "8", "9"}
 
 
+@pytest.mark.aws_verified
+@dynamodb_aws_verified(add_range=False, add_gsi_range=True)
+def test_scan_gsi_pagination_with_string_gsi_range_no_sk(table_name=None):
+    dynamodb = boto3.resource("dynamodb", region_name="us-east-1")
+    table = dynamodb.Table(table_name)
+
+    for i in range(3, 7):
+        table.put_item(Item={"pk": f"{i}", "gsi_pk": "john", "gsi_sk": "jane"})
+
+    for i in range(9, 6, -1):
+        table.put_item(Item={"pk": f"{i}", "gsi_pk": "john", "gsi_sk": "jane"})
+
+    for i in range(3):
+        table.put_item(Item={"pk": f"{i}", "gsi_pk": "john", "gsi_sk": "jane"})
+
+    page1 = table.scan(IndexName="test_gsi", Limit=6)
+    assert page1["Count"] == 6
+    assert page1["ScannedCount"] == 6
+    assert len(page1["Items"]) == 6
+
+    page2 = table.scan(
+        IndexName="test_gsi",
+        Limit=6,
+        ExclusiveStartKey=page1["LastEvaluatedKey"],
+    )
+    assert page2["Count"] == 4
+    assert page2["ScannedCount"] == 4
+    assert len(page2["Items"]) == 4
+    assert "LastEvaluatedKey" not in page2
+
+    results = page1["Items"] + page2["Items"]
+    subjects = set([int(r["pk"]) for r in results])
+    assert subjects == set(range(10))
+
+
 @mock_aws
 class TestFilterExpression:
     def test_scan_filter(self):
diff --git a/tests/test_ec2/test_route_tables.py b/tests/test_ec2/test_route_tables.py
index 9db133ef2..eb7b6605c 100644
--- a/tests/test_ec2/test_route_tables.py
+++ b/tests/test_ec2/test_route_tables.py
@@ -283,7 +283,6 @@ def test_route_tables_filters_transit_gateway():
 
     gateway = response["TransitGateway"]
     main_route_table.create_route(TransitGatewayId=gateway["TransitGatewayId"])
-
     route_tables = client.describe_route_tables(
         Filters=[
             {
@@ -297,6 +296,20 @@ def test_route_tables_filters_transit_gateway():
     route_table = route_tables[0]
     assert route_table["RouteTableId"] == main_route_table_id
 
+    # Filter using regex
+    route_tables = client.describe_route_tables(
+        Filters=[
+            {
+                "Name": "route.transit-gateway-id",
+                "Values": ["tgw-*"],
+            }
+        ]
+    )["RouteTables"]
+
+    assert len(route_tables) == 1
+    route_table = route_tables[0]
+    assert route_table["RouteTableId"] == main_route_table_id
+
 
 @mock_aws
 def test_route_table_associations():
diff --git a/tests/test_elbv2/test_elbv2.py b/tests/test_elbv2/test_elbv2.py
index 542ce0786..5abc626c8 100644
--- a/tests/test_elbv2/test_elbv2.py
+++ b/tests/test_elbv2/test_elbv2.py
@@ -1582,6 +1582,12 @@ def test_add_listener_certificate():
     ]
     assert len(certs) == 0
 
+    with pytest.raises(ClientError) as exc:
+        client.add_listener_certificates(
+            ListenerArn=listener_arn, Certificates=[{"CertificateArn": google_arn}] * 50
+        )
+    assert exc.value.response["Error"]["Code"] == "TooManyCertificates"
+
 
 @mock_aws
 def test_forward_config_action():
diff --git a/tests/test_resourcegroupstaggingapi/test_resourcegroupstaggingapi.py b/tests/test_resourcegroupstaggingapi/test_resourcegroupstaggingapi.py
index 60fc8cc6d..ed9dacc46 100644
--- a/tests/test_resourcegroupstaggingapi/test_resourcegroupstaggingapi.py
+++ b/tests/test_resourcegroupstaggingapi/test_resourcegroupstaggingapi.py
@@ -1,6 +1,7 @@
 import json
 
 import boto3
+import pytest
 from botocore.client import ClientError
 
 from moto import mock_aws
@@ -534,8 +535,9 @@ def test_get_resources_target_group():
     assert {"Key": "Test", "Value": "1"} in resp["ResourceTagMappingList"][0]["Tags"]
 
 
+@pytest.mark.parametrize("resource_type", ["s3", "s3:bucket"])
 @mock_aws
-def test_get_resources_s3():
+def test_get_resources_s3(resource_type):
     # Tests pagination
     s3_client = boto3.client("s3", region_name="eu-central-1")
 
@@ -556,14 +558,16 @@ def test_get_resources_s3():
         response_keys.add("key" + i_str)
 
     rtapi = boto3.client("resourcegroupstaggingapi", region_name="eu-central-1")
-    resp = rtapi.get_resources(ResourcesPerPage=2)
+    resp = rtapi.get_resources(ResourcesPerPage=2, ResourceTypeFilters=[resource_type])
     for resource in resp["ResourceTagMappingList"]:
         response_keys.remove(resource["Tags"][0]["Key"])
 
     assert len(response_keys) == 2
 
     resp = rtapi.get_resources(
-        ResourcesPerPage=2, PaginationToken=resp["PaginationToken"]
+        ResourcesPerPage=2,
+        PaginationToken=resp["PaginationToken"],
+        ResourceTypeFilters=[resource_type],
     )
     for resource in resp["ResourceTagMappingList"]:
         response_keys.remove(resource["Tags"][0]["Key"])