Repository: localstack/localstack

Issue:
add test cases for ASF op router
This demonstrates the behavior of #5944. It also adds a test that identifies the affected services.

@alexrashed feel free to close this one and pull the tests into your own PR :-)
ASF op router cannot deal with dashes in URI rules
## Problem

The ASF op router cannot route certain operations that have dashes in their URI variable name. This is because Werkzeug allows only `[a-zA-Z_][a-zA-Z0-9_]*` as variable names, see [werkzeug/routing.py](https://github.com/pallets/werkzeug/blob/6962882b39aa6a2abbeb8e14da212e3d35079c86/src/werkzeug/routing.py#L158). However, some services like apigatewayv2 have dashes in their URI variable names, see [apigatewayv2 spec](https://github.com/boto/botocore/blob/edf0c1a4679eca7a18c89802af9c159a9a423550/botocore/data/apigatewayv2/2018-11-29/service-2.json#L1279-L1284): `"/v2/tags/{resource-arn}"`.

Werkzeug will raise the following exception:

```
Traceback (most recent call last):
  ...
  File "/home/thomas/workspace/localstack/localstack/localstack/aws/protocol/op_router.py", line 209, in __init__
    self._map = _create_service_map(service)
  File "/home/thomas/workspace/localstack/localstack/localstack/aws/protocol/op_router.py", line 197, in _create_service_map
    return Map(rules=rules)
  File "/home/thomas/workspace/localstack/localstack/.venv/lib/python3.8/site-packages/werkzeug/routing.py", line 1500, in __init__
    self.add(rulefactory)
  File "/home/thomas/workspace/localstack/localstack/.venv/lib/python3.8/site-packages/werkzeug/routing.py", line 1541, in add
    rule.bind(self)
  File "/home/thomas/workspace/localstack/localstack/.venv/lib/python3.8/site-packages/werkzeug/routing.py", line 800, in bind
    self.compile()
  File "/home/thomas/workspace/localstack/localstack/.venv/lib/python3.8/site-packages/werkzeug/routing.py", line 875, in compile
    _build_regex(self.rule if self.is_leaf else self.rule.rstrip("/"))
  File "/home/thomas/workspace/localstack/localstack/.venv/lib/python3.8/site-packages/werkzeug/routing.py", line 842, in _build_regex
    for converter, arguments, variable in parse_rule(rule):
  File "/home/thomas/workspace/localstack/localstack/.venv/lib/python3.8/site-packages/werkzeug/routing.py", line 243, in parse_rule
    raise ValueError(f"malformed url rule: {rule!r}")
ValueError: malformed url rule: '/v2/tags/<resource-arn>'
```



## Test

This parser test validates the behavior:
```python
def test_rest_url_parameter_with_dashes():
    """
    Test if requestUri parameters with dashes in them (e.g., "/v2/tags/{resource-arn}") are parsed correctly.
    """
    _botocore_parser_integration_test(
        service="apigatewayv2",
        action="GetTags",
        ResourceArn="arn:aws:apigatewayv2:us-east-1:000000000000:foobar",
    )
```

## Required change

This means that here we need to replace dashes with underscores, but only in the variable names, not the entire URL, which looks pretty painful to do.  https://github.com/localstack/localstack/blob/836616ef86ef844cb9aa13365d8bcd56e30ef8b6/localstack/aws/protocol/op_router.py#L161-L162



Discussion:


Maybe it makes sense to start writing our own simple state machine, something like this:

```python
def main():
    uri = '/v2/tags/<resource-arn>'

    in_var = False
    pattern = []
    for c in uri:
        if c == '<':
            in_var = True
            pattern.append('{')
            continue

        if in_var:
            if c == '>':
                in_var = False
                pattern.append('}')
                continue
            if c == '-':
                pattern.append('_')
                continue

        pattern.append(c)

    uri = ''.join(pattern)
    print(uri)

if __name__ == '__main__':
    main()
```


Resolving patch:
diff --git a/localstack/aws/protocol/op_router.py b/localstack/aws/protocol/op_router.py
--- a/localstack/aws/protocol/op_router.py
+++ b/localstack/aws/protocol/op_router.py
@@ -1,6 +1,6 @@
 import re
 from collections import defaultdict
-from typing import Any, Dict, List, Mapping, NamedTuple, Optional, Tuple
+from typing import Any, AnyStr, Dict, List, Mapping, Match, NamedTuple, Optional, Tuple
 from urllib.parse import parse_qs, unquote
 
 from botocore.model import OperationModel, ServiceModel, StructureShape
@@ -144,22 +144,52 @@ def match_request(self, request: Request) -> _RequiredArgsRule:
         raise NotFound()
 
 
-# Regex to find path parameters which should be greedy according to the spec
-_greedy_regex = re.compile(r"{(\w+)\+}")
+# Regex to find path parameters in requestUris of AWS service specs (f.e. /{param1}/{param2+})
+_path_param_regex = re.compile(r"({.+?})")
+# Translation table which replaces characters forbidden in Werkzeug rule names with temporary replacements
+# Note: The temporary replacements must not occur in any requestUri of any operation in any service!
+_rule_replacements = {"-": "_0_"}
+# String translation table for #_rule_replacements for str#translate
+_rule_replacement_table = str.maketrans(_rule_replacements)
 
 
-def _request_uri_path_to_rule_string(request_uri_path: str):
+def _transform_path_params_to_rule_vars(match: Match[AnyStr]) -> str:
     """
-    Translates the given requestUri path (without a potential query suffix) to a Werkzeug rule string which can be used
-    with Werkzeug's route matching framework.
+    Transforms a request URI path param to a valid Werkzeug Rule string variable placeholder.
+    This transformation function should be used in combination with _path_param_regex on the request URIs (without any
+    query params).
 
-    :param request_uri_path: path section of the operation model's http request URI (without a potential query suffix)
-    :return: a Werkzeug routing framework compatible rule string
+    :param match: Regex match which contains a single group. The match group is a request URI path param, including the
+                    surrounding curly braces.
+    :return: Werkzeug rule string variable placeholder which is semantically equal to the given request URI path param
+
+    """
+    # get the group match and strip the curly braces
+    request_uri_variable: str = match.group(0)[1:-1]
+
+    # if the request URI param is greedy (f.e. /foo/{Bar+}), add Werkzeug's "path" prefix (/foo/{path:Bar})
+    greedy_prefix = ""
+    if request_uri_variable.endswith("+"):
+        greedy_prefix = "path:"
+        request_uri_variable = request_uri_variable.strip("+")
+
+    # replace forbidden chars (not allowed in Werkzeug rule variable names) with their placeholder
+    escaped_request_uri_variable = request_uri_variable.translate(_rule_replacement_table)
+
+    return f"<{greedy_prefix}{escaped_request_uri_variable}>"
+
+
+def _post_process_arg_name(arg_key: str) -> str:
+    """
+    Reverses previous manipulations to the path parameters names (like replacing forbidden characters with
+    placeholders).
+    :param arg_key: Path param key name extracted using Werkzeug rules
+    :return: Post-processed ("un-sanitized") path param key
     """
-    # replace any greedy URI params (f.e. /foo/{Bar+}) with the werkzeug notation (/foo/{path:Bar})
-    rule_string = _greedy_regex.sub(r"{path:\g<1>}", request_uri_path)
-    # replace the spec param notation (f.e. /foo/{Bar}) with the werkzeug path param notation (/foo/<Bar>)
-    return rule_string.replace("{", "<").replace("}", ">")
+    result = arg_key
+    for original, substitution in _rule_replacements.items():
+        result = result.replace(substitution, original)
+    return result
 
 
 def _create_service_map(service: ServiceModel) -> Map:
@@ -167,7 +197,6 @@ def _create_service_map(service: ServiceModel) -> Map:
     Creates a Werkzeug Map object with all rules necessary for the specific service.
     :param service: botocore service model to create the rules for
     :return: a Map instance which is used to perform the in-service operation routing
-             -
     """
     ops = [service.operation_model(op_name) for op_name in service.operation_names]
 
@@ -182,7 +211,7 @@ def _create_service_map(service: ServiceModel) -> Map:
     # create a matching rule for each (path, method) combination
     for (path, method), ops in path_index.items():
         # translate the requestUri to a Werkzeug rule string
-        rule_string = _request_uri_path_to_rule_string(path)
+        rule_string = _path_param_regex.sub(_transform_path_params_to_rule_vars, path)
 
         if len(ops) == 1:
             # if there is only a single operation for a (path, method) combination,
@@ -228,8 +257,10 @@ def match(self, request: Request) -> Tuple[OperationModel, Mapping[str, Any]]:
         if isinstance(rule, _RequestMatchingRule):
             rule = rule.match_request(request)
 
-        # the path params might still be url-encoded
-        args = {k: unquote(v) for k, v in args.items()}
+        # post process the arg keys and values
+        # - the path param keys need to be "un-sanitized", i.e. sanitized rule variable names need to be reverted
+        # - the path param values might still be url-encoded
+        args = {_post_process_arg_name(k): unquote(v) for k, v in args.items()}
 
         # extract the operation model from the rule
         operation: OperationModel = rule.endpoint


Test patch:
diff --git a/tests/unit/aws/protocol/test_op_router.py b/tests/unit/aws/protocol/test_op_router.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/aws/protocol/test_op_router.py
@@ -0,0 +1,26 @@
+import pytest
+from werkzeug.exceptions import NotFound
+
+from localstack.aws.protocol.op_router import RestServiceOperationRouter
+from localstack.aws.spec import list_services, load_service
+from localstack.http import Request
+
+
+def _collect_services():
+    for service in list_services():
+        if service.protocol.startswith("rest"):
+            yield service.service_name
+
+
+@pytest.mark.parametrize(
+    "service",
+    _collect_services(),
+)
+@pytest.mark.param
+def test_create_op_router_works_for_every_service(service):
+    router = RestServiceOperationRouter(load_service(service))
+
+    try:
+        router.match(Request("GET", "/"))
+    except NotFound:
+        pass
diff --git a/tests/unit/aws/protocol/test_parser.py b/tests/unit/aws/protocol/test_parser.py
--- a/tests/unit/aws/protocol/test_parser.py
+++ b/tests/unit/aws/protocol/test_parser.py
@@ -800,15 +800,30 @@ def test_parse_restjson_querystring_list_parsing():
 
 
 def test_restjson_operation_detection_with_query_suffix_in_requesturi():
-    # Test if the correct operation is detected if the requestURI pattern of the specification contains the first query
-    # parameter, f.e. API Gateway's ImportRestApi: "/restapis?mode=import"
+    """
+    Test if the correct operation is detected if the requestURI pattern of the specification contains the first query
+    parameter, f.e. API Gateway's ImportRestApi: "/restapis?mode=import
+    """
     _botocore_parser_integration_test(service="apigateway", action="ImportRestApi", body=b"Test")
 
 
+def test_rest_url_parameter_with_dashes():
+    """
+    Test if requestUri parameters with dashes in them (e.g., "/v2/tags/{resource-arn}") are parsed correctly.
+    """
+    _botocore_parser_integration_test(
+        service="apigatewayv2",
+        action="GetTags",
+        ResourceArn="arn:aws:apigatewayv2:us-east-1:000000000000:foobar",
+    )
+
+
 def test_restxml_operation_detection_with_query_suffix_without_value_in_requesturi():
-    # Test if the correct operation is detected if the requestURI pattern of the specification contains the first query
-    # parameter without a specific value, f.e. CloudFront's CreateDistributionWithTags:
-    # "/2020-05-31/distribution?WithTags"
+    """
+    Test if the correct operation is detected if the requestURI pattern of the specification contains the first query
+    parameter without a specific value, f.e. CloudFront's CreateDistributionWithTags:
+    "/2020-05-31/distribution?WithTags"
+    """
     _botocore_parser_integration_test(
         service="cloudfront",
         action="CreateDistributionWithTags",