Repository: localstack/localstack

Issue:
bug: opensearch/elasticsearch custom endpoint not saved or included in responses
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

When creating an opensearch/elasticsearch domain, a custom endpoint URL may be set on the domain via the `CustomEndpoint`/`CustomEndpointEnabled` request params. localstack respects these params when initially setting up the domain and correctly routes requests made to the custom endpoint, but it doesn't "remember" this config, which has two effects:

1. The response from the `CreateElasticsearchDomain` (and any subsequent `DescribeElasticsearchDomain`) request incorrectly has `CustomEndpoint` omitted and `CustomEndpointEnabled` set to `false`
2. When restoring an opensearch/elasticsearch domain from persisted state, any custom endpoint config is lost/ignored, and making a request to the custom endpoint returns a 404 (of course this is also broken by #8092, but this bug would still happen even if that one is fixed!)

### How are you starting LocalStack?

With a docker-compose file

### Steps To Reproduce

```
awslocal es create-elasticsearch-domain \
  --domain-name example \
  --domain-endpoint-options CustomEndpointEnabled=true,CustomEndpoint=example.com:1234/foo
```
This gives the following response (irrelevant properties omitted):
```jsonc
{
    "DomainStatus": {
        "Endpoint": "example.com:1234/foo",
        "DomainEndpointOptions": {
            "EnforceHTTPS": false,
            "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07",
            "CustomEndpointEnabled": false
        },
        ...
    }
}
```

Note that `CustomEndpointEnabled` is false when it should be true, and `CustomEndpoint` is omitted when it should be `example.com:1234/foo` (the same as the `Endpoint` property). 

The same response is given when subsequently running `awslocal es describe-elasticsearch-domain --domain-name example`

### Environment

```markdown
- OS: docker desktop (on a windows 11 host)
- LocalStack: latest (2.3.3.dev)
```


### Anything else?

_No response_


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [contributing guide](https://docs.localstack.cloud/contributing/).

Resolving patch:
diff --git a/localstack/services/opensearch/provider.py b/localstack/services/opensearch/provider.py
--- a/localstack/services/opensearch/provider.py
+++ b/localstack/services/opensearch/provider.py
@@ -117,6 +117,12 @@
     DedicatedMasterCount=1,
 )
 
+DEFAULT_OPENSEARCH_DOMAIN_ENDPOINT_OPTIONS = DomainEndpointOptions(
+    EnforceHTTPS=False,
+    TLSSecurityPolicy=TLSSecurityPolicy.Policy_Min_TLS_1_0_2019_07,
+    CustomEndpointEnabled=False,
+)
+
 
 def cluster_manager() -> ClusterManager:
     global __CLUSTER_MANAGER
@@ -355,18 +361,13 @@ def get_domain_status(domain_key: DomainKey, deleted=False) -> DomainStatus:
             AutomatedUpdateDate=datetime.fromtimestamp(0, tz=timezone.utc),
             OptionalDeployment=True,
         ),
-        DomainEndpointOptions=DomainEndpointOptions(
-            EnforceHTTPS=False,
-            TLSSecurityPolicy=TLSSecurityPolicy.Policy_Min_TLS_1_0_2019_07,
-            CustomEndpointEnabled=False,
-        ),
+        DomainEndpointOptions=stored_status.get("DomainEndpointOptions")
+        or DEFAULT_OPENSEARCH_DOMAIN_ENDPOINT_OPTIONS,
         AdvancedSecurityOptions=AdvancedSecurityOptions(
             Enabled=False, InternalUserDatabaseEnabled=False
         ),
         AutoTuneOptions=AutoTuneOptionsOutput(State=AutoTuneState.ENABLE_IN_PROGRESS),
     )
-    if stored_status.get("Endpoint"):
-        new_status["Endpoint"] = new_status.get("Endpoint")
     return new_status
 
 
@@ -400,6 +401,20 @@ def is_valid_domain_name(name: str) -> bool:
     return True if _domain_name_pattern.match(name) else False
 
 
+def validate_endpoint_options(endpoint_options: DomainEndpointOptions):
+    custom_endpoint = endpoint_options.get("CustomEndpoint", "")
+    custom_endpoint_enabled = endpoint_options.get("CustomEndpointEnabled", False)
+
+    if custom_endpoint and not custom_endpoint_enabled:
+        raise ValidationException(
+            "CustomEndpointEnabled flag should be set in order to use CustomEndpoint."
+        )
+    if custom_endpoint_enabled and not custom_endpoint:
+        raise ValidationException(
+            "Please provide CustomEndpoint field to create a custom endpoint."
+        )
+
+
 class OpensearchProvider(OpensearchApi, ServiceLifecycleHook):
     @staticmethod
     def get_store(account_id: str, region_name: str) -> OpenSearchStore:
@@ -485,6 +500,9 @@ def create_domain(
                 "Member must satisfy regular expression pattern: [a-z][a-z0-9\\-]+"
             )
 
+        if domain_endpoint_options:
+            validate_endpoint_options(domain_endpoint_options)
+
         with _domain_mutex:
             if domain_name in store.opensearch_domains:
                 raise ResourceAlreadyExistsException(
@@ -499,6 +517,10 @@ def create_domain(
 
             # "create" domain data
             store.opensearch_domains[domain_name] = get_domain_status(domain_key)
+            if domain_endpoint_options:
+                store.opensearch_domains[domain_name]["DomainEndpointOptions"] = (
+                    DEFAULT_OPENSEARCH_DOMAIN_ENDPOINT_OPTIONS | domain_endpoint_options
+                )
 
             # lazy-init the cluster (sets the Endpoint and Processing flag of the domain status)
             # TODO handle additional parameters (cluster config,...)


Test patch:
diff --git a/tests/aws/services/opensearch/test_opensearch.py b/tests/aws/services/opensearch/test_opensearch.py
--- a/tests/aws/services/opensearch/test_opensearch.py
+++ b/tests/aws/services/opensearch/test_opensearch.py
@@ -450,13 +450,33 @@ def test_create_domain_with_invalid_name(self, aws_client):
         with pytest.raises(botocore.exceptions.ClientError) as e:
             aws_client.opensearch.create_domain(
                 DomainName="123abc"
-            )  # domain needs to start with characters
+            )  # domain needs to start with alphabetic characters
         assert e.value.response["Error"]["Code"] == "ValidationException"
 
         with pytest.raises(botocore.exceptions.ClientError) as e:
             aws_client.opensearch.create_domain(DomainName="abc#")  # no special characters allowed
         assert e.value.response["Error"]["Code"] == "ValidationException"
 
+    @markers.aws.unknown
+    def test_create_domain_with_invalid_custom_endpoint(self, aws_client):
+        with pytest.raises(botocore.exceptions.ClientError) as e:
+            aws_client.opensearch.create_domain(
+                DomainName="abc",
+                DomainEndpointOptions={
+                    "CustomEndpoint": "custom-endpoint",
+                },
+            )  # CustomEndpoint cannot be set without CustomEndpointEnabled
+        assert e.value.response["Error"]["Code"] == "ValidationException"
+
+        with pytest.raises(botocore.exceptions.ClientError) as e:
+            aws_client.opensearch.create_domain(
+                DomainName="abc",
+                DomainEndpointOptions={
+                    "CustomEndpointEnabled": True,
+                },
+            )  # CustomEndpointEnabled cannot be set without CustomEndpoint
+        assert e.value.response["Error"]["Code"] == "ValidationException"
+
     @markers.aws.validated
     def test_exception_header_field(self, aws_client):
         """Test if the error response correctly sets the error code in the headers (see #6304)."""
@@ -707,6 +727,13 @@ def test_custom_endpoint(
             DomainName=domain_name, DomainEndpointOptions=domain_endpoint_options
         )
 
+        response = aws_client.opensearch.describe_domain(DomainName=domain_name)
+        response_domain_endpoint_options = response["DomainStatus"]["DomainEndpointOptions"]
+        assert response_domain_endpoint_options["EnforceHTTPS"] is False
+        assert response_domain_endpoint_options["TLSSecurityPolicy"]
+        assert response_domain_endpoint_options["CustomEndpointEnabled"] is True
+        assert response_domain_endpoint_options["CustomEndpoint"] == custom_endpoint
+
         response = aws_client.opensearch.list_domain_names(EngineType="OpenSearch")
         domain_names = [domain["DomainName"] for domain in response["DomainNames"]]
 
@@ -722,9 +749,7 @@ def test_custom_endpoint_disabled(
         self, opensearch_wait_for_cluster, opensearch_create_domain, aws_client
     ):
         domain_name = f"opensearch-domain-{short_uid()}"
-        custom_endpoint = "http://localhost:4566/my-custom-endpoint"
         domain_endpoint_options = {
-            "CustomEndpoint": custom_endpoint,
             "CustomEndpointEnabled": False,
         }
 
@@ -736,14 +761,16 @@ def test_custom_endpoint_disabled(
         response_domain_name = response["DomainStatus"]["DomainName"]
         assert domain_name == response_domain_name
 
+        response_domain_endpoint_options = response["DomainStatus"]["DomainEndpointOptions"]
+        assert response_domain_endpoint_options["EnforceHTTPS"] is False
+        assert response_domain_endpoint_options["TLSSecurityPolicy"]
+        assert response_domain_endpoint_options["CustomEndpointEnabled"] is False
+        assert "CustomEndpoint" not in response_domain_endpoint_options
+
         endpoint = f"http://{response['DomainStatus']['Endpoint']}"
 
         # wait for the cluster
         opensearch_wait_for_cluster(domain_name=domain_name)
-        response = requests.get(f"{custom_endpoint}/_cluster/health")
-        assert not response.ok
-        assert response.status_code == 404
-
         response = requests.get(f"{endpoint}/_cluster/health")
         assert response.ok
         assert response.status_code == 200