Repository: localstack/localstack

Issue:
bug: OpenSearch domain error running Terraform destroy
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

I'm using terraform to create/destroy infrastructure and specifically using Terratest & Localstack for testing purposes. I can create the OpenSearch domain without issues but when I try to destroy it fails and stop the rest of the destroy process. 

```
[ERROR] vertex "module.opensearch.aws_elasticsearch_domain.opensearch (destroy)" error: error waiting for Elasticsearch Domain (arn:aws:es:us-east-1:000000000000:domain/testing) delete: : Domain not found: testing
        status code: 409, request id: H9JVOYUA5R0OBS0JU8TIU2J4UHSJLTQ8DKFM412W34SUX67LV4P4
```

The domain is destroyed but because it fails checking the status after the destruction, the terraform state is corrupt, so even if you try to run a destroy after that, is going to fail because the domain is not there but the terraform state still have the information about it.

### Expected Behavior

Terraform should not fail destroying that, and I think is because of the way localstack is creating the response to the **_GET /2015-01-01/es/domain/testing HTTP/1.1_** request which terraform is using for checking when the destruction process is completed. I'll provide more details in the "**_Anything else?_**" section of the issue.

### How are you starting LocalStack?

With a `docker run` command

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

    docker run --name manual-go-test -d --rm -it -p 4566:4566 -p 4510-4559:4510-4559 localstack/localstack

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

Using the attached files just follow this steps:
- terraform init
- terraform plan
- terraform apply (yes when asking)
- awslocal opensearch list-domain-names (for checking the domain creation)
- terraform destroy (yes when asking, to reproduce the error)

Expected terraform plan output:
```
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_elasticsearch_domain.opensearch will be created
  + resource "aws_elasticsearch_domain" "opensearch" {
      + access_policies       = (known after apply)
      + advanced_options      = (known after apply)
      + arn                   = (known after apply)
      + domain_id             = (known after apply)
      + domain_name           = "testing"
      + elasticsearch_version = "OpenSearch_1.1"
      + endpoint              = (known after apply)
      + id                    = (known after apply)
      + kibana_endpoint       = (known after apply)
      + tags_all              = {
          + "Environment" = "Local"
          + "Service"     = "LocalStack"
        }

      + advanced_security_options {
          + enabled                        = (known after apply)
          + internal_user_database_enabled = (known after apply)

          + master_user_options {
              + master_user_arn      = (known after apply)
              + master_user_name     = (known after apply)
              + master_user_password = (sensitive value)
            }
        }

      + auto_tune_options {
          + desired_state       = (known after apply)
          + rollback_on_disable = (known after apply)

          + maintenance_schedule {
              + cron_expression_for_recurrence = (known after apply)
              + start_at                       = (known after apply)

              + duration {
                  + unit  = (known after apply)
                  + value = (known after apply)
                }
            }
        }

      + cluster_config {
          + dedicated_master_count   = 3
          + dedicated_master_enabled = true
          + dedicated_master_type    = "m3.xlarge.elasticsearch"
          + instance_count           = 3
          + instance_type            = "m3.xlarge.elasticsearch"
          + warm_count               = 3
          + warm_enabled             = true
          + warm_type                = "ultrawarm1.large.elasticsearch"

          + cold_storage_options {
              + enabled = (known after apply)
            }
        }

      + domain_endpoint_options {
          + custom_endpoint                 = (known after apply)
          + custom_endpoint_certificate_arn = (known after apply)
          + custom_endpoint_enabled         = (known after apply)
          + enforce_https                   = (known after apply)
          + tls_security_policy             = (known after apply)
        }

      + ebs_options {
          + ebs_enabled = true
          + volume_size = 512
          + volume_type = "gp2"
        }

      + encrypt_at_rest {
          + enabled    = true
          + kms_key_id = (known after apply)
        }

      + node_to_node_encryption {
          + enabled = true
        }
    }

  # aws_iam_service_linked_role.es[0] will be created
  + resource "aws_iam_service_linked_role" "es" {
      + arn              = (known after apply)
      + aws_service_name = "es.amazonaws.com"
      + create_date      = (known after apply)
      + id               = (known after apply)
      + name             = (known after apply)
      + path             = (known after apply)
      + tags_all         = {
          + "Environment" = "Local"
          + "Service"     = "LocalStack"
        }
      + unique_id        = (known after apply)
    }
```

Terraform apply output:
```
aws_iam_service_linked_role.es[0]: Creating...
aws_iam_service_linked_role.es[0]: Creation complete after 1s [id=arn:aws:iam::000000000000:role/aws-service-role/es.amazonaws.com/r-8427ee25]
aws_elasticsearch_domain.opensearch: Creating...
aws_elasticsearch_domain.opensearch: Still creating... [10s elapsed]
aws_elasticsearch_domain.opensearch: Still creating... [20s elapsed]
aws_elasticsearch_domain.opensearch: Still creating... [30s elapsed]
aws_elasticsearch_domain.opensearch: Creation complete after 35s [id=arn:aws:es:us-east-1:000000000000:domain/testing]
```

Error on terraform destroy:

```
 Error: error waiting for Elasticsearch Domain (arn:aws:es:us-east-1:000000000000:domain/testing) delete: : Domain not found: testing
│ 	status code: 409, request id: VCG59DH0QTJY4E96QBA50QRQHROVE9Y97OHKNH84FLPMU8KEM9Q3
```


I attached the providers.tf and the main.tf needed to reproduce the error with terraform. Both in a zip
[LocalStackOsIssue.zip](https://github.com/localstack/localstack/files/8939898/LocalStackOsIssue.zip)
.

### Environment

```markdown
- OS:macOS Monterrey 12.4
- LocalStack: latest
- Terraform: v1.2.2
- Terraform AWS Provider: hashicorp/aws 4.18.0
```


### Anything else?

After testing the same terraform files against aws and localstack we found this difference in the response of the **_GET /2015-01-01/es/domain/testing HTTP/1.1_** request that maybe is the root of the issue:

This is the response from AWS which works with terraform without problem:
```
---[ RESPONSE ]--------------------------------------
HTTP/1.1 409 Conflict
Content-Length: 39
Content-Type: application/json
Date: Wed, 15 Jun 2022 15:27:29 GMT
X-Amzn-Errortype: ResourceNotFoundException
X-Amzn-Requestid: fa4ds5f4-b5ef-4643-7d8f-689ec443b642
-----------------------------------------------------: timestamp=2022-06-15T17:27:30.735+0200
2022-06-15T17:27:30.735+0200 [DEBUG] provider.terraform-provider-aws_v4.18.0_x5: [aws-sdk-go] {"message":"Domain not found: testing"}: timestamp=2022-06-15T17:27:30.735+0200
2
```

This is the response from Localstack
```
---[ RESPONSE ]--------------------------------------
HTTP/1.1 409
Connection: close
Access-Control-Allow-Headers: authorization,cache-control,content-length,content-md5,content-type,etag,location,x-amz-acl,x-amz-content-sha256,x-amz-date,x-amz-request-id,x-amz-security-token,x-amz-tagging,x-amz-target,x-amz-user-agent,x-amz-version-id,x-amzn-requestid,x-localstack-target,amz-sdk-invocation-id,amz-sdk-request
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: etag,x-amz-version-id
Content-Type: application/json
Date: Wed, 15 Jun 2022 14:06:22 GMT
Server: hypercorn-h11
X-Amz-Request-Id: AQGT79VIOT0IP7L3C82N9JGH4OQFU0FHOW1SSLFGBXK5AJO9ZH6O
X-Amzn-Requestid: H9JVOYUA5R0OBS0JU8TIU2J4UHSJLTQ8DKFM412W34SUX67LV4P4


-----------------------------------------------------: timestamp=2022-06-15T16:06:22.090+0200
2022-06-15T16:06:22.090+0200 [DEBUG] provider.terraform-provider-aws_v4.18.0_x5: [aws-sdk-go] {"__type": "ResourceNotFoundException", "message": "Domain not found: testing"}: timestamp=2022-06-15T16:06:22.090+0200

```

As you can see, AWS is settinh the "X-Amzn-Errortype" header with the "ResourceNotFoundException" not in the body. We were looking into the terraform code and the AWS GO SDK and they use the header X-Amzn-Errortype to check if it is a ResourceNotFoundException. Maybe just adding that in Localstack is going to be enough to make it work. We also see in the localstack history that something similar was added to kinesis. 

This is a list of links following the possible trace of the error theory

- https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/service/opensearch/domain.go#L1034
- https://github.com/hashicorp/terraform-provider-aws/blob/a68d32b3cc5189ab3bc039dbeaf3836d75e8d0ee/internal/service/elasticsearch/wait.go#L104
- https://github.com/hashicorp/terraform-provider-aws/blob/a68d32b3cc5189ab3bc039dbeaf3836d75e8d0ee/internal/service/elasticsearch/find.go#L11
- https://github.com/hashicorp/terraform-provider-aws/blob/a68d32b3cc5189ab3bc039dbeaf3836d75e8d0ee/internal/tfresource/not_found_error.go#L31
- https://github.com/aws/aws-sdk-go/blob/main/service/elasticsearchservice/api.go#L1380
- https://github.com/aws/aws-sdk-go/blob/main/service/elasticsearchservice/api.go#L1396


Discussion:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [developer guide](https://docs.localstack.cloud/developer-guide/).

Resolving patch:
diff --git a/localstack/aws/protocol/serializer.py b/localstack/aws/protocol/serializer.py
--- a/localstack/aws/protocol/serializer.py
+++ b/localstack/aws/protocol/serializer.py
@@ -1105,7 +1105,12 @@ def _serialize_error(
         operation_model: OperationModel,
     ) -> None:
         # TODO handle error shapes with members
+        # TODO implement different service-specific serializer configurations
+        #   - currently we set both, the `__type` member as well as the `X-Amzn-Errortype` header
+        #   - the specification defines that it's either the __type field OR the header
+        #     (https://awslabs.github.io/smithy/1.0/spec/aws/aws-json-1_1-protocol.html#operation-error-serialization)
         body = {"__type": code}
+        response.headers["X-Amzn-Errortype"] = code
         message = self._get_error_message(error)
         if message is not None:
             body["message"] = message


Test patch:
diff --git a/tests/integration/test_opensearch.py b/tests/integration/test_opensearch.py
--- a/tests/integration/test_opensearch.py
+++ b/tests/integration/test_opensearch.py
@@ -288,6 +288,17 @@ def test_create_domain_with_invalid_name(self, opensearch_client):
             opensearch_client.create_domain(DomainName="abc#")  # no special characters allowed
         assert e.value.response["Error"]["Code"] == "ValidationException"
 
+    @pytest.mark.aws_validated
+    def test_exception_header_field(self, opensearch_client):
+        """Test if the error response correctly sets the error code in the headers (see #6304)."""
+        with pytest.raises(botocore.exceptions.ClientError) as e:
+            # use an invalid domain name to provoke an exception
+            opensearch_client.create_domain(DomainName="123")
+        assert (
+            e.value.response["ResponseMetadata"]["HTTPHeaders"]["x-amzn-errortype"]
+            == "ValidationException"
+        )
+
     def test_create_existing_domain_causes_exception(
         self, opensearch_client, opensearch_wait_for_cluster
     ):
diff --git a/tests/unit/aws/protocol/test_serializer.py b/tests/unit/aws/protocol/test_serializer.py
--- a/tests/unit/aws/protocol/test_serializer.py
+++ b/tests/unit/aws/protocol/test_serializer.py
@@ -31,7 +31,7 @@ def _botocore_serializer_integration_test(
     response: dict,
     status_code=200,
     expected_response_content: dict = None,
-):
+) -> dict:
     """
     Performs an integration test for the serializer using botocore as parser.
     It executes the following steps:
@@ -47,7 +47,7 @@ def _botocore_serializer_integration_test(
     :param status_code: Optional - expected status code of the response - defaults to 200
     :param expected_response_content: Optional - if the input data ("response") differs from the actually expected data
                                       (because f.e. it contains None values)
-    :return: None
+    :return: boto-parsed serialized response
     """
 
     # Load the appropriate service
@@ -94,7 +94,7 @@ def _botocore_error_serializer_integration_test(
     status_code: int,
     message: Optional[str],
     is_sender_fault: bool = False,
-):
+) -> dict:
     """
     Performs an integration test for the error serialization using botocore as parser.
     It executes the following steps:
@@ -111,7 +111,7 @@ def _botocore_error_serializer_integration_test(
                  "CloudFrontOriginAccessIdentityAlreadyExists")
     :param status_code: expected HTTP response status code
     :param message: expected error message
-    :return: None
+    :return: boto-parsed serialized error response
     """
 
     # Load the appropriate service
@@ -147,6 +147,7 @@ def _botocore_error_serializer_integration_test(
         assert type == "Sender"
     else:
         assert type is None
+    return parsed_response
 
 
 def test_rest_xml_serializer_cloudfront_with_botocore():
@@ -520,7 +521,7 @@ class UserPoolTaggingException(ServiceException):
         pass
 
     exception = UserPoolTaggingException("Exception message!")
-    _botocore_error_serializer_integration_test(
+    response = _botocore_error_serializer_integration_test(
         "cognito-idp",
         "CreateUserPool",
         exception,
@@ -529,6 +530,13 @@ class UserPoolTaggingException(ServiceException):
         "Exception message!",
     )
 
+    # some clients also expect the X-Amzn-Errortype header according to
+    # https://awslabs.github.io/smithy/1.0/spec/aws/aws-json-1_1-protocol.html#operation-error-serialization
+    assert (
+        response.get("ResponseMetadata", {}).get("HTTPHeaders", {}).get("x-amzn-errortype")
+        == "UserPoolTaggingException"
+    )
+
 
 def test_json_protocol_custom_error_serialization():
     exception = CommonServiceException(
@@ -536,7 +544,7 @@ def test_json_protocol_custom_error_serialization():
         "You shall not access this API! Sincerely, your friendly neighbourhood firefighter.",
         status_code=451,
     )
-    _botocore_error_serializer_integration_test(
+    response = _botocore_error_serializer_integration_test(
         "cognito-idp",
         "CreateUserPool",
         exception,
@@ -545,6 +553,13 @@ def test_json_protocol_custom_error_serialization():
         "You shall not access this API! Sincerely, your friendly neighbourhood firefighter.",
     )
 
+    # some clients also expect the X-Amzn-Errortype header according to
+    # https://awslabs.github.io/smithy/1.0/spec/aws/aws-json-1_1-protocol.html#operation-error-serialization
+    assert (
+        response.get("ResponseMetadata", {}).get("HTTPHeaders", {}).get("x-amzn-errortype")
+        == "APIAccessCensorship"
+    )
+
 
 def test_json_protocol_content_type_1_0():
     """AppRunner defines the jsonVersion 1.0, therefore the Content-Type needs to be application/x-amz-json-1.0."""
@@ -690,7 +705,7 @@ class ThrottledException(ServiceException):
         pass
 
     exception = ThrottledException("Exception message!")
-    _botocore_error_serializer_integration_test(
+    response = _botocore_error_serializer_integration_test(
         "xray",
         "UpdateSamplingRule",
         exception,
@@ -699,6 +714,13 @@ class ThrottledException(ServiceException):
         "Exception message!",
     )
 
+    # some clients also expect the X-Amzn-Errortype header according to
+    # https://awslabs.github.io/smithy/1.0/spec/aws/aws-restjson1-protocol.html#operation-error-serialization
+    assert (
+        response.get("ResponseMetadata", {}).get("HTTPHeaders", {}).get("x-amzn-errortype")
+        == "ThrottledException"
+    )
+
 
 def test_restjson_protocol_custom_error_serialization():
     exception = CommonServiceException(
@@ -706,7 +728,7 @@ def test_restjson_protocol_custom_error_serialization():
         "You shall not access this API! Sincerely, your friendly neighbourhood firefighter.",
         status_code=451,
     )
-    _botocore_error_serializer_integration_test(
+    response = _botocore_error_serializer_integration_test(
         "xray",
         "UpdateSamplingRule",
         exception,
@@ -715,6 +737,13 @@ def test_restjson_protocol_custom_error_serialization():
         "You shall not access this API! Sincerely, your friendly neighbourhood firefighter.",
     )
 
+    # some clients also expect the X-Amzn-Errortype header according to
+    # https://awslabs.github.io/smithy/1.0/spec/aws/aws-restjson1-protocol.html#operation-error-serialization
+    assert (
+        response.get("ResponseMetadata", {}).get("HTTPHeaders", {}).get("x-amzn-errortype")
+        == "APIAccessCensorship"
+    )
+
 
 def test_restjson_serializer_xray_with_botocore():
     parameters = {