Repository: getmoto/moto
Pull request #7905 (2024-07-30 12:17:37)

Feature: Add FailureLocation support to SageMaker Runtime
Not sure if that's a feature or a bugfix actually.

SageMaker `invoke_endpoint_async` is supposed to return [docs](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpointAsync.html#API_runtime_InvokeEndpointAsync_ResponseSyntax):
  - InferenceId
  - OutputLocation
  - FailureLocation  <-- this one is currently missing in Moto implementation

This makes code hard to test when we assume this field to be returned by AWS endpoint, and making it optional because cannot be properly mocked in tests does not look like a good direction.

---
  
Doing so, I added a specific endpoint as well to Moto backend.
Idea is that we were not really able to customize the (async) inference output that is stored in S3.
Previously, the output schema was forced to something like:
```python
{
    "Body": body,
    "ContentType": _type,
    "InvokedProductionVariant": variant,
    "CustomAttributes": attrs,
}
```

That's the expected format for the sync inference response, but should not be forced on async endpoint IMHO.

For example, if I expect my endpoint to produce and output like:
```JSON
{
  "bbox": "..."
  "text": "..."
}
```
Well we have to hack the test flow to make it work


gael-ft: Hi @bblommers, thanks for reviewing my proposal !

> How would you feel about keeping the existing endpoint, but adding another field to the supplied results to indicate whether a result is a failure or not?

I first thought I could do something similar.
But then the problem is the "result" payload:

```python
{
  "Body": "first body",
  "ContentType": "text/xml",
  "InvokedProductionVariant": "prod",
  "CustomAttributes": "my_attr",
},
```

Follows SageMaker real-time inference response 👍 
__But__ with SageMaker async inference, the "result" payload that will be stored in S3 __has no specific schema__.
So users basically push what they want/need.

As described above, I can consider my async inference output to be:

```python
{
  "bbox": "..."
  "text": "..."
}
```

And I am not expecting (in real world or moto) any `Body`, `ContentType`, ... fields.

---

To be honest, I considered as well doing some kind of condition to say:

> if async inference, then only upload `Body` field, and discard everything else

But then I just realized that I couldn't imagine any current existing test using this endpoint to validate async inference.
If they do, they must anyway (here is the PR to avoid it) override the S3 file right after so that it fits what they expect as result from their inference.

Given that statement above, I thought it was cleaner to have a second endpoint, to avoid having to set fields (because we expect them to be present in current Moto code) that will finally be completly discarded. (Talking about `ContentType`, `CustomAttributes`, ...)
bblommers: Ah, right, I understand - thank you for the explanation @gael-ft.

> But then I just realized that I couldn't imagine any current existing test using this endpoint to validate async inference.
>

Me neither, based on your explanation - but if there's one thing I've learned as a maintainer, is that people will always find a way to ~ab~use Moto in a way that I've never thought about. :slightly_smiling_face: 

So, second proposal: we add the async endpoint, but leave the current implementation as a fallback. People can/should use the proper async endpoint as it's supposed to be used, but people who somehow got the sync-endpoint to work can also still use it.

As a very rough draft:
```
if self.async_results_queue:
    # new implementation
    ...
elif self.results_queue:
    # existing implementation
    ...
```
Later on we can always phase out the old/existing implementation.

Does that work for you?
gael-ft: Sounds good to me !
I'll try to have a look at it in the coming days.
gael-ft: Hello @bblommers,

Added fallback to "sync" results queue, as discussed, and tested accordingly.

Patch:
diff --git a/docs/conf.py b/docs/conf.py
index 0c55adfa2..2c28acc06 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -13,9 +13,6 @@
 # All configuration values have a default; values that are commented out
 # serve to show the default.
 
-import sys
-import os
-import shlex
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
diff --git a/moto/autoscaling/models.py b/moto/autoscaling/models.py
index 4880b0a4e..241b3eaca 100644
--- a/moto/autoscaling/models.py
+++ b/moto/autoscaling/models.py
@@ -487,7 +487,7 @@ class FakeAutoScalingGroup(CloudFormationModel):
 
         self.metrics: List[str] = []
         self.warm_pool: Optional[FakeWarmPool] = None
-        self.created_time = created_time
+        self.created_time = created_time.isoformat()
 
     @property
     def tags(self) -> List[Dict[str, str]]:
diff --git a/moto/ec2/models/instances.py b/moto/ec2/models/instances.py
index 9ba06ba8e..b44784b98 100644
--- a/moto/ec2/models/instances.py
+++ b/moto/ec2/models/instances.py
@@ -173,6 +173,7 @@ class Instance(TaggedEC2Resource, BotoInstance, CloudFormationModel):
             private_ip=kwargs.get("private_ip"),
             associate_public_ip=self.associate_public_ip,
             security_groups=self.security_groups,
+            ipv6_address_count=kwargs.get("ipv6_address_count"),
         )
 
     @property
@@ -468,6 +469,7 @@ class Instance(TaggedEC2Resource, BotoInstance, CloudFormationModel):
         private_ip: Optional[str] = None,
         associate_public_ip: Optional[bool] = None,
         security_groups: Optional[List[SecurityGroup]] = None,
+        ipv6_address_count: Optional[int] = None,
     ) -> None:
         self.nics: Dict[int, NetworkInterface] = {}
         for nic in nic_spec:
@@ -541,6 +543,7 @@ class Instance(TaggedEC2Resource, BotoInstance, CloudFormationModel):
                     public_ip_auto_assign=nic.get("AssociatePublicIpAddress", False),
                     group_ids=group_ids,
                     delete_on_termination=nic.get("DeleteOnTermination") == "true",
+                    ipv6_address_count=ipv6_address_count,
                 )
 
             self.attach_eni(use_nic, device_index)
diff --git a/moto/ec2/models/security_groups.py b/moto/ec2/models/security_groups.py
index eac8cf5f2..c9e7b98f8 100644
--- a/moto/ec2/models/security_groups.py
+++ b/moto/ec2/models/security_groups.py
@@ -634,12 +634,17 @@ class SecurityGroupBackend:
         results = []
 
         if sg_rule_ids:
+            rules_sources = [
+                self.sg_old_ingress_ruls.items(),
+                self.sg_old_egress_ruls.items(),
+            ]
             # go thru all the rules in the backend to find a match
             for sg_rule_id in sg_rule_ids:
-                for sg in self.sg_old_ingress_ruls:
-                    for rule in self.sg_old_ingress_ruls[sg]:
-                        if rule.id == sg_rule_id:
-                            results.append(rule)
+                for rules in rules_sources:
+                    for sg, rules_list in rules:
+                        for rule in rules_list:
+                            if rule.id == sg_rule_id:
+                                results.append(rule)
 
             return results
 
diff --git a/moto/ec2/responses/instances.py b/moto/ec2/responses/instances.py
index 299e49d93..27e35b2b4 100644
--- a/moto/ec2/responses/instances.py
+++ b/moto/ec2/responses/instances.py
@@ -90,6 +90,7 @@ class InstanceResponse(EC2BaseResponse):
             "monitoring_state": "enabled"
             if self._get_param("Monitoring.Enabled") == "true"
             else "disabled",
+            "ipv6_address_count": self._get_int_param("Ipv6AddressCount"),
         }
         if len(kwargs["nics"]) and kwargs["subnet_id"]:
             raise InvalidParameterCombination(
@@ -600,6 +601,16 @@ INSTANCE_TEMPLATE = """<item>
                     <ipOwnerId>{{ account_id }}</ipOwnerId>
                   </association>
                 {% endif %}
+                {% if nic.ipv6_addresses %}
+                <ipv6AddressesSet>
+                  {% for ipv6 in nic.ipv6_addresses %}
+                  <item>
+                    <ipv6Address>{{ ipv6 }}</ipv6Address>
+                    <isPrimaryIpv6>false</isPrimaryIpv6>
+                  </item>
+                  {% endfor %}
+                </ipv6AddressesSet>
+                {% endif %}
                 <privateIpAddressesSet>
                   <item>
                     <privateIpAddress>{{ nic.private_ip_address }}</privateIpAddress>
diff --git a/moto/moto_api/_internal/models.py b/moto/moto_api/_internal/models.py
index ba5241aa4..2ab8861ce 100644
--- a/moto/moto_api/_internal/models.py
+++ b/moto/moto_api/_internal/models.py
@@ -84,6 +84,18 @@ class MotoAPIBackend(BaseBackend):
         backend = sagemakerruntime_backends[account_id][region]
         backend.results_queue.append((body, content_type, prod_variant, custom_attrs))
 
+    def set_sagemaker_async_result(
+        self,
+        is_failure: bool,
+        data: str,
+        account_id: str,
+        region: str,
+    ) -> None:
+        from moto.sagemakerruntime.models import sagemakerruntime_backends
+
+        backend = sagemakerruntime_backends[account_id][region]
+        backend.async_results_queue.append((is_failure, data))
+
     def set_rds_data_result(
         self,
         records: Optional[List[List[Dict[str, Any]]]],
diff --git a/moto/moto_api/_internal/responses.py b/moto/moto_api/_internal/responses.py
index fdd873577..dd5712475 100644
--- a/moto/moto_api/_internal/responses.py
+++ b/moto/moto_api/_internal/responses.py
@@ -215,6 +215,29 @@ class MotoAPIResponse(BaseResponse):
             )
         return 201, {}, ""
 
+    def set_sagemaker_async_result(
+        self,
+        request: Any,
+        full_url: str,  # pylint: disable=unused-argument
+        headers: Any,
+    ) -> TYPE_RESPONSE:
+        from .models import moto_api_backend
+
+        body = self._get_body(headers, request)
+        account_id = body.get("account_id", DEFAULT_ACCOUNT_ID)
+        region = body.get("region", "us-east-1")
+
+        for result in body.get("results", []):
+            data = result["data"]
+            is_failure = result.get("is_failure", False)
+            moto_api_backend.set_sagemaker_async_result(
+                data=data,
+                is_failure=is_failure,
+                account_id=account_id,
+                region=region,
+            )
+        return 201, {}, ""
+
     def set_sagemaker_result(
         self,
         request: Any,
diff --git a/moto/moto_api/_internal/urls.py b/moto/moto_api/_internal/urls.py
index 7821f11b5..9f3e66f18 100644
--- a/moto/moto_api/_internal/urls.py
+++ b/moto/moto_api/_internal/urls.py
@@ -20,6 +20,7 @@ url_paths = {
     "{0}/moto-api/static/lambda-simple/response": response_instance.set_lambda_simple_result,
     "{0}/moto-api/static/resilience-hub-assessments/response": response_instance.set_resilience_result,
     "{0}/moto-api/static/sagemaker/endpoint-results": response_instance.set_sagemaker_result,
+    "{0}/moto-api/static/sagemaker/async-endpoint-results": response_instance.set_sagemaker_async_result,
     "{0}/moto-api/static/timestream/query-results": response_instance.set_timestream_result,
     "{0}/moto-api/static/rds-data/statement-results": response_instance.set_rds_data_result,
     "{0}/moto-api/state-manager/get-transition": response_instance.get_transition,
diff --git a/moto/sagemakerruntime/models.py b/moto/sagemakerruntime/models.py
index 45c05c13e..d17863193 100644
--- a/moto/sagemakerruntime/models.py
+++ b/moto/sagemakerruntime/models.py
@@ -10,9 +10,10 @@ class SageMakerRuntimeBackend(BaseBackend):
 
     def __init__(self, region_name: str, account_id: str):
         super().__init__(region_name, account_id)
-        self.async_results: Dict[str, Dict[str, str]] = {}
+        self.async_results: Dict[str, Dict[str, Tuple[str, str]]] = {}
         self.results: Dict[str, Dict[bytes, Tuple[str, str, str, str]]] = {}
         self.results_queue: List[Tuple[str, str, str, str]] = []
+        self.async_results_queue: List[Tuple[bool, str]] = []
 
     def invoke_endpoint(
         self, endpoint_name: str, unique_repr: bytes
@@ -65,35 +66,81 @@ class SageMakerRuntimeBackend(BaseBackend):
             )
         return self.results[endpoint_name][unique_repr]
 
-    def invoke_endpoint_async(self, endpoint_name: str, input_location: str) -> str:
+    def invoke_endpoint_async(
+        self, endpoint_name: str, input_location: str
+    ) -> Tuple[str, str]:
+        """
+        This call will return static data by default.
+
+        You can use a dedicated API to override this, by configuring a queue of expected results.
+        Subsequent requests using the same `InputLocation` will return the same result.
+        Other requests using a different `InputLocation` will take the next result from the queue.
+        Please note that for backward compatibility, if the async queue is empty, the sync queue will be used, if configured.
+
+        Configuring this queue by making an HTTP request to `/moto-api/static/sagemaker/async-endpoint-results`.
+        An example invocation looks like this:
+
+        .. sourcecode:: python
+
+            expected_results = {
+                "account_id": "123456789012",  # This is the default - can be omitted
+                "region": "us-east-1",  # This is the default - can be omitted
+                "results": [
+                    {
+                        "data": json.dumps({"first": "output"}),
+                    },
+                    {
+                        "is_failure": True,
+                        "data": "second inference failed",
+                    },
+                    # other results as required
+                ],
+            }
+            requests.post(
+                "http://motoapi.amazonaws.com/moto-api/static/sagemaker/async-endpoint-results",
+                json=expected_results,
+            )
+
+            client = boto3.client("sagemaker-runtime", region_name="us-east-1")
+            details = client.invoke_endpoint_async(EndpointName="asdf", InputLocation="qwer")
+
+        """
         if endpoint_name not in self.async_results:
             self.async_results[endpoint_name] = {}
         if input_location in self.async_results[endpoint_name]:
             return self.async_results[endpoint_name][input_location]
-        if self.results_queue:
+        if self.async_results_queue:
+            is_failure, data = self.async_results_queue.pop(0)
+        elif self.results_queue:  # Backward compatibility
+            is_failure = False
             body, _type, variant, attrs = self.results_queue.pop(0)
+            data = json.dumps(
+                {
+                    "Body": body,
+                    "ContentType": _type,
+                    "InvokedProductionVariant": variant,
+                    "CustomAttributes": attrs,
+                }
+            )
         else:
-            body = "body"
-            _type = "content_type"
-            variant = "invoked_production_variant"
-            attrs = "custom_attributes"
-        json_data = {
-            "Body": body,
-            "ContentType": _type,
-            "InvokedProductionVariant": variant,
-            "CustomAttributes": attrs,
-        }
-        output = json.dumps(json_data).encode("utf-8")
+            is_failure = False
+            data = json.dumps({"default": "response"})
 
         output_bucket = f"sagemaker-output-{random.uuid4()}"
         output_location = "response.json"
+        failure_location = "failure.json"
+        result_location = failure_location if is_failure else output_location
         from moto.s3.models import s3_backends
 
         s3_backend = s3_backends[self.account_id][self.partition]
         s3_backend.create_bucket(output_bucket, region_name=self.region_name)
-        s3_backend.put_object(output_bucket, output_location, value=output)
+        s3_backend.put_object(
+            output_bucket, result_location, value=data.encode("utf-8")
+        )
+
         self.async_results[endpoint_name][input_location] = (
-            f"s3://{output_bucket}/{output_location}"
+            f"s3://{output_bucket}/{output_location}",
+            f"s3://{output_bucket}/{failure_location}",
         )
         return self.async_results[endpoint_name][input_location]
 
diff --git a/moto/sagemakerruntime/responses.py b/moto/sagemakerruntime/responses.py
index c30c4b2b6..f409325c5 100644
--- a/moto/sagemakerruntime/responses.py
+++ b/moto/sagemakerruntime/responses.py
@@ -49,9 +49,14 @@ class SageMakerRuntimeResponse(BaseResponse):
         endpoint_name = self.path.split("/")[2]
         input_location = self.headers.get("X-Amzn-SageMaker-InputLocation")
         inference_id = self.headers.get("X-Amzn-SageMaker-Inference-Id")
-        location = self.sagemakerruntime_backend.invoke_endpoint_async(
-            endpoint_name, input_location
+        output_location, failure_location = (
+            self.sagemakerruntime_backend.invoke_endpoint_async(
+                endpoint_name, input_location
+            )
         )
         resp = {"InferenceId": inference_id or str(random.uuid4())}
-        headers = {"X-Amzn-SageMaker-OutputLocation": location}
+        headers = {
+            "X-Amzn-SageMaker-OutputLocation": output_location,
+            "X-Amzn-SageMaker-FailureLocation": failure_location,
+        }
         return 200, headers, json.dumps(resp)
diff --git a/other_langs/terraform/autoscaling/data.tf b/other_langs/terraform/autoscaling/data.tf
new file mode 100644
index 000000000..3d71c6420
--- /dev/null
+++ b/other_langs/terraform/autoscaling/data.tf
@@ -0,0 +1,17 @@
+data "aws_caller_identity" "this" {}
+
+data "aws_vpc" "this" {
+  default = true
+}
+
+data "aws_subnets" "this" {
+  filter {
+    name   = "vpc-id"
+    values = [data.aws_vpc.this.id]
+  }
+}
+
+data "aws_subnet" "this" {
+  for_each = toset(data.aws_subnets.this.ids)
+  id       = each.value
+}
diff --git a/other_langs/terraform/autoscaling/main.tf b/other_langs/terraform/autoscaling/main.tf
new file mode 100644
index 000000000..be172a56e
--- /dev/null
+++ b/other_langs/terraform/autoscaling/main.tf
@@ -0,0 +1,33 @@
+resource "tls_private_key" "this" {
+  algorithm = "RSA"
+  rsa_bits  = 4096
+}
+
+resource "aws_key_pair" "this" {
+  key_name   = var.app_name
+  public_key = tls_private_key.this.public_key_openssh
+}
+
+resource "aws_launch_template" "this" {
+  name                   = var.app_name
+  image_id               = var.ami_id
+  instance_type          = var.instance_type
+  key_name               = aws_key_pair.this.key_name
+  vpc_security_group_ids = [aws_security_group.this.id]
+}
+
+resource "aws_autoscaling_group" "this" {
+  name                  = var.app_name
+  desired_capacity      = 10
+  min_size              = 1
+  max_size              = 50
+  health_check_type     = "EC2"
+  vpc_zone_identifier   = [for subnet_id in data.aws_subnet.this : subnet_id.id]
+
+  wait_for_capacity_timeout = 0
+
+  launch_template {
+    id      = aws_launch_template.this.id
+    version = aws_launch_template.this.latest_version
+  }
+}
diff --git a/other_langs/terraform/autoscaling/provider.tf b/other_langs/terraform/autoscaling/provider.tf
new file mode 100644
index 000000000..d14eeaf62
--- /dev/null
+++ b/other_langs/terraform/autoscaling/provider.tf
@@ -0,0 +1,23 @@
+terraform {
+  required_providers {
+    aws = {
+      source = "hashicorp/aws"
+    }
+  }
+}
+
+provider "aws" {
+  region                      = var.aws_region
+  skip_credentials_validation = true
+  skip_metadata_api_check     = true
+  skip_requesting_account_id  = true
+
+  endpoints {
+   autoscaling = "http://localhost:5000"
+    ec2        = "http://localhost:5000"
+    sts        = "http://localhost:5000"
+  }
+
+  access_key = "my-access-key"
+  secret_key = "my-access-key"
+}
diff --git a/other_langs/terraform/autoscaling/security_groups.tf b/other_langs/terraform/autoscaling/security_groups.tf
new file mode 100644
index 000000000..7d09f626c
--- /dev/null
+++ b/other_langs/terraform/autoscaling/security_groups.tf
@@ -0,0 +1,24 @@
+resource "aws_security_group" "this" {
+  name        = var.app_name
+  vpc_id      = data.aws_vpc.this.id
+
+  ingress {
+    description = "SSH ingress"
+    from_port   = 22
+    to_port     = 22
+    protocol    = "tcp"
+    cidr_blocks = [for cidr in data.aws_subnet.this : cidr.cidr_block]
+  }
+
+  egress {
+    from_port   = 0
+    to_port     = 0
+    protocol    = "-1"
+    cidr_blocks = ["0.0.0.0/0"]
+  }
+
+  tags = {
+    Name = var.app_name
+  }
+
+}
diff --git a/other_langs/terraform/autoscaling/variables.tf b/other_langs/terraform/autoscaling/variables.tf
new file mode 100644
index 000000000..90f7d1b4f
--- /dev/null
+++ b/other_langs/terraform/autoscaling/variables.tf
@@ -0,0 +1,23 @@
+variable "aws_region" {
+  description = "The AWS Region to use."
+  type        = string
+  default     = "us-east-1"
+}
+
+variable "app_name" {
+  description = "The application name."
+  type        = string
+  default     = "sandwich"
+}
+
+variable "ami_id" {
+  description = "The AMI to use."
+  type        = string
+  default     = "ami-12c6146b"
+}
+
+variable "instance_type" {
+  description = "The instance type to use."
+  type        = string
+  default     = "t2.nano"
+}
diff --git a/update_version_from_git.py b/update_version_from_git.py
index a6f644651..3e26e055a 100644
--- a/update_version_from_git.py
+++ b/update_version_from_git.py
@@ -26,6 +26,7 @@ import os
 import re
 import subprocess
 import sys
+
 from packaging.version import Version
 
 


Test patch:
diff --git a/.github/workflows/tests_terraform_examples.yml b/.github/workflows/tests_terraform_examples.yml
index b24d748f6..82cf15891 100644
--- a/.github/workflows/tests_terraform_examples.yml
+++ b/.github/workflows/tests_terraform_examples.yml
@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        service: ["acm", "awslambda", "cloudfront", "codebuild", "ds", "elb", "iam", "rds", "route53", "sqs"]
+        service: ["acm", "autoscaling", "awslambda", "cloudfront", "codebuild", "ds", "elb", "iam", "rds", "route53", "sqs"]
 
     steps:
     - uses: actions/checkout@v4
diff --git a/tests/test_autoscaling/test_autoscaling.py b/tests/test_autoscaling/test_autoscaling.py
index 614b75e79..5a040dbc3 100644
--- a/tests/test_autoscaling/test_autoscaling.py
+++ b/tests/test_autoscaling/test_autoscaling.py
@@ -1364,6 +1364,6 @@ def test_sets_created_time():
 
     asgs = conn.describe_auto_scaling_groups()["AutoScalingGroups"]
     assert len(asgs) == 1
-    assert asgs[0]["CreatedTime"] != datetime.strptime(
-        "2013-05-06T17:47:15.107Z", "%Y-%m-%dT%H:%M:%S.%f%z"
+    assert asgs[0]["CreatedTime"].strftime("%Y %m %d") == datetime.now().strftime(
+        "%Y %m %d"
     )
diff --git a/tests/test_ec2/test_instances.py b/tests/test_ec2/test_instances.py
index 7cd80dcbb..1273430a5 100644
--- a/tests/test_ec2/test_instances.py
+++ b/tests/test_ec2/test_instances.py
@@ -2735,3 +2735,44 @@ def test_describe_instance_with_enhanced_monitoring():
     )["Reservations"][0]["Instances"]
 
     assert result[0]["Monitoring"] == {"State": "disabled"}
+
+
+@mock_aws
+def test_instance_with_ipv6_address():
+    ec2_client = boto3.client("ec2", "us-east-1")
+    # Build a VPC.
+    vpc = ec2_client.create_vpc(
+        CidrBlock="10.0.0.0/16",
+        AmazonProvidedIpv6CidrBlock=True,
+    )["Vpc"]
+
+    # The IPv6 CIDR for the subnet needs to be /64; the block associated with the VPC will be /56.
+    ipv6_subnet_cidr = vpc["Ipv6CidrBlockAssociationSet"][0]["Ipv6CidrBlock"]
+    ipv6_subnet_cidr = ipv6_subnet_cidr[:-2] + "64"
+
+    # Build a subnet.
+    subnet = ec2_client.create_subnet(
+        VpcId=vpc["VpcId"], CidrBlock="10.0.0.0/24", Ipv6CidrBlock=ipv6_subnet_cidr
+    )["Subnet"]
+    assert "Ipv6CidrBlockAssociationSet" in subnet
+
+    # Build a security group.
+    security_group_id = ec2_client.create_security_group(
+        GroupName="ipv6-test",
+        Description="Test security group.",
+        VpcId=vpc["VpcId"],
+    )["GroupId"]
+
+    # Launch the instance.
+    instance = ec2_client.run_instances(
+        ImageId=EXAMPLE_AMI_ID,
+        InstanceType="t3.nano",
+        MinCount=1,
+        MaxCount=1,
+        Ipv6AddressCount=1,
+        SecurityGroupIds=[security_group_id],
+        SubnetId=subnet["SubnetId"],
+    )["Instances"][0]
+
+    assert len(instance["NetworkInterfaces"]) == 1
+    assert len(instance["NetworkInterfaces"][0]["Ipv6Addresses"]) == 1
diff --git a/tests/test_ec2/test_security_groups.py b/tests/test_ec2/test_security_groups.py
index 519b5c2c6..2dc3c797b 100644
--- a/tests/test_ec2/test_security_groups.py
+++ b/tests/test_ec2/test_security_groups.py
@@ -684,6 +684,7 @@ def test_create_and_describe_security_grp_rule():
         Description="Test SG", GroupName=sg_name, VpcId=vpc.id
     )
 
+    # check that the default rule is present using filters
     response = client.describe_security_group_rules(
         Filters=[{"Name": "group-id", "Values": [sg["GroupId"]]}]
     )
@@ -692,11 +693,21 @@ def test_create_and_describe_security_grp_rule():
     # Only the default rule is present
     assert len(rules) == 1
 
+    def _verify_egress_rule(rule):
+        assert rule["IsEgress"] is True
+        assert rule["IpProtocol"] == "-1"
+        assert rule["CidrIpv4"] == "0.0.0.0/0"
+        assert "GroupId" in rule
+
     # Test default egress rule content
-    assert rules[0]["IsEgress"] is True
-    assert rules[0]["IpProtocol"] == "-1"
-    assert rules[0]["CidrIpv4"] == "0.0.0.0/0"
-    assert "GroupId" in rules[0]
+    _verify_egress_rule(rules[0])
+
+    # check that the default rule is present using security group rule ids
+    response = client.describe_security_group_rules(
+        SecurityGroupRuleIds=[rules[0]["SecurityGroupRuleId"]]
+    )
+    rules = response["SecurityGroupRules"]
+    _verify_egress_rule(rules[0])
 
 
 @mock_aws
diff --git a/tests/test_sagemakerruntime/test_sagemakerruntime.py b/tests/test_sagemakerruntime/test_sagemakerruntime.py
index 0d6633a8b..d30ad1dae 100644
--- a/tests/test_sagemakerruntime/test_sagemakerruntime.py
+++ b/tests/test_sagemakerruntime/test_sagemakerruntime.py
@@ -68,37 +68,76 @@ def test_invoke_endpoint_async():
 
     sagemaker_result = {
         "results": [
+            {"data": json.dumps({"first": "output"})},
             {
-                "Body": "first body",
-                "ContentType": "text/xml",
-                "InvokedProductionVariant": "prod",
-                "CustomAttributes": "my_attr",
+                "is_failure": True,
+                "data": "second failure",
             },
-            {"Body": "second body"},
         ]
     }
     requests.post(
-        f"http://{base_url}/moto-api/static/sagemaker/endpoint-results",
+        f"http://{base_url}/moto-api/static/sagemaker/async-endpoint-results",
         json=sagemaker_result,
     )
 
     # Return the first item from the list
     body = client.invoke_endpoint_async(EndpointName="asdf", InputLocation="qwer")
     first_output_location = body["OutputLocation"]
+    first_failure_location = body["FailureLocation"]
 
     # Same input -> same output
     body = client.invoke_endpoint_async(EndpointName="asdf", InputLocation="qwer")
     assert body["OutputLocation"] == first_output_location
+    assert body["FailureLocation"] == first_failure_location
+
+    s3 = boto3.client("s3", "us-east-1")
+    bucket_name, obj = bucket_and_name_from_url(first_output_location)
+    resp = s3.get_object(Bucket=bucket_name, Key=obj)
+    resp = json.loads(resp["Body"].read().decode("utf-8"))
+    assert resp == {"first": "output"}
 
     # Different input -> second item
     body = client.invoke_endpoint_async(
         EndpointName="asdf", InputLocation="asf", InferenceId="sth"
     )
-    second_output_location = body["OutputLocation"]
+    second_failure_location = body["FailureLocation"]
     assert body["InferenceId"] == "sth"
 
+    bucket_name, obj = bucket_and_name_from_url(second_failure_location)
+    resp = s3.get_object(Bucket=bucket_name, Key=obj)
+    resp = resp["Body"].read().decode("utf-8")
+    assert resp == "second failure"
+
+
+@mock_aws
+def test_invoke_endpoint_async_should_read_sync_queue_if_async_not_configured():
+    client = boto3.client("sagemaker-runtime", region_name="us-east-1")
+    base_url = (
+        "localhost:5000" if settings.TEST_SERVER_MODE else "motoapi.amazonaws.com"
+    )
+
+    sagemaker_result = {
+        "results": [
+            {"Body": "support sync queue for backward compatibility"},
+        ]
+    }
+    requests.post(
+        f"http://{base_url}/moto-api/static/sagemaker/endpoint-results",
+        json=sagemaker_result,
+    )
+
+    # Return the first item from the list
+    body = client.invoke_endpoint_async(EndpointName="asdf", InputLocation="qwer")
+    first_output_location = body["OutputLocation"]
+    first_failure_location = body["FailureLocation"]
+
+    # Same input -> same output
+    body = client.invoke_endpoint_async(EndpointName="asdf", InputLocation="qwer")
+    assert body["OutputLocation"] == first_output_location
+    assert body["FailureLocation"] == first_failure_location
+
     s3 = boto3.client("s3", "us-east-1")
-    bucket_name, obj = bucket_and_name_from_url(second_output_location)
+    bucket_name, obj = bucket_and_name_from_url(first_output_location)
     resp = s3.get_object(Bucket=bucket_name, Key=obj)
     resp = json.loads(resp["Body"].read().decode("utf-8"))
-    assert resp["Body"] == "second body"
+    assert resp["Body"] == "support sync queue for backward compatibility"