Repository: getmoto/moto

Issue:
ssm put_parameter add to default
Hi,

I have been trying to understand how to add the following ssm parameter to the default sets provided in https://github.com/spulec/moto/blob/master/moto/ssm/resources/ami-amazon-linux-latest/eu-west-1.json:

- `/aws/service/ecs/optimized-ami/amazon-linux-2/recommended`

But obviously this isn't possible by just calling `put_parameter` as showing in https://github.com/spulec/moto/blob/master/tests/test_ssm/test_ssm_boto3.py#L390

Is there an obvious way to do this with moto, or do I need to mock the ssm client without moto?

The result value from aws cli should be the following (at the time or writing):

```json
{
    "Parameter": {
        "Name": "/aws/service/ecs/optimized-ami/amazon-linux-2/recommended",
        "Type": "String",
        "Value": "{\"ecs_agent_version\":\"1.63.1\",\"ecs_runtime_version\":\"Docker version 20.10.13\",\"image_id\":\"ami-002e2fef4b94f8fd0\",\"image_name\":\"amzn2-ami-ecs-hvm-2.0.20220921-x86_64-ebs\",\"image_version\":\"2.0.20220921\",\"os\":\"Amazon Linux 2\",\"schema_version\":1,\"source_image_name\":\"amzn2-ami-minimal-hvm-2.0.20220912.1-x86_64-ebs\"}",
        "Version": 94,
        "LastModifiedDate": 1664230158.399,
        "ARN": "arn:aws:ssm:eu-west-1::parameter/aws/service/ecs/optimized-ami/amazon-linux-2/recommended",
        "DataType": "text"
    }
}
```


Discussion:
Hi @thoroc, there is no way to do this using Moto at the moment, unfortunately.

I'll mark it as an enhancement for Moto to store these parameters (similarly to what we do for `/aws/service/ami-amazon-linux-latest` and `/aws/service/global-infrastructure`-params).
Hi @bblommers,

Thank you for confirming that. I'll explore other avenue to run my tests.
Btw are you thinking about doing the same as for this PR: https://github.com/spulec/moto/pull/5427?
Yes, that's the idea @thoroc. If you're looking to tackle this yourself, PR's are always welcome!
Where would it make more sense to save this data from AWS? I was thinking of retrieving everything under `/aws/service/ecs/optimized-ami/`.

I have tried to retrieve everything under `/aws/service/ecs/` and everything under `/aws/service/ecs/optimized-ami/` and comparing the output look the same.
The exact location where to store the data doesn't matter much, as long as the naming is sensible, for example `moto/ssm/resources/ecs/`.

If there is no difference, than we might as well retrieve everything under `/aws/service/ecs` - if AWS decides to add more data, we're already set-up to retrieve it.
Ok I was wrong, somehow I have managed to retrieve a subset only with a python script (I think I needed to use the paginator). Using the aws cli there are a lot more items calling `/aws/service/ecs` than calling `/aws/service/ecs/optimized-ami`, which makes sense.
FYI @thoroc , our `scripts` directory has some examples on calling SSM with a paginator. You can always copy/pasting one of those.
Ok I was doing something stupid, but the result is that I can confirm there is currently as many parameters in `/aws/service/ecs/` as in `/aws/service/ecs/optimized-ami`.

To confirm this I have run the following commands:
```sh
λ aws ssm get-parameters-by-path --path=/aws/service/ecs/optimized-ami --recursive | grep ARN | wc -l
3990
```
and
```sh
λ aws ssm get-parameters-by-path --path=/aws/service/ecs --recursive | grep ARN | wc -l
3990
```

That means the Python code was returning the correct values. Sorry for the confusion.
> You can always copy/pasting one of those.

Ah I see. One thing I am wondering though. Is it the correct thing to convert the field `LastModifiedDate` from a string to a timestamp? the response from the AWS cli is a string after all for this field.

```sh
λ aws ssm get-parameter --name=/aws/service/ecs/optimized-ami/amazon-linux-2/ami-04030f151d88f43bc
{
    "Parameter": {
        "Name": "/aws/service/ecs/optimized-ami/amazon-linux-2/ami-04030f151d88f43bc",
        "Type": "String",
        "Value": "{\"ecs_agent_version\":\"1.62.2\",\"ecs_runtime_version\":\"Docker version 20.10.13\",\"image_id\":\"ami-04030f151d88f43bc\",\"image_name\":\"amzn2-ami-ecs-hvm-2.0.20220822-x86_64-ebs\",\"image_version\":\"2.0.20220822\",\"os\":\"Amazon Linux 2\",\"schema_version\":1,\"source_image_name\":\"amzn2-ami-minimal-hvm-2.0.20220805.0-x86_64-ebs\"}",
        "Version": 1,
        "LastModifiedDate": "2022-08-24T20:46:23.449000+01:00",
        "ARN": "arn:aws:ssm:eu-west-1::parameter/aws/service/ecs/optimized-ami/amazon-linux-2/ami-04030f151d88f43bc",
        "DataType": "text"
    }
}
```
Moto intercepts the underlying HTTP request that is send by the SDK, and the AWS CLI/boto3 will then convert the HTTP response to whatever format they want. I haven't checked, but I'm assuming that the HTTP response from AWS contains a timestamp, so that's what Moto should return as well

Resolving patch:
diff --git a/moto/ec2/models/amis.py b/moto/ec2/models/amis.py
--- a/moto/ec2/models/amis.py
+++ b/moto/ec2/models/amis.py
@@ -159,17 +159,18 @@ def _load_amis(self) -> None:
             ami["owner_alias"] = "amazon"
             self.amis[ami_id] = Ami(self, **ami)
         if "MOTO_AMIS_PATH" not in environ:
-            try:
-                latest_amis = load_resource(
-                    __name__, f"../resources/latest_amis/{self.region_name}.json"  # type: ignore[attr-defined]
-                )
-                for ami in latest_amis:
-                    ami_id = ami["ami_id"]
-                    ami["owner_alias"] = "amazon"
-                    self.amis[ami_id] = Ami(self, **ami)
-            except FileNotFoundError:
-                # Will error on unknown (new) regions - just return an empty list here
-                pass
+            for path in ["latest_amis", "ecs/optimized_amis"]:
+                try:
+                    latest_amis = load_resource(
+                        __name__, f"../resources/{path}/{self.region_name}.json"  # type: ignore[attr-defined]
+                    )
+                    for ami in latest_amis:
+                        ami_id = ami["ami_id"]
+                        ami["owner_alias"] = "amazon"
+                        self.amis[ami_id] = Ami(self, **ami)
+                except FileNotFoundError:
+                    # Will error on unknown (new) regions - just return an empty list here
+                    pass
 
     def create_image(
         self,
diff --git a/moto/ssm/models.py b/moto/ssm/models.py
--- a/moto/ssm/models.py
+++ b/moto/ssm/models.py
@@ -47,15 +47,35 @@ def __init__(self, account_id, region_name):
         # each value is a list of all of the versions for a parameter
         # to get the current value, grab the last item of the list
         super().__init__(list)
-        self.parameters_loaded = False
+        self.latest_amis_loaded = False
+        self.latest_region_defaults_loaded = False
+        self.latest_service_defaults_loaded = False
+        self.latest_ecs_amis_loaded = False
         self.account_id = account_id
         self.region_name = region_name
 
     def _check_loading_status(self, key):
-        if not self.parameters_loaded and key and str(key).startswith("/aws"):
-            self._load_global_parameters()
+        key = str(key or "")
+        if key.startswith("/aws/service/ami-amazon-linux-latest"):
+            if not self.latest_amis_loaded:
+                self._load_latest_amis()
+                self.latest_amis_loaded = True
+        if key.startswith("/aws/service/global-infrastructure/regions"):
+            if not self.latest_region_defaults_loaded:
+                self._load_tree_parameters(path="resources/regions.json")
+                self.latest_region_defaults_loaded = True
+        if key.startswith("/aws/service/global-infrastructure/services"):
+            if not self.latest_service_defaults_loaded:
+                self._load_tree_parameters(path="resources/services.json")
+                self.latest_service_defaults_loaded = True
+        if key.startswith("/aws/service/ecs/optimized-ami"):
+            if not self.latest_ecs_amis_loaded:
+                self._load_tree_parameters(
+                    f"resources/ecs/optimized_amis/{self.region_name}.json"
+                )
+                self.latest_ecs_amis_loaded = True
 
-    def _load_global_parameters(self):
+    def _load_latest_amis(self):
         try:
             latest_amis_linux = load_resource(
                 __name__, f"resources/ami-amazon-linux-latest/{self.region_name}.json"
@@ -78,11 +98,12 @@ def _load_global_parameters(self):
                     data_type=param["DataType"],
                 )
             )
-        regions = load_resource(__name__, "resources/regions.json")
-        services = load_resource(__name__, "resources/services.json")
-        params = []
-        params.extend(convert_to_params(regions))
-        params.extend(convert_to_params(services))
+
+    def _load_tree_parameters(self, path: str):
+        try:
+            params = convert_to_params(load_resource(__name__, path))
+        except FileNotFoundError:
+            params = []
 
         for param in params:
             last_modified_date = time.time()
@@ -105,7 +126,6 @@ def _load_global_parameters(self):
                     data_type="text",
                 )
             )
-        self.parameters_loaded = True
 
     def _get_secretsmanager_parameter(self, secret_name):
         secrets_backend = secretsmanager_backends[self.account_id][self.region_name]
diff --git a/scripts/ssm_get_optimized_amis.py b/scripts/ssm_get_optimized_amis.py
new file mode 100644
--- /dev/null
+++ b/scripts/ssm_get_optimized_amis.py
@@ -0,0 +1,98 @@
+import time
+import json
+from datetime import timezone
+from pathlib import Path
+
+import boto3
+import botocore
+
+from moto.core.utils import unix_time
+from moto.ec2.utils import gen_moto_amis
+from moto.ssm.utils import convert_to_tree
+
+session = boto3.Session()
+
+
+def retrieve_by_path(client, path):
+    print("Attempting to retrieve data for path={}", path)
+    response = client.get_parameters_by_path(Path=path, Recursive=True)
+    parameters = response["Parameters"]
+    next_token = response["NextToken"]
+    while next_token:
+        response = client.get_parameters_by_path(Path=path, Recursive=True, NextToken=next_token)
+        parameters.extend(response["Parameters"])
+        next_token = response.get("NextToken")
+
+    return parameters
+
+def save_to_file(destination_path: str, params: dict):
+    print("Attempting to save data to {}", destination_path)
+    file_path = Path(Path.cwd(), destination_path)
+    file_path.parent.mkdir(exist_ok=True, parents=True)
+
+    with file_path.open("w") as fb:
+        json.dump(params, fb, sort_keys=True, indent=2)
+
+
+def retrieve_ec2_data(image_ids: list, region: str):
+    ec2_client = session.client("ec2", region_name=region)
+
+    images = ec2_client.describe_images(ImageIds=image_ids)["Images"]
+
+    return gen_moto_amis(images)
+
+
+def main():
+    """
+    Retrieve the latest AMI-details from SSM
+     - Download from AWS
+     - Store this in the dedicated moto/ssm/resources-folder
+    """
+    for region in session.get_available_regions("ssm"):
+        ssm_client = session.client('ssm', region_name=region)
+
+        default_param_path = "/aws/service/ecs/optimized-ami"
+
+        # Retrieve default AMI values
+        try:
+            print("Retrieving data for {}" , region)
+
+            parameters = retrieve_by_path(ssm_client, default_param_path)
+
+            if not parameters:
+                continue
+
+            image_ids = []
+
+            for param in parameters:
+                param["LastModifiedDate"] = unix_time(param["LastModifiedDate"].astimezone(timezone.utc).replace(tzinfo=None))
+
+                if isinstance(param["Value"], str) and param["Value"][0] == "{":
+                    param["Value"] = json.loads(param["Value"])
+                    image_ids.append(param["Value"]["image_id"])
+
+                # We can recreate these params
+                param.pop("DataType", None)
+                param.pop("Type", None)
+                param.pop("ARN", None)
+
+            image_ids = list(set(image_ids))
+
+            tree = convert_to_tree(parameters)
+
+            destination_path = f"moto/ssm/resources/ecs/optimized_amis/{region}.json"
+            save_to_file(destination_path, tree)
+
+            # Retrieve details about AMIs from EC2
+            image_as_dicts = retrieve_ec2_data(image_ids, region)
+            destination_path = f"moto/ec2/resources/ecs/optimized_amis/{region}.json"
+            save_to_file(destination_path, image_as_dicts)
+
+            time.sleep(0.5)
+        except botocore.exceptions.ClientError as e:
+            print(e)
+            # We might encounter an error if we do not have access to a region - just ignore and try the next region
+
+
+if __name__ == "__main__":
+    main()


Test patch:
diff --git a/tests/test_ec2/test_amis.py b/tests/test_ec2/test_amis.py
--- a/tests/test_ec2/test_amis.py
+++ b/tests/test_ec2/test_amis.py
@@ -841,7 +841,7 @@ def test_filter_description():
         Owners=["amazon"],
         Filters=[{"Name": "description", "Values": ["Amazon Linux AMI*"]}],
     )["Images"]
-    resp.should.have.length_of(9)
+    len(resp).should.be.greater_than(9)
 
     # Search for full description
     resp = client.describe_images(
diff --git a/tests/test_ssm/test_ssm_defaults.py b/tests/test_ssm/test_ssm_defaults.py
--- a/tests/test_ssm/test_ssm_defaults.py
+++ b/tests/test_ssm/test_ssm_defaults.py
@@ -23,6 +23,16 @@ def test_ssm_get_by_path():
     pacific.should.have.key("LastModifiedDate")
 
 
+@mock_ssm
+def test_global_infrastructure_services():
+    client = boto3.client("ssm", region_name="us-west-1")
+    path = "/aws/service/global-infrastructure/services"
+    params = client.get_parameters_by_path(Path=path)["Parameters"]
+    params[0]["Name"].should.equal(
+        "/aws/service/global-infrastructure/services/accessanalyzer"
+    )
+
+
 @mock_ssm
 def test_ssm_region_query():
     client = boto3.client("ssm", region_name="us-west-1")
diff --git a/tests/test_ssm/test_ssm_ecs_images.py b/tests/test_ssm/test_ssm_ecs_images.py
new file mode 100644
--- /dev/null
+++ b/tests/test_ssm/test_ssm_ecs_images.py
@@ -0,0 +1,19 @@
+import boto3
+import sure  # noqa # pylint: disable=unused-import
+
+from moto import mock_ec2, mock_ssm
+
+
+@mock_ec2
+@mock_ssm
+def test_ssm_get_latest_ami_by_path():
+    ssm = boto3.client("ssm", region_name="us-east-1")
+    path = "/aws/service/ecs/optimized-ami"
+    params = ssm.get_parameters_by_path(Path=path, Recursive=True)["Parameters"]
+    params.should.have.length_of(10)
+
+    ec2 = boto3.client("ec2", region_name="us-east-1")
+    for param in params:
+        if "Value" in param and isinstance(param["Value"], dict):
+            ami = param["Value"]["image_id"]
+            ec2.describe_images(ImageIds=[ami])["Images"].should.have.length_of(1)