Repository: ansible-collections/amazon.aws

Issue:
aws_ssm lookup documentation disagrees with code
### Summary

In debugging some code of mine that uses the `aws_ssm` lookup I was studying its documentation and found a few errors.

- The documentation indicates [here](https://github.com/ansible-collections/amazon.aws/blob/main/plugins/lookup/aws_ssm.py#L26-L29) that the default value for `on_missing` is `skip`, and that is what it is set to [in the code](https://github.com/ansible-collections/amazon.aws/blob/main/plugins/lookup/aws_ssm.py#L176), but this disagrees with the default value explicitly listed [in the documentation](https://github.com/ansible-collections/amazon.aws/blob/main/plugins/lookup/aws_ssm.py#L63).
- The default value for `on_denied` is set to `skip` [in the code](https://github.com/ansible-collections/amazon.aws/blob/main/plugins/lookup/aws_ssm.py#L177), but this disagrees with the default value explicitly listed [in the documentation](https://github.com/ansible-collections/amazon.aws/blob/main/plugins/lookup/aws_ssm.py#L73).

### Issue Type

Documentation Report

### Component Name

aws_ssm

### Ansible Version

```console (paste below)
$ ansible --version
ansible [core 2.12.1]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/jeremy_frasier/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jeremy_frasier/.pyenv/versions/3.10.1/envs/assessor-portal-packer/lib/python3.10/site-packages/ansible
  ansible collection location = /home/jeremy_frasier/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/jeremy_frasier/.pyenv/versions/assessor-portal-packer/bin/ansible
  python version = 3.10.1 (main, Dec 14 2021, 13:00:00) [GCC 11.1.0]
  jinja version = 3.0.3
  libyaml = True
```


### Collection Versions

```console (paste below)
$ ansible-galaxy collection list

# /home/jeremy_frasier/.pyenv/versions/3.10.1/envs/assessor-portal-packer/lib/python3.10/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0  
ansible.netcommon             2.5.0  
ansible.posix                 1.3.0  
ansible.utils                 2.4.3  
ansible.windows               1.9.0  
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.10.0 
check_point.mgmt              2.2.0  
chocolatey.chocolatey         1.1.0  
cisco.aci                     2.1.0  
cisco.asa                     2.1.0  
cisco.intersight              1.0.18 
cisco.ios                     2.6.0  
cisco.iosxr                   2.6.0  
cisco.ise                     1.2.1  
cisco.meraki                  2.5.0  
cisco.mso                     1.3.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.8.2  
cisco.ucs                     1.6.0  
cloud.common                  2.1.0  
cloudscale_ch.cloud           2.2.0  
community.aws                 2.1.0  
community.azure               1.1.0  
community.ciscosmb            1.0.4  
community.crypto              2.0.2  
community.digitalocean        1.14.0 
community.dns                 2.0.3  
community.docker              2.0.2  
community.fortios             1.0.0  
community.general             4.2.0  
community.google              1.0.0  
community.grafana             1.3.0  
community.hashi_vault         2.1.0  
community.hrobot              1.2.1  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.2  
community.mongodb             1.3.2  
community.mysql               2.3.2  
community.network             3.0.0  
community.okd                 2.1.0  
community.postgresql          1.6.0  
community.proxysql            1.3.0  
community.rabbitmq            1.1.0  
community.routeros            2.0.0  
community.skydive             1.0.0  
community.sops                1.2.0  
community.vmware              1.17.0 
community.windows             1.9.0  
community.zabbix              1.5.1  
containers.podman             1.9.0  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.13 
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            4.3.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.13.0 
fortinet.fortimanager         2.1.4  
fortinet.fortios              2.1.3  
frr.frr                       1.0.3  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.4  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.3  
infoblox.nios_modules         1.2.1  
inspur.sm                     1.3.0  
junipernetworks.junos         2.8.0  
kubernetes.core               2.2.2  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.12.1
netapp.elementsw              21.7.0 
netapp.ontap                  21.14.1
netapp.storagegrid            21.9.0 
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.2.13 
netbox.netbox                 3.4.0  
ngine_io.cloudstack           2.2.2  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.5.3  
openvswitch.openvswitch       2.1.0  
ovirt.ovirt                   1.6.6  
purestorage.flasharray        1.11.0 
purestorage.flashblade        1.9.0  
sensu.sensu_go                1.12.1 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.26.0 
theforeman.foreman            2.2.0  
vyos.vyos                     2.6.0  
wti.remote                    1.0.3
```

### Configuration

```console (paste below)
$ ansible-config dump --only-changed
<no output>
```


### OS / Environment

Arch Linux, but the specific OS isn't relevant here

### Additional Information

_No response_

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/aws_s3.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/aws_s3.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @jillr @linabuzachis @lwade @s-hertel @tremble
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/plugins/lookup/aws_ssm.py b/plugins/lookup/aws_ssm.py
--- a/plugins/lookup/aws_ssm.py
+++ b/plugins/lookup/aws_ssm.py
@@ -23,18 +23,15 @@
     The first argument you pass the lookup can either be a parameter name or a hierarchy of
     parameters. Hierarchies start with a forward slash and end with the parameter name. Up to
     5 layers may be specified.
-  - If looking up an explicitly listed parameter by name which does not exist then the lookup will
-    return a None value which will be interpreted by Jinja2 as an empty string.  You can use the
-    ```default``` filter to give a default value in this case but must set the second parameter to
-    true (see examples below)
+  - If looking up an explicitly listed parameter by name which does not exist then the lookup
+    will generate an error. You can use the ```default``` filter to give a default value in
+    this case but must set the ```on_missing``` parameter to ```skip``` or ```warn```. You must
+    also set the second parameter of the ```default``` filter to ```true``` (see examples below).
   - When looking up a path for parameters under it a dictionary will be returned for each path.
-    If there is no parameter under that path then the return will be successful but the
-    dictionary will be empty.
+    If there is no parameter under that path then the lookup will generate an error.
   - If the lookup fails due to lack of permissions or due to an AWS client error then the aws_ssm
-    will generate an error, normally crashing the current ansible task.  This is normally the right
-    thing since ignoring a value that IAM isn't giving access to could cause bigger problems and
-    wrong behaviour or loss of data.  If you want to continue in this case then you will have to set
-    up two ansible tasks, one which sets a variable and ignores failures one which uses the value
+    will generate an error. If you want to continue in this case then you will have to set up
+    two ansible tasks, one which sets a variable and ignores failures and one which uses the value
     of that variable with a default.  See the examples below.
 
 options:
@@ -81,26 +78,26 @@
 - name: lookup ssm parameter store in the current region
   debug: msg="{{ lookup('aws_ssm', 'Hello' ) }}"
 
-- name: lookup ssm parameter store in nominated region
+- name: lookup ssm parameter store in specified region
   debug: msg="{{ lookup('aws_ssm', 'Hello', region='us-east-2' ) }}"
 
-- name: lookup ssm parameter store without decrypted
+- name: lookup ssm parameter store without decryption
   debug: msg="{{ lookup('aws_ssm', 'Hello', decrypt=False ) }}"
 
-- name: lookup ssm parameter store in nominated aws profile
+- name: lookup ssm parameter store using a specified aws profile
   debug: msg="{{ lookup('aws_ssm', 'Hello', aws_profile='myprofile' ) }}"
 
 - name: lookup ssm parameter store using explicit aws credentials
   debug: msg="{{ lookup('aws_ssm', 'Hello', aws_access_key=my_aws_access_key, aws_secret_key=my_aws_secret_key, aws_security_token=my_security_token ) }}"
 
-- name: lookup ssm parameter store with all options.
+- name: lookup ssm parameter store with all options
   debug: msg="{{ lookup('aws_ssm', 'Hello', decrypt=false, region='us-east-2', aws_profile='myprofile') }}"
 
-- name: lookup a key which doesn't exist, returns ""
-  debug: msg="{{ lookup('aws_ssm', 'NoKey') }}"
+- name: lookup ssm parameter and fail if missing
+  debug: msg="{{ lookup('aws_ssm', 'missing-parameter') }}"
 
 - name: lookup a key which doesn't exist, returning a default ('root')
-  debug: msg="{{ lookup('aws_ssm', 'AdminID') | default('root', true) }}"
+  debug: msg="{{ lookup('aws_ssm', 'AdminID', on_missing="skip") | default('root', true) }}"
 
 - name: lookup a key which doesn't exist failing to store it in a fact
   set_fact:
@@ -124,9 +121,6 @@
   debug: msg='Path contains {{ item }}'
   loop: '{{ lookup("aws_ssm", "/demo/", "/demo1/", bypath=True)}}'
 
-- name: lookup ssm parameter and fail if missing
-  debug: msg="{{ lookup('aws_ssm', 'missing-parameter', on_missing="error" ) }}"
-
 - name: lookup ssm parameter warn if access is denied
   debug: msg="{{ lookup('aws_ssm', 'missing-parameter', on_denied="warn" ) }}"
 '''
@@ -173,8 +167,8 @@ def _boto3_conn(region, credentials):
 class LookupModule(LookupBase):
     def run(self, terms, variables=None, boto_profile=None, aws_profile=None,
             aws_secret_key=None, aws_access_key=None, aws_security_token=None, region=None,
-            bypath=False, shortnames=False, recursive=False, decrypt=True, on_missing="skip",
-            on_denied="skip"):
+            bypath=False, shortnames=False, recursive=False, decrypt=True, on_missing="error",
+            on_denied="error"):
         '''
             :arg terms: a list of lookups to run.
                 e.g. ['parameter_name', 'parameter_name_too' ]
@@ -220,33 +214,22 @@ def run(self, terms, variables=None, boto_profile=None, aws_profile=None,
         if bypath:
             ssm_dict['Recursive'] = recursive
             for term in terms:
-                ssm_dict["Path"] = term
                 display.vvv("AWS_ssm path lookup term: %s in region: %s" % (term, region))
-                try:
-                    response = client.get_parameters_by_path(**ssm_dict)
-                except botocore.exceptions.ClientError as e:
-                    raise AnsibleError("SSM lookup exception: {0}".format(to_native(e)))
-                paramlist = list()
-                paramlist.extend(response['Parameters'])
-
-                # Manual pagination, since boto doesn't support it yet for get_parameters_by_path
-                while 'NextToken' in response:
-                    response = client.get_parameters_by_path(NextToken=response['NextToken'], **ssm_dict)
-                    paramlist.extend(response['Parameters'])
-
-                # shorten parameter names. yes, this will return duplicate names with different values.
+
+                paramlist = self.get_path_parameters(client, ssm_dict, term, on_missing.lower(), on_denied.lower())
+                # Shorten parameter names. Yes, this will return
+                # duplicate names with different values.
                 if shortnames:
                     for x in paramlist:
                         x['Name'] = x['Name'][x['Name'].rfind('/') + 1:]
 
                 display.vvvv("AWS_ssm path lookup returned: %s" % str(paramlist))
-                if len(paramlist):
-                    ret.append(boto3_tag_list_to_ansible_dict(paramlist,
-                                                              tag_name_key_name="Name",
-                                                              tag_value_key_name="Value"))
-                else:
-                    ret.append({})
-            # Lookup by parameter name - always returns a list with one or no entry.
+
+                ret.append(boto3_tag_list_to_ansible_dict(paramlist,
+                                                          tag_name_key_name="Name",
+                                                          tag_value_key_name="Value"))
+        # Lookup by parameter name - always returns a list with one or
+        # no entry.
         else:
             display.vvv("AWS_ssm name lookup term: %s" % terms)
             for term in terms:
@@ -254,6 +237,30 @@ def run(self, terms, variables=None, boto_profile=None, aws_profile=None,
         display.vvvv("AWS_ssm path lookup returning: %s " % str(ret))
         return ret
 
+    def get_path_parameters(self, client, ssm_dict, term, on_missing, on_denied):
+        ssm_dict["Path"] = term
+        paginator = client.get_paginator('get_parameters_by_path')
+        try:
+            paramlist = paginator.paginate(**ssm_dict).build_full_result()['Parameters']
+        except is_boto3_error_code('AccessDeniedException'):
+            if on_denied == 'error':
+                raise AnsibleError("Failed to access SSM parameter path %s (AccessDenied)" % term)
+            elif on_denied == 'warn':
+                self._display.warning('Skipping, access denied for SSM parameter path %s' % term)
+                paramlist = [{}]
+            elif on_denied == 'skip':
+                paramlist = [{}]
+        except botocore.exceptions.ClientError as e:  # pylint: disable=duplicate-except
+            raise AnsibleError("SSM lookup exception: {0}".format(to_native(e)))
+
+        if not len(paramlist):
+            if on_missing == "error":
+                raise AnsibleError("Failed to find SSM parameter path %s (ResourceNotFound)" % term)
+            elif on_missing == "warn":
+                self._display.warning('Skipping, did not find SSM parameter path %s' % term)
+
+        return paramlist
+
     def get_parameter_value(self, client, ssm_dict, term, on_missing, on_denied):
         ssm_dict["Name"] = term
         try:


Test patch:
diff --git a/tests/unit/plugins/lookup/test_aws_ssm.py b/tests/unit/plugins/lookup/test_aws_ssm.py
--- a/tests/unit/plugins/lookup/test_aws_ssm.py
+++ b/tests/unit/plugins/lookup/test_aws_ssm.py
@@ -128,43 +128,51 @@ def test_path_lookup_variable(mocker):
     lookup._load_name = "aws_ssm"
 
     boto3_double = mocker.MagicMock()
-    get_path_fn = boto3_double.Session.return_value.client.return_value.get_parameters_by_path
-    get_path_fn.return_value = path_success_response
+    get_paginator_fn = boto3_double.Session.return_value.client.return_value.get_paginator
+    paginator = get_paginator_fn.return_value
+    paginator.paginate.return_value.build_full_result.return_value = path_success_response
     boto3_client_double = boto3_double.Session.return_value.client
 
     mocker.patch.object(boto3, 'session', boto3_double)
     args = copy(dummy_credentials)
-    args["bypath"] = 'true'
+    args["bypath"] = True
+    args["recursive"] = True
     retval = lookup.run(["/testpath"], {}, **args)
     assert(retval[0]["/testpath/won"] == "simple_value_won")
     assert(retval[0]["/testpath/too"] == "simple_value_too")
     boto3_client_double.assert_called_with('ssm', 'eu-west-1', aws_access_key_id='notakey',
                                            aws_secret_access_key="notasecret", aws_session_token=None)
-    get_path_fn.assert_called_with(Path="/testpath", Recursive=False, WithDecryption=True)
+    get_paginator_fn.assert_called_with('get_parameters_by_path')
+    paginator.paginate.assert_called_with(Path="/testpath", Recursive=True, WithDecryption=True)
+    paginator.paginate.return_value.build_full_result.assert_called_with()
 
 
-def test_return_none_for_missing_variable(mocker):
-    """
-    during jinja2 templates, we can't shouldn't normally raise exceptions since this blocks the ability to use defaults.
+def test_warn_on_missing_match_retvals_to_call_params_with_some_missing_variables(mocker):
+    """If we get a complex list of variables with some missing and some
+    not, and on_missing is warn, we still have to return a list which
+    matches with the original variable list.
 
-    for this reason we return ```None``` for missing variables
     """
     lookup = aws_ssm.LookupModule()
     lookup._load_name = "aws_ssm"
 
     boto3_double = mocker.MagicMock()
+
     boto3_double.Session.return_value.client.return_value.get_parameter.side_effect = mock_get_parameter
 
     mocker.patch.object(boto3, 'session', boto3_double)
-    retval = lookup.run(["missing_variable"], {}, **dummy_credentials)
+    args = copy(dummy_credentials)
+    args["on_missing"] = 'warn'
+    retval = lookup.run(["simple", "missing_variable", "/testpath/won", "simple"], {}, **args)
     assert(isinstance(retval, list))
-    assert(retval[0] is None)
+    assert(retval == ["simple_value", None, "simple_value_won", "simple_value"])
 
 
-def test_match_retvals_to_call_params_even_with_some_missing_variables(mocker):
-    """
-    If we get a complex list of variables with some missing and some not, we still have to return a
-    list which matches with the original variable list.
+def test_skip_on_missing_match_retvals_to_call_params_with_some_missing_variables(mocker):
+    """If we get a complex list of variables with some missing and some
+    not, and on_missing is skip, we still have to return a list which
+    matches with the original variable list.
+
     """
     lookup = aws_ssm.LookupModule()
     lookup._load_name = "aws_ssm"
@@ -174,7 +182,9 @@ def test_match_retvals_to_call_params_even_with_some_missing_variables(mocker):
     boto3_double.Session.return_value.client.return_value.get_parameter.side_effect = mock_get_parameter
 
     mocker.patch.object(boto3, 'session', boto3_double)
-    retval = lookup.run(["simple", "missing_variable", "/testpath/won", "simple"], {}, **dummy_credentials)
+    args = copy(dummy_credentials)
+    args["on_missing"] = 'skip'
+    retval = lookup.run(["simple", "missing_variable", "/testpath/won", "simple"], {}, **args)
     assert(isinstance(retval, list))
     assert(retval == ["simple_value", None, "simple_value_won", "simple_value"])
 
@@ -246,7 +256,7 @@ def test_skip_on_missing_variable(mocker):
     boto3_double.Session.return_value.client.return_value.get_parameter.side_effect = mock_get_parameter
 
     missing_credentials = copy(dummy_credentials)
-    missing_credentials['on_missing'] = "warn"
+    missing_credentials['on_missing'] = "skip"
     mocker.patch.object(boto3, 'session', boto3_double)
     retval = lookup.run(["missing_variable"], {}, **missing_credentials)
     assert(isinstance(retval, list))
@@ -307,7 +317,7 @@ def test_skip_on_denied_variable(mocker):
     boto3_double.Session.return_value.client.return_value.get_parameter.side_effect = mock_get_parameter
 
     denied_credentials = copy(dummy_credentials)
-    denied_credentials['on_denied'] = "warn"
+    denied_credentials['on_denied'] = "skip"
     mocker.patch.object(boto3, 'session', boto3_double)
     retval = lookup.run(["denied_variable"], {}, **denied_credentials)
     assert(isinstance(retval, list))