Repository: ansible-collections/amazon.aws

Issue:
ResourceNotFoundException is raised when querying ACM certificates
### Summary

A `ResourceNotFoundException` may be raised when a ansible module invokes the [get_certificates](https://github.com/ansible-collections/amazon.aws/blob/main/plugins/module_utils/acm.py#L93) function defined in module_utils/acm.py. It looks like there is an intermittent timing issue:
1. The `get_certificates` function gets all certificates defined in ACM using paginated queries.
2. Iterate over every certificate ARN obtained from step 1, and get additional data about the certificate. However some certs may have been deleted between steps 1 and 2, causing a `ResourceNotFoundException`.

### Issue Type

Bug Report

### Component Name

acm in module_utils

### Ansible Version

```console (paste below)
$ ansible --version
nsible [core 2.12.1]
  config file = None
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = ~/.local/lib/python3.8/site-packages/ansible
  ansible collection location =~/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/rh/rh-python38/root/usr/local/bin/ansible
  python version = 3.8.11 (default, Sep  1 2021, 12:33:46) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]
  jinja version = 3.0.3
  libyaml = True

```


### Collection Versions

```console (paste below)
$ ansible-galaxy collection list
# ~/.ansible/collections/ansible_collections
Collection       Version
---------------- -------
amazon.aws       3.0.0  
community.aws    2.1.0  
community.crypto 2.1.0  

# ~/.local/lib/python3.8/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0  
ansible.netcommon             2.4.0  
ansible.posix                 1.3.0  
ansible.utils                 2.4.2  
ansible.windows               1.8.0  
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.10.0 
check_point.mgmt              2.1.1  
chocolatey.chocolatey         1.1.0  
cisco.aci                     2.1.0  
cisco.asa                     2.1.0  
cisco.intersight              1.0.17 
cisco.ios                     2.5.0  
cisco.iosxr                   2.5.0  
cisco.ise                     1.2.1  
cisco.meraki                  2.5.0  
cisco.mso                     1.2.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.7.1  
cisco.ucs                     1.6.0  
cloud.common                  2.1.0  
cloudscale_ch.cloud           2.2.0  
community.aws                 2.1.0  
community.azure               1.1.0  
community.ciscosmb            1.0.4  
community.crypto              2.0.1  
community.digitalocean        1.12.0 
community.dns                 2.0.3  
community.docker              2.0.1  
community.fortios             1.0.0  
community.general             4.0.2  
community.google              1.0.0  
community.grafana             1.2.3  
community.hashi_vault         2.0.0  
community.hrobot              1.2.1  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.2  
community.mongodb             1.3.2  
community.mysql               2.3.1  
community.network             3.0.0  
community.okd                 2.1.0  
community.postgresql          1.5.0  
community.proxysql            1.3.0  
community.rabbitmq            1.1.0  
community.routeros            2.0.0  
community.skydive             1.0.0  
community.sops                1.2.0  
community.vmware              1.16.0 
community.windows             1.8.0  
community.zabbix              1.5.0  
containers.podman             1.8.2  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.13 
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            4.2.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.12.0 
fortinet.fortimanager         2.1.4  
fortinet.fortios              2.1.3  
frr.frr                       1.0.3  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.3  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.0  
infoblox.nios_modules         1.1.2  
inspur.sm                     1.3.0  
junipernetworks.junos         2.6.0  
kubernetes.core               2.2.1  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.12.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.13.1
netapp.storagegrid            21.7.0 
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.2.13 
netbox.netbox                 3.3.0  
ngine_io.cloudstack           2.2.2  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.5.3  
openvswitch.openvswitch       2.0.2  
ovirt.ovirt                   1.6.5  
purestorage.flasharray        1.11.0 
purestorage.flashblade        1.8.1  
sensu.sensu_go                1.12.0 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.24.0 
theforeman.foreman            2.2.0  
vyos.vyos                     2.6.0  
wti.remote                    1.0.3  

# /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0  
ansible.netcommon             2.4.0  
ansible.posix                 1.3.0  
ansible.utils                 2.4.2  
ansible.windows               1.8.0  
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.10.0 
check_point.mgmt              2.1.1  
chocolatey.chocolatey         1.1.0  
cisco.aci                     2.1.0  
cisco.asa                     2.1.0  
cisco.intersight              1.0.17 
cisco.ios                     2.5.0  
cisco.iosxr                   2.5.0  
cisco.ise                     1.2.1  
cisco.meraki                  2.5.0  
cisco.mso                     1.2.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.7.1  
cisco.ucs                     1.6.0  
cloud.common                  2.1.0  
cloudscale_ch.cloud           2.2.0  
community.aws                 2.1.0  
community.azure               1.1.0  
community.ciscosmb            1.0.4  
community.crypto              2.0.1  
community.digitalocean        1.12.0 
community.dns                 2.0.3  
community.docker              2.0.1  
community.fortios             1.0.0  
community.general             4.0.2  
community.google              1.0.0  
community.grafana             1.2.3  
community.hashi_vault         2.0.0  
community.hrobot              1.2.1  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.2  
community.mongodb             1.3.2  
community.mysql               2.3.1  
community.network             3.0.0  
community.okd                 2.1.0  
community.postgresql          1.5.0  
community.proxysql            1.3.0  
community.rabbitmq            1.1.0  
community.routeros            2.0.0  
community.skydive             1.0.0  
community.sops                1.2.0  
community.vmware              1.16.0 
community.windows             1.8.0  
community.zabbix              1.5.0  
containers.podman             1.8.2  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.13 
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            4.2.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.12.0 
fortinet.fortimanager         2.1.4  
fortinet.fortios              2.1.3  
frr.frr                       1.0.3  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.3  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.0  
infoblox.nios_modules         1.1.2  
inspur.sm                     1.3.0  
junipernetworks.junos         2.6.0  
kubernetes.core               2.2.1  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.12.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.13.1
netapp.storagegrid            21.7.0 
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.2.13 
netbox.netbox                 3.3.0  
ngine_io.cloudstack           2.2.2  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.5.3  
openvswitch.openvswitch       2.0.2  
ovirt.ovirt                   1.6.5  
purestorage.flasharray        1.11.0 
purestorage.flashblade        1.8.1  
sensu.sensu_go                1.12.0 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.24.0 
theforeman.foreman            2.2.0  
vyos.vyos                     2.6.0  
wti.remote                    1.0.3  

```


### AWS SDK versions

```console (paste below)
$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: ~/.local/lib/python3.8/site-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.20.21
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: ~/.local/lib/python3.8/site-packages
Requires: jmespath, botocore, s3transfer
Required-by: 
---
Name: botocore
Version: 1.23.21
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: ~/.local/lib/python3.8/site-packages
Requires: jmespath, urllib3, python-dateutil
Required-by: s3transfer, boto3
```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed

```


### OS / Environment

CentOS Linux release 7.9.2009 (Core)
Linux 5ada6014185e 5.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13 11:06:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

### Steps to Reproduce

The issue occurs intermittently when running the integration tests for the `acm` module in `community.aws`. Specifically when a `community.aws.aws_acm` task deletes a cert, followed by the `community.aws.aws_acm_info`. The `ResourceNotFoundException` occurs in the `aws_acm_info` task.

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
  - name: delete cert
    community.aws.aws_acm:
      state: absent
      domain_name: 'mydoamain.com'
  - name: check cert was deleted
    community.aws.aws_acm_info:
      tags:
        Name: 'my_certificate_name'
```
 
The problem occurs when a cert is removed from ACM while `acm.py:get_certificates` iterates a list of cert ARNs. The problem is more likely to occur if the `acm.py:get_certificates` takes a long time to iterate over the certificate ARNs. For example this could happen if there are lots of certificates in ACM.

If there are N certificates in ACM, `get_certificates` could send up to 1 + 3 * N requests to the AWS API, not counting the retries (for each cert, `get_certificates` makes up to 2 or 3 AWS API calls depending on the cert status.)

### Expected Results

I was expecting `community.aws.aws_acm_info` to return a list of certificates successfully, even if certificates are being deleted while querying the certs. Internally the `aws_acm_info` module invokes the `amazon.aws:get_certificates` function which is defined in module_utils/acm.py. That function should skip certificates when a `ResourceNotFound` exception is raised. I.e. don't return certs that are being deleted.
The problem exists in the `main` branch.

### Actual Results

An exception is raised while executing the `aws_acm_info` task.
 
```console (paste below)
TASK [aws_acm : check cert 3 was deleted] ********************************************************************************************************************
task path: /root/ansible_collections/community/aws/tests/output/.tmp/integration/aws_acm-uxj6vjlf-ÅÑŚÌβŁÈ/tests/integration/targets/aws_acm/tasks/main.yml:435
Using module file /root/ansible_collections/community/aws/plugins/modules/aws_acm_info.py
Pipelining is enabled.
<testhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<testhost> EXEC /bin/sh -c 'ANSIBLE_DEBUG_BOTOCORE_LOGS=True /usr/bin/python3.8 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/acm.py", line 111, in get_certificates
    cert_data = self.describe_certificate_with_backoff(client, certificate['CertificateArn'])
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 118, in _retry_wrapper
    return _retry_func(
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/acm.py", line 82, in describe_certificate_with_backoff
    return client.describe_certificate(CertificateArn=certificate_arn)['Certificate']
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the DescribeCertificate operation: Could not find certificate arn:aws:acm:us-west-1:XXXXXXX:certificate/XXXXXXXXXXXX.
fatal: [testhost]: FAILED! => {
    "boto3_version": "1.16.0",
    "botocore_version": "1.19.0",
    "changed": false,
    "error": {
        "code": "ResourceNotFoundException",
        "message": "Could not find certificate arn:aws:acm:us-west-1:XXXXXXX:certificate/XXXXXXXXXXX.”
    },
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct
ResourceNotFoundException is raised when querying ACM certificates
### Summary

A `ResourceNotFoundException` may be raised when a ansible module invokes the [get_certificates](https://github.com/ansible-collections/amazon.aws/blob/main/plugins/module_utils/acm.py#L93) function defined in module_utils/acm.py. It looks like there is an intermittent timing issue:
1. The `get_certificates` function gets all certificates defined in ACM using paginated queries.
2. Iterate over every certificate ARN obtained from step 1, and get additional data about the certificate. However some certs may have been deleted between steps 1 and 2, causing a `ResourceNotFoundException`.

### Issue Type

Bug Report

### Component Name

acm in module_utils

### Ansible Version

```console (paste below)
$ ansible --version
nsible [core 2.12.1]
  config file = None
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = ~/.local/lib/python3.8/site-packages/ansible
  ansible collection location =~/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/rh/rh-python38/root/usr/local/bin/ansible
  python version = 3.8.11 (default, Sep  1 2021, 12:33:46) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]
  jinja version = 3.0.3
  libyaml = True

```


### Collection Versions

```console (paste below)
$ ansible-galaxy collection list
# ~/.ansible/collections/ansible_collections
Collection       Version
---------------- -------
amazon.aws       3.0.0  
community.aws    2.1.0  
community.crypto 2.1.0  

# ~/.local/lib/python3.8/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0  
ansible.netcommon             2.4.0  
ansible.posix                 1.3.0  
ansible.utils                 2.4.2  
ansible.windows               1.8.0  
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.10.0 
check_point.mgmt              2.1.1  
chocolatey.chocolatey         1.1.0  
cisco.aci                     2.1.0  
cisco.asa                     2.1.0  
cisco.intersight              1.0.17 
cisco.ios                     2.5.0  
cisco.iosxr                   2.5.0  
cisco.ise                     1.2.1  
cisco.meraki                  2.5.0  
cisco.mso                     1.2.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.7.1  
cisco.ucs                     1.6.0  
cloud.common                  2.1.0  
cloudscale_ch.cloud           2.2.0  
community.aws                 2.1.0  
community.azure               1.1.0  
community.ciscosmb            1.0.4  
community.crypto              2.0.1  
community.digitalocean        1.12.0 
community.dns                 2.0.3  
community.docker              2.0.1  
community.fortios             1.0.0  
community.general             4.0.2  
community.google              1.0.0  
community.grafana             1.2.3  
community.hashi_vault         2.0.0  
community.hrobot              1.2.1  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.2  
community.mongodb             1.3.2  
community.mysql               2.3.1  
community.network             3.0.0  
community.okd                 2.1.0  
community.postgresql          1.5.0  
community.proxysql            1.3.0  
community.rabbitmq            1.1.0  
community.routeros            2.0.0  
community.skydive             1.0.0  
community.sops                1.2.0  
community.vmware              1.16.0 
community.windows             1.8.0  
community.zabbix              1.5.0  
containers.podman             1.8.2  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.13 
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            4.2.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.12.0 
fortinet.fortimanager         2.1.4  
fortinet.fortios              2.1.3  
frr.frr                       1.0.3  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.3  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.0  
infoblox.nios_modules         1.1.2  
inspur.sm                     1.3.0  
junipernetworks.junos         2.6.0  
kubernetes.core               2.2.1  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.12.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.13.1
netapp.storagegrid            21.7.0 
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.2.13 
netbox.netbox                 3.3.0  
ngine_io.cloudstack           2.2.2  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.5.3  
openvswitch.openvswitch       2.0.2  
ovirt.ovirt                   1.6.5  
purestorage.flasharray        1.11.0 
purestorage.flashblade        1.8.1  
sensu.sensu_go                1.12.0 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.24.0 
theforeman.foreman            2.2.0  
vyos.vyos                     2.6.0  
wti.remote                    1.0.3  

# /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0  
ansible.netcommon             2.4.0  
ansible.posix                 1.3.0  
ansible.utils                 2.4.2  
ansible.windows               1.8.0  
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.10.0 
check_point.mgmt              2.1.1  
chocolatey.chocolatey         1.1.0  
cisco.aci                     2.1.0  
cisco.asa                     2.1.0  
cisco.intersight              1.0.17 
cisco.ios                     2.5.0  
cisco.iosxr                   2.5.0  
cisco.ise                     1.2.1  
cisco.meraki                  2.5.0  
cisco.mso                     1.2.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.7.1  
cisco.ucs                     1.6.0  
cloud.common                  2.1.0  
cloudscale_ch.cloud           2.2.0  
community.aws                 2.1.0  
community.azure               1.1.0  
community.ciscosmb            1.0.4  
community.crypto              2.0.1  
community.digitalocean        1.12.0 
community.dns                 2.0.3  
community.docker              2.0.1  
community.fortios             1.0.0  
community.general             4.0.2  
community.google              1.0.0  
community.grafana             1.2.3  
community.hashi_vault         2.0.0  
community.hrobot              1.2.1  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.2  
community.mongodb             1.3.2  
community.mysql               2.3.1  
community.network             3.0.0  
community.okd                 2.1.0  
community.postgresql          1.5.0  
community.proxysql            1.3.0  
community.rabbitmq            1.1.0  
community.routeros            2.0.0  
community.skydive             1.0.0  
community.sops                1.2.0  
community.vmware              1.16.0 
community.windows             1.8.0  
community.zabbix              1.5.0  
containers.podman             1.8.2  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.13 
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            4.2.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.12.0 
fortinet.fortimanager         2.1.4  
fortinet.fortios              2.1.3  
frr.frr                       1.0.3  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.3  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.0  
infoblox.nios_modules         1.1.2  
inspur.sm                     1.3.0  
junipernetworks.junos         2.6.0  
kubernetes.core               2.2.1  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.12.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.13.1
netapp.storagegrid            21.7.0 
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.2.13 
netbox.netbox                 3.3.0  
ngine_io.cloudstack           2.2.2  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.0  
openstack.cloud               1.5.3  
openvswitch.openvswitch       2.0.2  
ovirt.ovirt                   1.6.5  
purestorage.flasharray        1.11.0 
purestorage.flashblade        1.8.1  
sensu.sensu_go                1.12.0 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.24.0 
theforeman.foreman            2.2.0  
vyos.vyos                     2.6.0  
wti.remote                    1.0.3  

```


### AWS SDK versions

```console (paste below)
$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location: ~/.local/lib/python3.8/site-packages
Requires: 
Required-by: 
---
Name: boto3
Version: 1.20.21
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: ~/.local/lib/python3.8/site-packages
Requires: jmespath, botocore, s3transfer
Required-by: 
---
Name: botocore
Version: 1.23.21
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: None
License: Apache License 2.0
Location: ~/.local/lib/python3.8/site-packages
Requires: jmespath, urllib3, python-dateutil
Required-by: s3transfer, boto3
```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed

```


### OS / Environment

CentOS Linux release 7.9.2009 (Core)
Linux 5ada6014185e 5.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13 11:06:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

### Steps to Reproduce

The issue occurs intermittently when running the integration tests for the `acm` module in `community.aws`. Specifically when a `community.aws.aws_acm` task deletes a cert, followed by the `community.aws.aws_acm_info`. The `ResourceNotFoundException` occurs in the `aws_acm_info` task.

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
  - name: delete cert
    community.aws.aws_acm:
      state: absent
      domain_name: 'mydoamain.com'
  - name: check cert was deleted
    community.aws.aws_acm_info:
      tags:
        Name: 'my_certificate_name'
```
 
The problem occurs when a cert is removed from ACM while `acm.py:get_certificates` iterates a list of cert ARNs. The problem is more likely to occur if the `acm.py:get_certificates` takes a long time to iterate over the certificate ARNs. For example this could happen if there are lots of certificates in ACM.

If there are N certificates in ACM, `get_certificates` could send up to 1 + 3 * N requests to the AWS API, not counting the retries (for each cert, `get_certificates` makes up to 2 or 3 AWS API calls depending on the cert status.)

### Expected Results

I was expecting `community.aws.aws_acm_info` to return a list of certificates successfully, even if certificates are being deleted while querying the certs. Internally the `aws_acm_info` module invokes the `amazon.aws:get_certificates` function which is defined in module_utils/acm.py. That function should skip certificates when a `ResourceNotFound` exception is raised. I.e. don't return certs that are being deleted.
The problem exists in the `main` branch.

### Actual Results

An exception is raised while executing the `aws_acm_info` task.
 
```console (paste below)
TASK [aws_acm : check cert 3 was deleted] ********************************************************************************************************************
task path: /root/ansible_collections/community/aws/tests/output/.tmp/integration/aws_acm-uxj6vjlf-ÅÑŚÌβŁÈ/tests/integration/targets/aws_acm/tasks/main.yml:435
Using module file /root/ansible_collections/community/aws/plugins/modules/aws_acm_info.py
Pipelining is enabled.
<testhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<testhost> EXEC /bin/sh -c 'ANSIBLE_DEBUG_BOTOCORE_LOGS=True /usr/bin/python3.8 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/acm.py", line 111, in get_certificates
    cert_data = self.describe_certificate_with_backoff(client, certificate['CertificateArn'])
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 118, in _retry_wrapper
    return _retry_func(
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
  File "/tmp/ansible_aws_acm_info_payload_25vnfkvy/ansible_aws_acm_info_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/acm.py", line 82, in describe_certificate_with_backoff
    return client.describe_certificate(CertificateArn=certificate_arn)['Certificate']
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the DescribeCertificate operation: Could not find certificate arn:aws:acm:us-west-1:XXXXXXX:certificate/XXXXXXXXXXXX.
fatal: [testhost]: FAILED! => {
    "boto3_version": "1.16.0",
    "botocore_version": "1.19.0",
    "changed": false,
    "error": {
        "code": "ResourceNotFoundException",
        "message": "Could not find certificate arn:aws:acm:us-west-1:XXXXXXX:certificate/XXXXXXXXXXX.”
    },
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/module_utils/acm.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/module_utils/acm.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* [`plugins/module_utils/acm.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/module_utils/acm.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->

Resolving patch:
diff --git a/plugins/module_utils/acm.py b/plugins/module_utils/acm.py
--- a/plugins/module_utils/acm.py
+++ b/plugins/module_utils/acm.py
@@ -38,6 +38,7 @@
 from ansible.module_utils._text import to_bytes
 from ansible.module_utils.common.dict_transformations import camel_dict_to_snake_dict
 
+from .core import is_boto3_error_code
 from .ec2 import AWSRetry
 from .ec2 import ansible_dict_to_boto3_tag_list
 from .ec2 import boto3_tag_list_to_ansible_dict
@@ -109,19 +110,28 @@ def get_certificates(self, client, module, domain_name=None, statuses=None, arn=
         for certificate in certificates:
             try:
                 cert_data = self.describe_certificate_with_backoff(client, certificate['CertificateArn'])
-            except (BotoCoreError, ClientError) as e:
+            except is_boto3_error_code('ResourceNotFoundException'):
+                # The certificate was deleted after the call to list_certificates_with_backoff.
+                continue
+            except (BotoCoreError, ClientError) as e:  # pylint: disable=duplicate-except
                 module.fail_json_aws(e, msg="Couldn't obtain certificate metadata for domain %s" % certificate['DomainName'])
 
             # in some states, ACM resources do not have a corresponding cert
             if cert_data['Status'] not in ['PENDING_VALIDATION', 'VALIDATION_TIMED_OUT', 'FAILED']:
                 try:
                     cert_data.update(self.get_certificate_with_backoff(client, certificate['CertificateArn']))
-                except (BotoCoreError, ClientError, KeyError) as e:
+                except is_boto3_error_code('ResourceNotFoundException'):
+                    # The certificate was deleted after the call to list_certificates_with_backoff.
+                    continue
+                except (BotoCoreError, ClientError, KeyError) as e:  # pylint: disable=duplicate-except
                     module.fail_json_aws(e, msg="Couldn't obtain certificate data for domain %s" % certificate['DomainName'])
             cert_data = camel_dict_to_snake_dict(cert_data)
             try:
                 tags = self.list_certificate_tags_with_backoff(client, certificate['CertificateArn'])
-            except (BotoCoreError, ClientError) as e:
+            except is_boto3_error_code('ResourceNotFoundException'):
+                # The certificate was deleted after the call to list_certificates_with_backoff.
+                continue
+            except (BotoCoreError, ClientError) as e:  # pylint: disable=duplicate-except
                 module.fail_json_aws(e, msg="Couldn't obtain tags for domain %s" % certificate['DomainName'])
 
             cert_data['tags'] = boto3_tag_list_to_ansible_dict(tags)


Test patch: