Repository: ansible-collections/community.aws

Issue:
lambda.py invokes update_function_configuration() and afterwards update_function_code() without checking Configuration.State and Configuration.LastUpdateStatus fields in between
### Summary

lambda.py invokes `update_function_configuration()` [here](https://github.com/ansible-collections/community.aws/blob/main/plugins/modules/lambda.py#L458) and `afterwards update_function_code()` [here](https://github.com/ansible-collections/community.aws/blob/main/plugins/modules/lambda.py#L499) without checking `Configuration.State` and `Configuration.LastUpdateStatus` fields in between. If lambda function was updated in scope of the first call an error similar to below one is created during the second call:

```
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: botocore.errorfactory.ResourceConflictException: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:MY_REGION:MY_AWS_ACCOUNT_ID:function:MY_LAMBDA_FUNC_NAME
fatal: [localhost]: FAILED! => {"boto3_version": "1.20.22", "botocore_version": "1.23.22", "changed": false, "error": {"code": "ResourceConflictException", "message": "The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:MY_REGION:MY_AWS_ACCOUNT_ID:function:MY_LAMBDA_FUNC_NAME"}, "message": "The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:MY_REGION:MY_AWS_ACCOUNT_ID:function:MY_LAMBDA_FUNC_NAME", "msg": "Trying to upload new code: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:MY_REGION:MY_AWS_ACCOUNT_ID:function:MY_LAMBDA_FUNC_NAME", "response_metadata": {"http_headers": {"connection": "keep-alive", "content-length": "201", "content-type": "application/json", "date": "Thu, 09 Dec 2021 13:23:20 GMT", "x-amzn-errortype": "ResourceConflictException", "x-amzn-requestid": "abfc47a6-6c1c-42e4-b19d-853da87674bc"}, "http_status_code": 409, "request_id": "abfc47a6-6c1c-42e4-b19d-853da87674bc", "retry_attempts": 0}, "type": "User"}
```



### Issue Type

Bug Report

### Component Name

lambda

### Ansible Version

```console (paste below)
$ ansible --version
ansible [core 2.12.0]
  config file = /home/my_dir/ansible.cfg
  configured module search path = ['/home/my_dir/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/my_dir/.venv/lib/python3.8/site-packages/ansible
  ansible collection location = /home/my_dir/.ansible/my_company/collections
  executable location = /home/my_dir/.venv/bin/ansible
  python version = 3.8.5 (v3.8.5:580fbb018f, Jul 20 2020, 12:11:27) [Clang 6.0 (clang-600.0.57)]
  jinja version = 2.11.3
  libyaml = True```


### Collection Versions

```console (paste below)
$ ansible-galaxy collection list
# /home/my_dir/.ansible/my_company/collections/ansible_collections
Collection            Version
--------------------- -------
amazon.aws            3.0.0
ansible.netcommon     1.4.1
ansible.posix         1.3.0
community.aws         2.1.0
community.general     4.0.0
community.hashi_vault 1.3.2
community.kubernetes  1.2.1
community.sops        1.1.0
google.cloud          1.0.1
kubernetes.core       2.2.1

# /home/my_dir/.venv/lib/python3.8/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    2.1.0
ansible.netcommon             2.4.0
ansible.posix                 1.3.0
ansible.utils                 2.4.2
ansible.windows               1.8.0
arista.eos                    3.1.0
awx.awx                       19.4.0
azure.azcollection            1.10.0
check_point.mgmt              2.1.1
chocolatey.chocolatey         1.1.0
cisco.aci                     2.1.0
cisco.asa                     2.1.0
cisco.intersight              1.0.17
cisco.ios                     2.5.0
cisco.iosxr                   2.5.0
cisco.ise                     1.2.1
cisco.meraki                  2.5.0
cisco.mso                     1.2.0
cisco.nso                     1.0.3
cisco.nxos                    2.7.1
cisco.ucs                     1.6.0
cloud.common                  2.1.0
cloudscale_ch.cloud           2.2.0
community.aws                 2.1.0
community.azure               1.1.0
community.ciscosmb            1.0.4
community.crypto              2.0.1
community.digitalocean        1.12.0
community.dns                 2.0.3
community.docker              2.0.1
community.fortios             1.0.0
community.general             4.0.2
community.google              1.0.0
community.grafana             1.2.3
community.hashi_vault         2.0.0
community.hrobot              1.2.1
community.kubernetes          2.0.1
community.kubevirt            1.0.0
community.libvirt             1.0.2
community.mongodb             1.3.2
community.mysql               2.3.1
community.network             3.0.0                                                                                                                                                                                                                                                                                 [0/1887]
community.okd                 2.1.0
community.postgresql          1.5.0
community.proxysql            1.3.0
community.rabbitmq            1.1.0
community.routeros            2.0.0
community.skydive             1.0.0
community.sops                1.2.0
community.vmware              1.16.0
community.windows             1.8.0
community.zabbix              1.5.0
containers.podman             1.8.2
cyberark.conjur               1.1.0
cyberark.pas                  1.0.13
dellemc.enterprise_sonic      1.1.0
dellemc.openmanage            4.2.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.12.0
fortinet.fortimanager         2.1.4
fortinet.fortios              2.1.3
frr.frr                       1.0.3
gluster.gluster               1.0.2
google.cloud                  1.0.2
hetzner.hcloud                1.6.0
hpe.nimble                    1.1.3
ibm.qradar                    1.0.3
infinidat.infinibox           1.3.0
infoblox.nios_modules         1.1.2
inspur.sm                     1.3.0
junipernetworks.junos         2.6.0
kubernetes.core               2.2.1
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.12.0
netapp.elementsw              21.7.0
netapp.ontap                  21.13.1
netapp.storagegrid            21.7.0
netapp.um_info                21.8.0
netapp_eseries.santricity     1.2.13
netbox.netbox                 3.3.0
ngine_io.cloudstack           2.2.2
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.0
openstack.cloud               1.5.3
openvswitch.openvswitch       2.0.2
ovirt.ovirt                   1.6.5
purestorage.flasharray        1.11.0
purestorage.flashblade        1.8.1
sensu.sensu_go                1.12.0
servicenow.servicenow         1.0.6
splunk.es                     1.0.2
t_systems_mms.icinga_director 1.24.0
theforeman.foreman            2.2.0
vyos.vyos                     2.6.0
wti.remote                    1.0.3
```


### AWS SDK versions

```console (paste below)
$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: mitch@garnaat.com
License: MIT
Location:/home/my_dir/.venv/lib/python3.8/site-packages
Requires:
Required-by:
---
Name: boto3
Version: 1.20.22
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /home/my_dir/.venv/lib/python3.8/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.23.22
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /home/my_dir/.venv/lib/python3.8/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: awscli, boto3, s3transfer
```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed
COLLECTIONS_PATHS(/home/my_dir/ansible.cfg) = ['/home/my_dir/.ansible/my_company/collections']
DEFAULT_HOST_LIST(/home/my_dir/ansible.cfg) = ['/dev/null']
DEFAULT_VAULT_PASSWORD_FILE(env: ANSIBLE_VAULT_PASSWORD_FILE) = /home/my_dir/.vault-ansible
LOCALHOST_WARNING(/home/my_dir/ansible.cfg) = False
RETRY_FILES_ENABLED(/home/my_dir/ansible.cfg) = False
```


### OS / Environment

macOS Big Sur 11.6

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->
1. Change memory assignment in aws lambda gui to value different than the one set by Ansible
2. Wait until `aws lambda get-function --function-name YOUR_FUNCTION --query 'Configuration.[State, LastUpdateStatus]'` will return
```
[
    "Active",
    "Successful"
]
```
3. Invoke below ansible snippet
```yaml (paste below)
- name: "Deploy lambda function"
  lambda:
    dead_letter_arn: "{{ 'arn:aws:sqs:{}:{}:{}-{}-lambda-deadletter'.format(aws_region, account, x, lambda_name) }}"
    description: "{{ lambda_description }}"
    environment_variables: "{{ lambda_env }}"
    handler: "{{ lambda_handler }}"
    memory_size: "{{ lambda_memory }}"
    name: "{{ lambda_full_name }}"
    role: "arn:aws:iam::{{ account }}:role/service-role/{{ lambda_iam_name }}"
    runtime: "{{ lambda_runtime }}"
    s3_bucket: "{{ lambda_s3_bucket }}"
    s3_key: "{{ lambda_s3_url }}
    timeout: "{{ lambda_timeout }}"
    vpc_security_group_ids: "{{ group_id }}"
    vpc_subnet_ids: "{{ subnets }}"
    region: "{{ aws_region }}"
    aws_access_key: "{{ access_key }}"
    aws_secret_key: "{{ secret_key }}"
    security_token: "{{ session_token }}"
```

### Expected Results

4. Get the below error due to lambda function being still updated after `update_function_configuration()` call:
```
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: botocore.errorfactory.ResourceConflictException: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:MY_REGION:MY_AWS_ACCOUNT_ID:function:MY_LAMBDA_FUNC_NAME
fatal: [localhost]: FAILED! => {"boto3_version": "1.20.22", "botocore_version": "1.23.22", "changed": false, "error": {"code": "ResourceConflictException", "message": "The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:MY_REGION:MY_AWS_ACCOUNT_ID:function:MY_LAMBDA_FUNC_NAME"}, "message": "The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:MY_REGION:MY_AWS_ACCOUNT_ID:function:MY_LAMBDA_FUNC_NAME", "msg": "Trying to upload new code: An error occurred (ResourceConflictException) when calling the UpdateFunctionCode operation: The operation cannot be performed at this time. An update is in progress for resource: arn:aws:lambda:MY_REGION:MY_AWS_ACCOUNT_ID:function:MY_LAMBDA_FUNC_NAME", "response_metadata": {"http_headers": {"connection": "keep-alive", "content-length": "201", "content-type": "application/json", "date": "Thu, 09 Dec 2021 13:23:20 GMT", "x-amzn-errortype": "ResourceConflictException", "x-amzn-requestid": "abfc47a6-6c1c-42e4-b19d-853da87674bc"}, "http_status_code": 409, "request_id": "abfc47a6-6c1c-42e4-b19d-853da87674bc", "retry_attempts": 0}, "type": "User"}
```


### Actual Results

```console (paste below)

```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:

Resolving patch:
diff --git a/plugins/modules/execute_lambda.py b/plugins/modules/execute_lambda.py
--- a/plugins/modules/execute_lambda.py
+++ b/plugins/modules/execute_lambda.py
@@ -202,6 +202,9 @@ def main():
     elif name:
         invoke_params['FunctionName'] = name
 
+    if not module.check_mode:
+        wait_for_lambda(client, module, name)
+
     try:
         response = client.invoke(**invoke_params)
     except is_boto3_error_code('ResourceNotFoundException') as nfe:
@@ -255,5 +258,15 @@ def main():
     module.exit_json(changed=True, result=results)
 
 
+def wait_for_lambda(client, module, name):
+    try:
+        waiter = client.get_waiter('function_active')
+        waiter.wait(FunctionName=name)
+    except botocore.exceptions.WaiterError as e:
+        module.fail_json_aws(e, msg='Timeout while waiting on lambda to be Active')
+    except (botocore.exceptions.ClientError, botocore.exceptions.BotoCoreError) as e:
+        module.fail_json_aws(e, msg='Failed while waiting on lambda to be Active')
+
+
 if __name__ == '__main__':
     main()
diff --git a/plugins/modules/lambda.py b/plugins/modules/lambda.py
--- a/plugins/modules/lambda.py
+++ b/plugins/modules/lambda.py
@@ -216,7 +216,7 @@
 import re
 
 try:
-    from botocore.exceptions import ClientError, BotoCoreError
+    from botocore.exceptions import ClientError, BotoCoreError, WaiterError
 except ImportError:
     pass  # protected by AnsibleAWSModule
 
@@ -320,6 +320,18 @@ def set_tag(client, module, tags, function):
     return changed
 
 
+def wait_for_lambda(client, module, name):
+    try:
+        client_active_waiter = client.get_waiter('function_active')
+        client_updated_waiter = client.get_waiter('function_updated')
+        client_active_waiter.wait(FunctionName=name)
+        client_updated_waiter.wait(FunctionName=name)
+    except WaiterError as e:
+        module.fail_json_aws(e, msg='Timeout while waiting on lambda to finish updating')
+    except (ClientError, BotoCoreError) as e:
+        module.fail_json_aws(e, msg='Failed while waiting on lambda to finish updating')
+
+
 def main():
     argument_spec = dict(
         name=dict(required=True),
@@ -453,6 +465,9 @@ def main():
 
         # Upload new configuration if configuration has changed
         if len(func_kwargs) > 1:
+            if not check_mode:
+                wait_for_lambda(client, module, name)
+
             try:
                 if not check_mode:
                     response = client.update_function_configuration(aws_retry=True, **func_kwargs)
@@ -494,6 +509,9 @@ def main():
 
         # Upload new code if needed (e.g. code checksum has changed)
         if len(code_kwargs) > 2:
+            if not check_mode:
+                wait_for_lambda(client, module, name)
+
             try:
                 if not check_mode:
                     response = client.update_function_code(aws_retry=True, **code_kwargs)


Test patch: