Repository: ansible-collections/community.aws

Issue:
s3_lifecycle - occasionally fails to add/remove all rules from playbook
### Summary

Sometimes not all rules from playbook are added or removed.
It occurs even if 'wait' parameter is set to 'yes'.

It was observed that shortly (~30s) after setting the rules with the plugin.
get-bucket-lifecycle returns alternatively new and old rules in a random manner.

Similar issue has been reported in boto3 library.
https://github.com/boto/boto3/issues/2491

### Issue Type

Bug Report

### Component Name

s3_lifecycle

### Ansible Version

```console (paste below)
$ ansible --version
ansible [core 2.12.4]
  config file = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg
  configured module search path = ['/home/rubin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/lib/python3.8/site-packages/ansible
  ansible collection location = /home/rubin/.ansible/collections:/usr/share/ansible/collections
  executable location = ./env/bin/ansible
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.1
  libyaml = True
```


### Collection Versions

```console (paste below)
$ ansible-galaxy collection list
Collection                    Version
----------------------------- -------
amazon.aws                    2.2.0  
ansible.netcommon             2.6.1  
ansible.posix                 1.3.0  
ansible.utils                 2.5.2  
ansible.windows               1.9.0  
arista.eos                    3.1.0  
awx.awx                       19.4.0 
azure.azcollection            1.12.0 
check_point.mgmt              2.3.0  
chocolatey.chocolatey         1.2.0  
cisco.aci                     2.2.0  
cisco.asa                     2.1.0  
cisco.intersight              1.0.18 
cisco.ios                     2.8.1  
cisco.iosxr                   2.9.0  
cisco.ise                     1.2.1  
cisco.meraki                  2.6.1  
cisco.mso                     1.4.0  
cisco.nso                     1.0.3  
cisco.nxos                    2.9.1  
cisco.ucs                     1.8.0  
cloud.common                  2.1.0  
cloudscale_ch.cloud           2.2.1  
community.aws                 2.4.0  
community.azure               1.1.0  
community.ciscosmb            1.0.4  
community.crypto              2.2.4  
community.digitalocean        1.16.0 
community.dns                 2.0.9  
community.docker              2.3.0  
community.fortios             1.0.0  
community.general             4.7.0  
community.google              1.0.0  
community.grafana             1.3.3  
community.hashi_vault         2.4.0  
community.hrobot              1.2.3  
community.kubernetes          2.0.1  
community.kubevirt            1.0.0  
community.libvirt             1.0.2  
community.mongodb             1.3.3  
community.mysql               2.3.5  
community.network             3.1.0  
community.okd                 2.1.0  
community.postgresql          1.7.1  
community.proxysql            1.3.1  
community.rabbitmq            1.1.0  
community.routeros            2.0.0  
community.sap                 1.0.0  
community.skydive             1.0.0  
community.sops                1.2.1  
community.vmware              1.18.0 
community.windows             1.9.0  
community.zabbix              1.5.1  
containers.podman             1.9.3  
cyberark.conjur               1.1.0  
cyberark.pas                  1.0.13 
dellemc.enterprise_sonic      1.1.0  
dellemc.openmanage            4.4.0  
dellemc.os10                  1.1.1  
dellemc.os6                   1.0.7  
dellemc.os9                   1.0.4  
f5networks.f5_modules         1.15.0 
fortinet.fortimanager         2.1.4  
fortinet.fortios              2.1.4  
frr.frr                       1.0.3  
gluster.gluster               1.0.2  
google.cloud                  1.0.2  
hetzner.hcloud                1.6.0  
hpe.nimble                    1.1.4  
ibm.qradar                    1.0.3  
infinidat.infinibox           1.3.3  
infoblox.nios_modules         1.2.1  
inspur.sm                     1.3.0  
junipernetworks.junos         2.10.0 
kubernetes.core               2.3.0  
mellanox.onyx                 1.0.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.15.0
netapp.elementsw              21.7.0 
netapp.ontap                  21.17.3
netapp.storagegrid            21.10.0
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.3.0  
netbox.netbox                 3.6.0  
ngine_io.cloudstack           2.2.3  
ngine_io.exoscale             1.0.0  
ngine_io.vultr                1.1.1  
openstack.cloud               1.7.2  
openvswitch.openvswitch       2.1.0  
ovirt.ovirt                   1.6.6  
purestorage.flasharray        1.12.1 
purestorage.flashblade        1.9.0  
sensu.sensu_go                1.13.0 
servicenow.servicenow         1.0.6  
splunk.es                     1.0.2  
t_systems_mms.icinga_director 1.28.0 
theforeman.foreman            2.2.0  
vyos.vyos                     2.8.0  
wti.remote                    1.0.3  
```


### AWS SDK versions

```console (paste below)
$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.21.39
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/lib/python3.8/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.24.39
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/lib/python3.8/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed
ANSIBLE_PIPELINING(/work/ansible/ansible.cfg) = True
DEFAULT_HOST_LIST(/work/ansible/ansible.cfg) = ['/work/ansible/hosts']
DEFAULT_LOG_PATH(/work/ansible/ansible.cfg) = /work/ansible/tmp/ansible.log
INTERPRETER_PYTHON(/work/ansible/ansible.cfg) = auto

```


### OS / Environment

Ubuntu 20.04.3
Python 3.8.10

### Steps to Reproduce

<!--- Paste example playbooks or commands between quotes below -->

1. Create playbook file to create/remove 3 lifecycle rules
```console (paste below)
$ cat << EOF >> lifecycle_wait_issue.yml
---
- hosts: local
  gather_facts: False

  vars:
    bucket: bucket_name
    state: present

  tasks:
  - name: "exemplary cleanup rule"
    community.aws.s3_lifecycle:
      name: "{{ bucket }}"
      rule_id: cleanup_rule_1
      noncurrent_version_expiration_days: 1
      prefix: ""
      status: enabled
      state: "{{ state }}"
      wait: yes

  - name: "{{ bucket }} - exemplary loop rule"
    community.aws.s3_lifecycle:
      name: "{{ bucket }}"
      rule_id: "{{ item.rule_id }}"
      expiration_days: 14
      prefix: "{{ item.prefix }}"
      status: enabled
      state: "{{ state }}"
      wait: yes
    loop:
      - { rule_id: loop_rule_id_1, prefix: prefix_1 }
      - { rule_id: loop_rule_id_2, prefix: prefix_2 }
EOF
```
2. Run below script
NOTE!
Below steps assume there are no other lifecycle rules in the bucket.
```bash
BUCKET=<set-your-bucket-here>
reproduced=false
step=1
#set -x
while [ $step -le 10 ] ; do
  echo "[$step/10] Adding rules..."
  # Add 3 rules according to the playbook
  ansible-playbook -vv lifecycle_wait_issue.yml -e "bucket=$BUCKET" -e "state=present"
  rules=$(aws s3api get-bucket-lifecycle --bucket $BUCKET --query 'Rules[*][ID]' --output text)
  cnt=$(echo "$rules" | wc -l)
  # 3 rules are expected
  if [ $cnt != 3 ] ; then
    echo "issue reproduced, 3 rules expected, $cnt rules found:"
    echo "$rules"
    reproduced=true
    break
  fi
  echo "[$step/10] Removing rules..."
  # Remove all rules
  ansible-playbook -vv lifecycle_wait_issue.yml -e "bucket=$BUCKET" -e "state=absent"
  # expected error: The lifecycle configuration does not exist
  rules=$(aws s3api get-bucket-lifecycle --bucket $BUCKET --query 'Rules[*][ID]' --output text)
  if [ $? -ne 254 ] ; then
    cnt=$(echo "$rules" | wc -l)
    echo "issue reproduced, 0 rules expected, $cnt rules found:"
    echo "$rules"
    reproduced=true
    break
  fi
  step=$(( step + 1))
done

if ! $reproduced ; then
  echo 'issue not reproduced'
fi
```

### Expected Results

Script prints: 
```console
issue not reproduced
```
which means it was able to successfully:
- add 3 rules
- remove 3 added rules
in 10 consecutive steps.

### Actual Results

Script prints:
```console (paste below)
issue reproduced, ...
```
Example 1:
```console
[1/10] Adding rules...
ansible-playbook [core 2.12.4]
  config file = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg
  configured module search path = ['/home/rubin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/lib/python3.8/site-packages/ansible
  ansible collection location = /home/rubin/.ansible/collections:/usr/share/ansible/collections
  executable location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/bin/ansible-playbook
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.1
  libyaml = True
Using /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg as config file
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: lifecycle_wait_issue.yml *********************************************************************************************************************************************************************************
1 plays in lifecycle_wait_issue.yml

PLAY [local] *******************************************************************************************************************************************************************************************************
META: ran handlers

TASK [exemplary cleanup rule] **************************************************************************************************************************************************************************************
task path: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/lifecycle_wait_issue.yml:10
changed: [localhost] => {"_config": {"Rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}, "_retries": 9, "ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": true, "new_rule": {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, "old_rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}], "rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}

TASK [lgi-onemw-staging-dawn-base - exemplary loop rule] ***********************************************************************************************************************************************************
task path: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/lifecycle_wait_issue.yml:20
ok: [localhost] => (item={'rule_id': 'loop_rule_id_1', 'prefix': 'prefix_1'}) => {"_config": {"Rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}, "_retries": 10, "ansible_loop_var": "item", "changed": false, "item": {"prefix": "prefix_1", "rule_id": "loop_rule_id_1"}, "new_rule": {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, "old_rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}], "rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}
ok: [localhost] => (item={'rule_id': 'loop_rule_id_2', 'prefix': 'prefix_2'}) => {"_config": {"Rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}, "_retries": 10, "ansible_loop_var": "item", "changed": false, "item": {"prefix": "prefix_2", "rule_id": "loop_rule_id_2"}, "new_rule": {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, "old_rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}], "rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[1/10] Removing rules...
ansible-playbook [core 2.12.4]
  config file = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg
  configured module search path = ['/home/rubin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/lib/python3.8/site-packages/ansible
  ansible collection location = /home/rubin/.ansible/collections:/usr/share/ansible/collections
  executable location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/bin/ansible-playbook
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.1
  libyaml = True
Using /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg as config file
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: lifecycle_wait_issue.yml *********************************************************************************************************************************************************************************
1 plays in lifecycle_wait_issue.yml

PLAY [local] *******************************************************************************************************************************************************************************************************
META: ran handlers

TASK [exemplary cleanup rule] **************************************************************************************************************************************************************************************
task path: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/lifecycle_wait_issue.yml:10
changed: [localhost] => {"_retries": 9, "ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": true, "old_rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}], "rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}]}

TASK [lgi-onemw-staging-dawn-base - exemplary loop rule] ***********************************************************************************************************************************************************
task path: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/lifecycle_wait_issue.yml:20
changed: [localhost] => (item={'rule_id': 'loop_rule_id_1', 'prefix': 'prefix_1'}) => {"_retries": 9, "ansible_loop_var": "item", "changed": true, "item": {"prefix": "prefix_1", "rule_id": "loop_rule_id_1"}, "old_rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}], "rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}]}
changed: [localhost] => (item={'rule_id': 'loop_rule_id_2', 'prefix': 'prefix_2'}) => {"_retries": 8, "ansible_loop_var": "item", "changed": true, "item": {"prefix": "prefix_2", "rule_id": "loop_rule_id_2"}, "old_rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}], "rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}]}
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   


An error occurred (NoSuchLifecycleConfiguration) when calling the GetBucketLifecycle operation: The lifecycle configuration does not exist
[2/10] Adding rules...
ansible-playbook [core 2.12.4]
  config file = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg
  configured module search path = ['/home/rubin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/lib/python3.8/site-packages/ansible
  ansible collection location = /home/rubin/.ansible/collections:/usr/share/ansible/collections
  executable location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/bin/ansible-playbook
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.1
  libyaml = True
Using /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg as config file
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: lifecycle_wait_issue.yml ********************************************************************************************************************************************************************************************************************************************************
1 plays in lifecycle_wait_issue.yml

PLAY [local] ******************************************************************************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [exemplary cleanup rule] *************************************************************************************************************************************************************************************************************************************************************
task path: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/lifecycle_wait_issue.yml:10
changed: [localhost] => {"_config": {"Rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}, "_retries": 8, "ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": true, "new_rule": {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, "old_rules": [], "rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}

TASK [lgi-onemw-staging-dawn-base - exemplary loop rule] **********************************************************************************************************************************************************************************************************************************
task path: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/lifecycle_wait_issue.yml:20
changed: [localhost] => (item={'rule_id': 'loop_rule_id_1', 'prefix': 'prefix_1'}) => {"_config": {"Rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}]}, "_retries": 9, "ansible_loop_var": "item", "changed": true, "item": {"prefix": "prefix_1", "rule_id": "loop_rule_id_1"}, "new_rule": {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, "old_rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}], "rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}]}
changed: [localhost] => (item={'rule_id': 'loop_rule_id_2', 'prefix': 'prefix_2'}) => {"_config": {"Rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}]}, "_retries": 9, "ansible_loop_var": "item", "changed": true, "item": {"prefix": "prefix_2", "rule_id": "loop_rule_id_2"}, "new_rule": {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, "old_rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}], "rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}]}
META: ran handlers
META: ran handlers

PLAY RECAP ********************************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[2/10] Removing rules...
ansible-playbook [core 2.12.4]
  config file = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg
  configured module search path = ['/home/rubin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/lib/python3.8/site-packages/ansible
  ansible collection location = /home/rubin/.ansible/collections:/usr/share/ansible/collections
  executable location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/bin/ansible-playbook
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.1
  libyaml = True
Using /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg as config file
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: lifecycle_wait_issue.yml ********************************************************************************************************************************************************************************************************************************************************
1 plays in lifecycle_wait_issue.yml

PLAY [local] ******************************************************************************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [exemplary cleanup rule] *************************************************************************************************************************************************************************************************************************************************************
task path: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/lifecycle_wait_issue.yml:10
changed: [localhost] => {"_retries": 9, "ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": true, "old_rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}], "rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}]}

TASK [lgi-onemw-staging-dawn-base - exemplary loop rule] **********************************************************************************************************************************************************************************************************************************
task path: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/lifecycle_wait_issue.yml:20
changed: [localhost] => (item={'rule_id': 'loop_rule_id_1', 'prefix': 'prefix_1'}) => {"_retries": 8, "ansible_loop_var": "item", "changed": true, "item": {"prefix": "prefix_1", "rule_id": "loop_rule_id_1"}, "old_rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}], "rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}]}
changed: [localhost] => (item={'rule_id': 'loop_rule_id_2', 'prefix': 'prefix_2'}) => {"_retries": 9, "ansible_loop_var": "item", "changed": true, "item": {"prefix": "prefix_2", "rule_id": "loop_rule_id_2"}, "old_rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}], "rules": [{"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}
META: ran handlers
META: ran handlers

PLAY RECAP ********************************************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

issue reproduced, 0 rules expected, 2 rules found:
cleanup_rule_1
loop_rule_id_2

```
Example 2:
```console
[1/10] Adding rules...
ansible-playbook [core 2.12.4]
  config file = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg
  configured module search path = ['/home/rubin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/lib/python3.8/site-packages/ansible
  ansible collection location = /home/rubin/.ansible/collections:/usr/share/ansible/collections
  executable location = /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/env/bin/ansible-playbook
  python version = 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
  jinja version = 3.1.1
  libyaml = True
Using /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/ansible.cfg as config file
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: lifecycle_wait_issue.yml *********************************************************************************************************************************************************************************
1 plays in lifecycle_wait_issue.yml

PLAY [local] *******************************************************************************************************************************************************************************************************
META: ran handlers

TASK [exemplary cleanup rule] **************************************************************************************************************************************************************************************
task path: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/lifecycle_wait_issue.yml:10
changed: [localhost] => {"_config": {"Rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}, "_retries": 8, "ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": true, "new_rule": {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}, "old_rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}], "rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Filter": {"Prefix": ""}, "ID": "cleanup_rule_1", "NoncurrentVersionExpiration": {"NoncurrentDays": 1}, "Status": "Enabled"}]}

TASK [lgi-onemw-staging-dawn-base - exemplary loop rule] ***********************************************************************************************************************************************************
task path: /work/infra/jenkins/onemw-tools/onemw-infrastructure/ansible/lifecycle_wait_issue.yml:20
changed: [localhost] => (item={'rule_id': 'loop_rule_id_1', 'prefix': 'prefix_1'}) => {"_config": {"Rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}]}, "_retries": 9, "ansible_loop_var": "item", "changed": true, "item": {"prefix": "prefix_1", "rule_id": "loop_rule_id_1"}, "new_rule": {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}, "old_rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}], "rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}]}
ok: [localhost] => (item={'rule_id': 'loop_rule_id_2', 'prefix': 'prefix_2'}) => {"_config": {"Rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}]}, "_retries": 10, "ansible_loop_var": "item", "changed": false, "item": {"prefix": "prefix_2", "rule_id": "loop_rule_id_2"}, "new_rule": {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, "old_rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}], "rules": [{"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_2"}, "ID": "loop_rule_id_2", "Status": "Enabled"}, {"Expiration": {"Days": 14}, "Filter": {"Prefix": "prefix_1"}, "ID": "loop_rule_id_1", "Status": "Enabled"}]}
META: ran handlers
META: ran handlers

PLAY RECAP *********************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

issue reproduced, 3 rules expected, 2 rules found:
loop_rule_id_2
loop_rule_id_1
```
It cannot reliably add or remove rules from playbook.

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/s3_lifecycle.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/s3_lifecycle.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @jillr @markuman @s-hertel @tremble
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/plugins/modules/s3_lifecycle.py b/plugins/modules/s3_lifecycle.py
--- a/plugins/modules/s3_lifecycle.py
+++ b/plugins/modules/s3_lifecycle.py
@@ -485,15 +485,23 @@ def create_lifecycle_rule(client, module):
 
     _changed = changed
     _retries = 10
-    while wait and _changed and _retries:
+    _not_changed_cnt = 6
+    while wait and _changed and _retries and _not_changed_cnt:
         # We've seen examples where get_bucket_lifecycle_configuration returns
-        # the updated rules, then the old rules, then the updated rules again,
+        # the updated rules, then the old rules, then the updated rules again and
+        # again couple of times.
+        # Thus try to read the rule few times in a row to check if it has changed.
         time.sleep(5)
         _retries -= 1
         new_rules = fetch_rules(client, module, name)
         (_changed, lifecycle_configuration) = compare_and_update_configuration(client, module,
                                                                                new_rules,
                                                                                new_rule)
+        if not _changed:
+            _not_changed_cnt -= 1
+            _changed = True
+        else:
+            _not_changed_cnt = 6
 
     new_rules = fetch_rules(client, module, name)
 
@@ -531,13 +539,21 @@ def destroy_lifecycle_rule(client, module):
 
     _changed = changed
     _retries = 10
-    while wait and _changed and _retries:
+    _not_changed_cnt = 6
+    while wait and _changed and _retries and _not_changed_cnt:
         # We've seen examples where get_bucket_lifecycle_configuration returns
-        # the updated rules, then the old rules, then the updated rules again,
+        # the updated rules, then the old rules, then the updated rules again and
+        # again couple of times.
+        # Thus try to read the rule few times in a row to check if it has changed.
         time.sleep(5)
         _retries -= 1
         new_rules = fetch_rules(client, module, name)
         (_changed, lifecycle_configuration) = compare_and_remove_rule(new_rules, rule_id, prefix)
+        if not _changed:
+            _not_changed_cnt -= 1
+            _changed = True
+        else:
+            _not_changed_cnt = 6
 
     new_rules = fetch_rules(client, module, name)
 


Test patch:
diff --git a/tests/integration/targets/s3_lifecycle/tasks/main.yml b/tests/integration/targets/s3_lifecycle/tasks/main.yml
--- a/tests/integration/targets/s3_lifecycle/tasks/main.yml
+++ b/tests/integration/targets/s3_lifecycle/tasks/main.yml
@@ -24,6 +24,42 @@
           - output.changed
           - output.name == bucket_name
           - not output.requester_pays
+  # ============================================================
+    - name: Create a number of policies and check if all of them are created
+      community.aws.s3_lifecycle:
+        name: "{{ bucket_name }}"
+        rule_id: "{{ item }}"
+        expiration_days: 10
+        prefix: "{{ item }}"
+        status: enabled
+        state: present
+        wait: yes
+      register: output
+      loop:
+        - rule_1
+        - rule_2
+        - rule_3
+    - assert:
+        that:
+          - (output.results | last).rules | length == 3
+  # ============================================================
+    - name: Remove previously created policies and check if all of them are removed
+      community.aws.s3_lifecycle:
+        name: "{{ bucket_name }}"
+        rule_id: "{{ item }}"
+        expiration_days: 10
+        prefix: "{{ item }}"
+        status: enabled
+        state: absent
+        wait: yes
+      register: output
+      loop:
+        - rule_1
+        - rule_2
+        - rule_3
+    - assert:
+        that:
+          - (output.results | last).rules | length == 0
   # ============================================================
     - name: Create a lifecycle policy
       s3_lifecycle: