Repository: ansible/ansible

Issue:
s3_bucket: Force parameter doesn't handle versioned objects
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and devel branch are affected too.
Always add information AFTER of these html comments.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
 - Bug Report

##### COMPONENT NAME
<!---
Insert below this comment the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path
-->
s3_bucket
##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.3.0
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
None
##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
Mac OS 10.12.6
##### SUMMARY
<!--- Explain the problem briefly -->
The s3_bucket module's `force` parameter doesn't currently handle versioned objects, instead only deleting unversioned current objects.
##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Example giving using boto3 in ipython session:
```python
import boto3
session = boto3.session.Session(region_name='eu-west-1')
s3_client = session.client('s3')
s3_client.create_bucket(Bucket='ansible-test', CreateBucketConfiguration={'LocationConstraint': 'eu-west-1'})
s3_client.put_bucket_versioning(Bucket='ansible-test', VersioningConfiguration={'Status': 'Enabled'})
s3_client.put_object(Body=b'testing', Bucket='ansible-test', Key='test-versioning')
s3_client.put_object(Body=b'testing2', Bucket='ansible-test', Key='test-versioning')
```
Now attempt to run playbook below.
<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: Delete versioned S3 bucket
  hosts: localhost
  connection: local
  become: yes
  gather_facts: false

  tasks:
  - name: Delete backup buckets
    s3_bucket:
      name: "ansible-test"
      state: absent
      region: "eu-west-1"
      force: yes
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
All objects in the bucket, including versioned objects to be deleted, and the bucket to be successfully deleted. 
##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
S3 bucket name has been changed to `s3-bucket-name` instead of actual output
```
The full traceback is:
  File "/tmp/ansible_w8964hbx/ansible_module_s3_bucket.py", line 372, in _destroy_bucket
    bucket = connection.delete_bucket(name)
  File "/*/venv/lib/python3.5/site-packages/boto/s3/connection.py", line 648, in delete_bucket
    response.status, response.reason, body)

failed: [localhost] (item=s3-bucket-name) => {
    "changed": false,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_secret_key": null,
            "ceph": false,
            "ec2_url": null,
            "force": true,
            "name": "s3-bucket-name",
            "policy": null,
            "profile": null,
            "region": "eu-west-1",
            "requester_pays": false,
            "s3_url": null,
            "security_token": null,
            "state": "absent",
            "tags": null,
            "validate_certs": true,
            "versioning": null
        }
    },
    "item": "s3-bucket-name",
    "msg": "The bucket you tried to delete is not empty. You must delete all versions in the bucket."
}
```



Discussion:
I'm intending to do the work required to fix this myself, basing it off the existing PR for migrating to boto3 https://github.com/ansible/ansible/pull/37189 given the improvements already wrapped into that. 

Raising this for visibility's sake.
cc @willthames @wimnat
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/s3_bucket.py b/lib/ansible/modules/cloud/amazon/s3_bucket.py
--- a/lib/ansible/modules/cloud/amazon/s3_bucket.py
+++ b/lib/ansible/modules/cloud/amazon/s3_bucket.py
@@ -30,7 +30,8 @@
 options:
   force:
     description:
-      - When trying to delete a bucket, delete all keys in the bucket first (an s3 bucket must be empty for a successful deletion)
+      - When trying to delete a bucket, delete all keys (including versions and delete markers)
+        in the bucket first (an s3 bucket must be empty for a successful deletion)
     type: bool
     default: 'no'
   name:
@@ -417,6 +418,13 @@ def paginated_list(s3_client, **pagination_params):
         yield [data['Key'] for data in page.get('Contents', [])]
 
 
+def paginated_versions_list(s3_client, **pagination_params):
+    pg = s3_client.get_paginator('list_object_versions')
+    for page in pg.paginate(**pagination_params):
+        # We have to merge the Versions and DeleteMarker lists here, as DeleteMarkers can still prevent a bucket deletion
+        yield [(data['Key'], data['VersionId']) for data in (page.get('Versions', []) + page.get('DeleteMarkers', []))]
+
+
 def destroy_bucket(s3_client, module):
 
     force = module.params.get("force")
@@ -432,10 +440,10 @@ def destroy_bucket(s3_client, module):
         module.exit_json(changed=False)
 
     if force:
-        # if there are contents then we need to delete them before we can delete the bucket
+        # if there are contents then we need to delete them (including versions) before we can delete the bucket
         try:
-            for keys in paginated_list(s3_client, Bucket=name):
-                formatted_keys = [{'Key': key} for key in keys]
+            for key_version_pairs in paginated_versions_list(s3_client, Bucket=name):
+                formatted_keys = [{'Key': key, 'VersionId': version} for key, version in key_version_pairs]
                 if formatted_keys:
                     s3_client.delete_objects(Bucket=name, Delete={'Objects': formatted_keys})
         except (BotoCoreError, ClientError) as e:


Test patch: