Repository: ansible-collections/amazon.aws

Issue:
amazon.aws.s3_bucket: Setting policy to None/null does not remove bucket policy
##### SUMMARY
When removing the `policy` attribute from an `s3_bucket` task, the module does not seem to remove the policy and reports no changes.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
s3_bucket

##### ANSIBLE VERSION
```paste below
ansible [core 2.11.1] 
  config file = /path/to/my/project/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.9.5 (default, May 24 2021, 12:50:35) [GCC 11.1.0]
  jinja version = 3.0.1
  libyaml = True
```

##### CONFIGURATION
(`ansible-config dump --only-changed` reported no changes)

Note that I used an S3-compatible third-party provider for running the module. I am aware that this might not be supported. However, I think that the issue also affects AWS, as the cause seems lie within the module itself (see below).

##### OS / ENVIRONMENT
Manjaro 21.0.6

##### STEPS TO REPRODUCE
```yaml
- name: Create a bucket with a policy and then remove the policy
  hosts: localhost
  connection: local
  become: no
  vars:
    ansible_python_interpreter: "{{ansible_playbook_python}}"
   bucket_name: "my-unique-bucket-name"
  tasks:
    - name: Create a bucket with attached policy
      amazon.aws.s3_bucket:
        name: "{{ bucket_name }}"
        state: present
        policy: "{{ lookup('template', 'bucket-policy-public-access.json.j2') }}"
    - name: Try to remove the bucket's policy by omitting the policy key 
      amazon.aws.s3_bucket:
        name: "{{ bucket_name }}"
        state: present
    - name: Try to remove the bucket's policy by explicitly setting the policy to `null`
      amazon.aws.s3_bucket:
        name: "{{ bucket_name }}"
        state: present
        policy: null
```
The file `bucket-policy-public-access.json.j2` contained
```json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowPublicRead",
    "Effect": "Allow",
    "Principal": "*",
    "Action": ["s3:GetObject"],
    "Resource": [ "arn:aws:s3:::{{ bucket_name }}/*" ]
  }]
}
```

##### EXPECTED RESULTS
I expected the first task to create a bucket with an attached policy. I then expected either of the second and third tasks to remove said policy


##### ACTUAL RESULTS
The first task created the bucket with attached policy, as expected. Neither of the second and third tasks removed the policy from the bucket. I attached a sanitized output below.

From looking at the module code, I think there might be a bug here: https://github.com/ansible-collections/amazon.aws/blob/0cdc47bb57109995c77bd986479db3c453751f3f/plugins/modules/s3_bucket.py#L360-L366 (current `main` as of submitting this issue). Line 360 ensures that only non-`None` values of `policy` will enter line 364. ~As far as I can tell, the predicate on line 364 always evaluates to `False` under this condition. Hence, Line 366 will never be reached, which means that bucket policies may not be deleted.~ (EDIT: this is not true. Setting `policy` to the string value `'null'` will lead to policy deletion. See my other comment below. Maybe this is simply an issue of documentation?)

<!--- Paste verbatim command output between quotes -->
```paste below
ansible-playbook [core 2.11.1] 
  config file = /home/user/s3_bucket_example/ansible.cfg
  configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.9/site-packages/ansible
  ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible-playbook
  python version = 3.9.5 (default, May 24 2021, 12:50:35) [GCC 11.1.0]
  jinja version = 3.0.1
  libyaml = True
Using /home/user/s3_bucket_example/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /home/user/s3_bucket_example/inventory as it did not pass its verify_file() method
script declined parsing /home/user/s3_bucket_example/inventory as it did not pass its verify_file() method
auto declined parsing /home/user/s3_bucket_example/inventory as it did not pass its verify_file() method
Set default localhost to localhost
Parsed /home/user/s3_bucket_example/inventory inventory source with ini plugin
Loading collection amazon.aws from /home/user/.ansible/collections/ansible_collections/amazon/aws
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: main.yml *************************************************************
Positional arguments: main.yml
verbosity: 4
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/home/user/s3_bucket_example/inventory',)
forks: 5
1 plays in main.yml
Read vars_file 'vars/main.yml'
Read vars_file 'vars/main.yml'
Read vars_file 'vars/main.yml'

PLAY [Do some things] **********************************************************
Read vars_file 'vars/main.yml'

TASK [Gathering Facts] *********************************************************
task path: /home/user/s3_bucket_example/main.yml:2
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: user
<localhost> EXEC /bin/sh -c 'echo ~user && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp `"&& mkdir "` echo /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073 `" && echo ansible-tmp-1623491053.5475247-40397-239535370960073="` echo /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073 `" ) && sleep 0'
Using module file /usr/lib/python3.9/site-packages/ansible/modules/setup.py
<localhost> PUT /home/user/.ansible/tmp/ansible-local-40393dgnhyxmv/tmp6mg63skd TO /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073/AnsiballZ_setup.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073/ /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073/AnsiballZ_setup.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073/AnsiballZ_setup.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/user/.ansible/tmp/ansible-tmp-1623491053.5475247-40397-239535370960073/ > /dev/null 2>&1 && sleep 0'
ok: [localhost]
Read vars_file 'vars/main.yml'
META: ran handlers
Read vars_file 'vars/main.yml'

TASK [amazon.aws.s3_bucket] ****************************************************
task path: /home/user/s3_bucket_example/main.yml:11
File lookup using /home/user/s3_bucket_example/templates/bucket-policy-public-access.json.j2 as file
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: user
<localhost> EXEC /bin/sh -c 'echo ~user && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp `"&& mkdir "` echo /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415 `" && echo ansible-tmp-1623491054.2478297-40459-110095007754415="` echo /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415 `" ) && sleep 0'
Using module file /home/user/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py
<localhost> PUT /home/user/.ansible/tmp/ansible-local-40393dgnhyxmv/tmpq_ku2_9n TO /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415/AnsiballZ_s3_bucket.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415/ /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/user/.ansible/tmp/ansible-tmp-1623491054.2478297-40459-110095007754415/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => (item={'name': 'my-unique-bucket-name'}) => {
    "ansible_loop_var": "item",
    "changed": true,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "ceph": false,
            "debug_botocore_endpoint_logs": false,
            "delete_public_access": false,
            "ec2_url": null,
            "encryption": null,
            "encryption_key_id": null,
            "force": false,
            "name": "my-unique-bucket-name",
            "policy": "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Sid\": \"AllowPublicRead\", \"Effect\": \"Allow\", \"Principal\": \"*\", \"Action\": [\"s3:GetObject\"], \"Resource\": [\"arn:aws:s3:::my-unique-bucket-name/*\"]}]}",
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": null,
            "requester_pays": null,
            "s3_url": "https://s3.hidrive.strato.com",
            "security_token": null,
            "state": "absent",
            "tags": null,
            "validate_certs": true,
            "versioning": null
        }
    },
    "item": {
        "name": "my-unique-bucket-name"
    }
}
Read vars_file 'vars/main.yml'

TASK [amazon.aws.s3_bucket] ****************************************************
task path: /home/user/s3_bucket_example/main.yml:19
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: user
<localhost> EXEC /bin/sh -c 'echo ~user && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp `"&& mkdir "` echo /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489 `" && echo ansible-tmp-1623491055.092912-40475-1394715121489="` echo /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489 `" ) && sleep 0'
Using module file /home/user/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py
<localhost> PUT /home/user/.ansible/tmp/ansible-local-40393dgnhyxmv/tmp0s83nn1b TO /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489/AnsiballZ_s3_bucket.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489/ /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/user/.ansible/tmp/ansible-tmp-1623491055.092912-40475-1394715121489/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => (item={'name': 'my-unique-bucket-name'}) => {
    "ansible_loop_var": "item",
    "changed": true,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "ceph": false,
            "debug_botocore_endpoint_logs": false,
            "delete_public_access": false,
            "ec2_url": null,
            "encryption": null,
            "encryption_key_id": null,
            "force": false,
            "name": "my-unique-bucket-name",
            "policy": null,
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": null,
            "requester_pays": null,
            "s3_url": "https://s3.hidrive.strato.com",
            "security_token": null,
            "state": "present",
            "tags": null,
            "validate_certs": true,
            "versioning": null
        }
    },
    "item": {
        "name": "my-unique-bucket-name"
    },
    "name": "my-unique-bucket-name",
    "policy": null
}
Read vars_file 'vars/main.yml'

TASK [amazon.aws.s3_bucket] ****************************************************
task path: /home/user/s3_bucket_example/main.yml:26
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: user
<localhost> EXEC /bin/sh -c 'echo ~user && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/user/.ansible/tmp `"&& mkdir "` echo /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275 `" && echo ansible-tmp-1623491055.923643-40491-193397245992275="` echo /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275 `" ) && sleep 0'
Using module file /home/user/.ansible/collections/ansible_collections/amazon/aws/plugins/modules/s3_bucket.py
<localhost> PUT /home/user/.ansible/tmp/ansible-local-40393dgnhyxmv/tmp8ldwx5gt TO /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275/AnsiballZ_s3_bucket.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275/ /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275/AnsiballZ_s3_bucket.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/user/.ansible/tmp/ansible-tmp-1623491055.923643-40491-193397245992275/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => (item={'name': 'my-unique-bucket-name'}) => {
    "ansible_loop_var": "item",
    "changed": false,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "ceph": false,
            "debug_botocore_endpoint_logs": false,
            "delete_public_access": false,
            "ec2_url": null,
            "encryption": null,
            "encryption_key_id": null,
            "force": false,
            "name": "my-unique-bucket-name",
            "policy": null,
            "profile": null,
            "public_access": null,
            "purge_tags": true,
            "region": null,
            "requester_pays": null,
            "s3_url": "https://s3.hidrive.strato.com",
            "security_token": null,
            "state": "present",
            "tags": null,
            "validate_certs": true,
            "versioning": null
        }
    },
    "item": {
        "name": "my-unique-bucket-name"
    },
    "name": "my-unique-bucket-name",
    "policy": null
}
Read vars_file 'vars/main.yml'
META: ran handlers
Read vars_file 'vars/main.yml'
META: ran handlers

PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
```



Discussion:
Files identified in the description:
* [`plugins/modules/s3_bucket.py`](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/s3_bucket.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @abikouo @jillr @s-hertel @tremble @wimnat
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
One addendum: I just realized that removing bucket policies works when setting the `policy` configuration key to `'null'`, i.e. the string value. Maybe this is the intended functionality, and it is simply a matter of adding a short example to the module documentation? I would be happy to provide one, e.g.

```yaml
# Ensure no policy is attached to the bucket
        - amazon.aws.s3_bucket:
            name: mys3bucket
            state: present
            policy: 'null'
```
Hi @2d6,

Thank you for taking the time to report this. Would you like to submit a documentation fix?

I just created a small PR to add this case to the documentation :)

Resolving patch:
diff --git a/plugins/modules/s3_bucket.py b/plugins/modules/s3_bucket.py
--- a/plugins/modules/s3_bucket.py
+++ b/plugins/modules/s3_bucket.py
@@ -41,7 +41,7 @@
     type: str
   policy:
     description:
-      - The JSON policy as a string.
+      - The JSON policy as a string. Set to the string C("null") to force the absence of a policy.
     type: json
   s3_url:
     description:
@@ -212,7 +212,7 @@
     public_access:
         block_public_acls: true
         ignore_public_acls: true
-        ## keys == 'false' can be ommited, undefined keys defaults to 'false'
+        ## keys == 'false' can be omitted, undefined keys defaults to 'false'
         # block_public_policy: false
         # restrict_public_buckets: false
 
@@ -233,6 +233,12 @@
     name: mys3bucket
     state: present
     delete_object_ownership: true
+
+# Delete a bucket policy from bucket
+- amazon.aws.s3_bucket:
+    name: mys3bucket
+    state: present
+    policy: "null"
 '''
 
 import json


Test patch: