Repository: ansible/ansible

Issue:
S3 module stopped working in 2.4.1
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
aws_s3 module

##### ANSIBLE VERSION
```
ansible 2.4.1.0
  config file = /home/jwitkowski/throtle-ansible/ansible.cfg
  configured module search path = [u'/home/jwitkowski/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/jwitkowski/throtle-ansible-venv/lib/python2.7/site-packages/ansible
  executable location = /home/jwitkowski/throtle-ansible-venv/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
```

##### CONFIGURATION
N/A

##### OS / ENVIRONMENT
N/A

##### SUMMARY
The same ansible code that worked in v2.3.2 to create S3 buckets no longer works in 2.4.1.  There is nothing fancy or tricky being done and I've tried to switch the module from the old "s3:" to the new "aws_s3:" to no avail.

##### STEPS TO REPRODUCE
```yaml
---
- name: Apply Service Roles to Instance
  hosts: local
  gather_facts: True
  vars:
    s3_buckets: ["test-bucket"]
    s3_iam_users: [ { name: "testUser", bucket: "test-bucket" } ]
    s3_path_suffixes: ['Download','Upload']
  environment:
    AWS_DEFAULT_REGION: "{{ region }}"
    AWS_ACCESS_KEY_ID: "{{ aws_access }}"
    AWS_SECRET_ACCESS_KEY: "{{ aws_secret }}"
  tasks:
    - name: S3 | Create AWS S3 buckets
      s3:
        bucket: "{{ item.0 }}"
        mode: create
        object: "{{ s3_path_prefix }}/{{ item.1.dir|default(item.1.name) }}/{{ item.2 }}/"
      with_nested:
        - "{{ s3_buckets }}"
        - "{{ s3_iam_users }}"
        - "{{ s3_path_suffixes }}"

    - name: S3 | Create AWS IAM user for S3
      iam:
        iam_type: user
        name: "{{ item.name }}"
        state: present
        access_key_state: create
      register: s3_user_creation
      with_items: "{{ s3_iam_users }}"

    - name: S3 | Attach Bucket policy to user
      iam_policy:
        iam_type: user
        iam_name: "{{ item.name }}"
        policy_name: "s3_partner_{{ item.name }}"
        state: present
        policy_json: "{{ lookup('template', 'vars/templates/s3_iam_policy.j2') }}"
      with_items: "{{ s3_iam_users }}"

    - debug:
        msg:
          - "***** SECRET KEY IS ONLY GIVEN ONCE *****"
          - "PLEASE TAKE NOTE:"
          - "    AWS_ACCESS_KEY: {{ item.user_meta.access_keys[0].access_key_id }}"
          - "    AWS_SECRET_KEY: {{ item.user_meta.access_keys[0].secret_access_key }}"
          - "*****************************************"
      with_items: "{{ s3_user_creation.results }}"
      when: item.user_meta is defined and s3_user_creation.changed
```

##### EXPECTED RESULTS
In ansible 2.3.2 the above runs without issue and creates the IAM user, bucket policy, and attaches the policy to the user with no issues.

##### ACTUAL RESULTS
in ansible 2.4.1 latest release:
```
Using module file /home/jwitkowski/throtle-ansible-venv/lib/python2.7/site-packages/ansible/modules/cloud/amazon/_s3.py
<localhost> EXEC /bin/sh -c 'AWS_DEFAULT_REGION=us-east-1 AWS_ACCESS_KEY_ID=AWS_SECRET_ACCESS_KEY= python && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_aqikeG/ansible_module_s3.py", line 830, in <module>
    main()
  File "/tmp/ansible_aqikeG/ansible_module_s3.py", line 796, in main
    create_dirkey(module, s3, bucket, dirobj)
  File "/tmp/ansible_aqikeG/ansible_module_s3.py", line 378, in create_dirkey
    bucket = s3.Bucket(bucket)
  File "/home/jwitkowski/throtle-ansible-venv/lib/python2.7/site-packages/botocore/client.py", line 553, in __getattr__
    self.__class__.__name__, item)
AttributeError: 'S3' object has no attribute 'Bucket'

failed: [localhost] (item=[u'throtle-data', {u'bucket': u'throtle-data', u'name': u'testing'}, u'Download']) => {
    "changed": false,
    "failed": true,
    "item": [
        "throtle-data",
        {
            "bucket": "throtle-data",
            "name": "testing"
        },
        "Download"
    ],
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_aqikeG/ansible_module_s3.py\", line 830, in <module>\n    main()\n  File \"/tmp/ansible_aqikeG/ansible_module_s3.py\", line 796, in main\n    create_dirkey(module, s3, bucket, dirobj)\n  File \"/tmp/ansible_aqikeG/ansible_module_s3.py\", line 378, in create_dirkey\n    bucket = s3.Bucket(bucket)\n  File \"/home/jwitkowski/throtle-ansible-venv/lib/python2.7/site-packages/botocore/client.py\", line 553, in __getattr__\n    self.__class__.__name__, item)\nAttributeError: 'S3' object has no attribute 'Bucket'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}
Using module file /home/jwitkowski/throtle-ansible-venv/lib/python2.7/site-packages/ansible/modules/cloud/amazon/_s3.py
<localhost> EXEC /bin/sh -c 'AWS_DEFAULT_REGION=us-east-1 AWS_ACCESS_KEY_ID=AWS_SECRET_ACCESS_KEY= python && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_AGetjO/ansible_module_s3.py", line 830, in <module>
    main()
  File "/tmp/ansible_AGetjO/ansible_module_s3.py", line 796, in main
    create_dirkey(module, s3, bucket, dirobj)
  File "/tmp/ansible_AGetjO/ansible_module_s3.py", line 378, in create_dirkey
    bucket = s3.Bucket(bucket)
  File "/home/jwitkowski/throtle-ansible-venv/lib/python2.7/site-packages/botocore/client.py", line 553, in __getattr__
    self.__class__.__name__, item)
AttributeError: 'S3' object has no attribute 'Bucket'

failed: [localhost] (item=[u'throtle-data', {u'bucket': u'throtle-data', u'name': u'testing'}, u'Upload']) => {
    "changed": false,
    "failed": true,
    "item": [
        "throtle-data",
        {
            "bucket": "throtle-data",
            "name": "testing"
        },
        "Upload"
    ],
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_AGetjO/ansible_module_s3.py\", line 830, in <module>\n    main()\n  File \"/tmp/ansible_AGetjO/ansible_module_s3.py\", line 796, in main\n    create_dirkey(module, s3, bucket, dirobj)\n  File \"/tmp/ansible_AGetjO/ansible_module_s3.py\", line 378, in create_dirkey\n    bucket = s3.Bucket(bucket)\n  File \"/home/jwitkowski/throtle-ansible-venv/lib/python2.7/site-packages/botocore/client.py\", line 553, in __getattr__\n    self.__class__.__name__, item)\nAttributeError: 'S3' object has no attribute 'Bucket'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}

```



Discussion:
cc @lwade @s-hertel
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I see - I think I have a fix for this issue, but it won't be making 2.4.1. I'll ensure the fix is in the 2.4.2 beta release, and of course fix the `devel` version.

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/aws_s3.py b/lib/ansible/modules/cloud/amazon/aws_s3.py
--- a/lib/ansible/modules/cloud/amazon/aws_s3.py
+++ b/lib/ansible/modules/cloud/amazon/aws_s3.py
@@ -408,12 +408,10 @@ def create_dirkey(module, s3, bucket, obj):
     if module.check_mode:
         module.exit_json(msg="PUT operation skipped - running in check mode", changed=True)
     try:
-        bucket = s3.Bucket(bucket)
-        key = bucket.new_key(obj)
-        key.set_contents_from_string('')
+        s3.put_object(Bucket=bucket, Key=obj, Body=b'')
         for acl in module.params.get('permission'):
             s3.put_object_acl(ACL=acl, Bucket=bucket, Key=obj)
-        module.exit_json(msg="Virtual directory %s created in bucket %s" % (obj, bucket.name), changed=True)
+        module.exit_json(msg="Virtual directory %s created in bucket %s" % (obj, bucket), changed=True)
     except botocore.exceptions.ClientError as e:
         module.fail_json(msg="Failed while creating object %s." % obj, exception=traceback.format_exc(), **camel_dict_to_snake_dict(e.response))
 


Test patch:
diff --git a/test/integration/targets/aws_s3/aliases b/test/integration/targets/aws_s3/aliases
--- a/test/integration/targets/aws_s3/aliases
+++ b/test/integration/targets/aws_s3/aliases
@@ -1,2 +1,2 @@
 cloud/aws
-posix/ci/cloud/aws
+posix/ci/cloud/group1/aws
diff --git a/test/integration/targets/aws_s3/tasks/main.yml b/test/integration/targets/aws_s3/tasks/main.yml
--- a/test/integration/targets/aws_s3/tasks/main.yml
+++ b/test/integration/targets/aws_s3/tasks/main.yml
@@ -2,7 +2,7 @@
 # tasks file for test_s3
 # ============================================================
 - name: test create bucket
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name }}"
     mode: create
     aws_access_key: "{{ ec2_access_key }}"
@@ -15,7 +15,7 @@
       - result.changed == True
 # ============================================================
 - name: trying to create a bucket name that already exists
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name }}"
     mode: create
     aws_access_key: "{{ ec2_access_key }}"
@@ -45,7 +45,7 @@
   register: file1stat
 # ============================================================
 - name: test putting an object in the bucket
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name }}"
     mode: put
     src: "{{ tmp1.path }}"
@@ -66,7 +66,7 @@
   tempfile:
   register: tmp2
 - name: test get object
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name }}"
     mode: get
     dest: "{{ tmp2.path }}"
@@ -89,7 +89,7 @@
       - file1stat.stat.checksum == file2stat.stat.checksum
 # ============================================================
 - name: test geturl of the object
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name }}"
     mode: geturl
     object: delete.txt
@@ -107,7 +107,7 @@
       - result.changed == True
 # ============================================================
 - name: test getstr of the object
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name }}"
     mode: getstr
     object: delete.txt
@@ -124,7 +124,7 @@
       - result.contents == content
 # ============================================================
 - name: test list to get all objects in the bucket
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name }}"
     mode: list
     security_token: "{{security_token}}"
@@ -140,7 +140,7 @@
       - result.msg == "LIST operation complete"
 # ============================================================
 - name: test delobj to just delete an object in the bucket
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name }}"
     mode: delobj
     object: delete.txt
@@ -155,9 +155,41 @@
     that:
       - "'Object deleted from bucket' in result.msg"
       - result.changed == True
+- name: assert that delete.txt is no longer an object in the bucket deleteme
+  assert:
+    that:
+      - "'Object deleted from bucket' in result.msg"
+      - result.changed == True
+# ============================================================
+- name: test creation of empty path
+  aws_s3:
+    bucket: "{{ bucket_name }}"
+    mode: create
+    object: foo/bar/baz/
+    security_token: "{{ security_token }}"
+    aws_access_key: "{{ ec2_access_key }}"
+    aws_secret_key: "{{ ec2_secret_key }}"
+  retries: 3
+  delay: 3
+  register: result
+- name: assert that empty path is created
+  assert:
+    that:
+      - "'Virtual directory foo/bar/baz/ created' in result.msg"
+      - result.changed == True
+- name: test deletion of empty path
+  aws_s3:
+    bucket: "{{ bucket_name }}"
+    mode: delobj
+    object: foo/bar/baz/
+    security_token: "{{ security_token }}"
+    aws_access_key: "{{ ec2_access_key }}"
+    aws_secret_key: "{{ ec2_secret_key }}"
+  retries: 3
+  delay: 3
 # ============================================================
 - name: test delete bucket
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name }}"
     mode: delete
     security_token: "{{security_token}}"
@@ -182,7 +214,7 @@
     path: "{{ tmp2.path }}"
 # ============================================================
 - name: test create a bucket with a dot in the name
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name + '.bucket' }}"
     mode: create
     security_token: "{{security_token}}"
@@ -195,7 +227,7 @@
       - result.changed == True
 # ============================================================
 - name: test delete a bucket with a dot in the name
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name + '.bucket' }}"
     mode: delete
     security_token: "{{security_token}}"
@@ -208,7 +240,7 @@
       - result.changed == True
 # ============================================================
 - name: test delete a nonexistent bucket
-  s3:
+  aws_s3:
     bucket: "{{ bucket_name + '.bucket' }}"
     mode: delete
     security_token: "{{security_token}}"