Repository: ansible/ansible

Issue:
Failed Uploads to S3 Bucket
From @mechcozmo on 2014-10-22T14:55:54Z
##### Issue Type: 
Bug Report

##### Component Name
s3 module

##### Ansible Version: 
1.7.2

```
$ansible --version
ansible 1.7.2
```

Installed from Homebrew
##### Environment:

I am running Ansible from Mac OS X 10.9.5, targeting EC2 instances running Amazon Linux (latest release, 2014.09)
##### Summary:

Ansible fails to upload a file on the managed host to an S3 bucket.

(Moved from https://github.com/ansible/ansible/issues/9385 by request)

Ansible is failing to upload a file to an S3 bucket.  AWS credentials are specified in the command, and the file it is attempting to upload is visible in the CLI.

```
failed: [mcap2.censoredhostname.com] => {"failed": true, "parsed": false}
invalid output was: Traceback (most recent call last):
  File "/home/ec2-user/.ansible/tmp/ansible-tmp-1413820175.16-111146613354561/s3", line 2031, in <module>
    main()
  File "/home/ec2-user/.ansible/tmp/ansible-tmp-1413820175.16-111146613354561/s3", line 390, in main
    pathrtn = path_check(src)
  File "/home/ec2-user/.ansible/tmp/ansible-tmp-1413820175.16-111146613354561/s3", line 211, in path_check
    if os.path.exists(path):
  File "/usr/lib64/python2.6/genericpath.py", line 18, in exists
    st = os.stat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found`
```
##### Steps To Reproduce:

I am using Ansible to capture the memory of EC2 instances and upload them to S3 with a series of plays.  The first two plays run without errors (provisioning instances, configuring instances).  The last play contains two tasks: 1) Run LiME to capture the memory image and 2) Upload the resulting file to S3

The failure occurs with the second task, when attempting to upload the file to S3.  While LiME must be run as root, the S3 upload does not have to be.  I have tried both ways of uploading content, and both end in failure.  The only difference is that when `sudo: true` is specified for the entire playbook, instead of only at the LiME task level, the failure output includes `invalid output was: SUDO-SUCCESS-hsuyekhqxevdebwqiylmgyxgixbozcwz` on a line before the traceback.

Here is the play being run:

```
- hosts: memcaphosts
  user: ec2-user

  tasks:
    - name: Run LiME
    command: /usr/sbin/insmod /home/ec2-user/bin/lime-3.14.20-20.44.amzn1.x86_64.ko "path=/home/ec2-user/{{ inventory_hostname_short }}.lime format=lime"
    sudo: true

  - name: Upload memory capture to S3
    s3: bucket=memory-captures object=/home/ec2-user/{{ inventory_hostname_short }}.lime mode=put aws_access_key='[[[CENSORED]]]' aws_secret_key='[[[CENSORED]]]'
```
##### Expected Results:

I expect the S3 bucket 'memory-captures' to contain the memory images.  Each memory capture (for this instance size) is 4GB.
##### Actual Results:

What actually happens is Ansible crashes, leaving the files on-disk and unuploaded.

Copied from original issue: ansible/ansible-modules-core#214

Failed Uploads to S3 Bucket
From @mechcozmo on 2014-10-22T14:55:54Z
##### Issue Type: 
Bug Report

##### Component Name
s3 module

##### Ansible Version: 
1.7.2

```
$ansible --version
ansible 1.7.2
```

Installed from Homebrew
##### Environment:

I am running Ansible from Mac OS X 10.9.5, targeting EC2 instances running Amazon Linux (latest release, 2014.09)
##### Summary:

Ansible fails to upload a file on the managed host to an S3 bucket.

(Moved from https://github.com/ansible/ansible/issues/9385 by request)

Ansible is failing to upload a file to an S3 bucket.  AWS credentials are specified in the command, and the file it is attempting to upload is visible in the CLI.

```
failed: [mcap2.censoredhostname.com] => {"failed": true, "parsed": false}
invalid output was: Traceback (most recent call last):
  File "/home/ec2-user/.ansible/tmp/ansible-tmp-1413820175.16-111146613354561/s3", line 2031, in <module>
    main()
  File "/home/ec2-user/.ansible/tmp/ansible-tmp-1413820175.16-111146613354561/s3", line 390, in main
    pathrtn = path_check(src)
  File "/home/ec2-user/.ansible/tmp/ansible-tmp-1413820175.16-111146613354561/s3", line 211, in path_check
    if os.path.exists(path):
  File "/usr/lib64/python2.6/genericpath.py", line 18, in exists
    st = os.stat(path)
TypeError: coercing to Unicode: need string or buffer, NoneType found`
```
##### Steps To Reproduce:

I am using Ansible to capture the memory of EC2 instances and upload them to S3 with a series of plays.  The first two plays run without errors (provisioning instances, configuring instances).  The last play contains two tasks: 1) Run LiME to capture the memory image and 2) Upload the resulting file to S3

The failure occurs with the second task, when attempting to upload the file to S3.  While LiME must be run as root, the S3 upload does not have to be.  I have tried both ways of uploading content, and both end in failure.  The only difference is that when `sudo: true` is specified for the entire playbook, instead of only at the LiME task level, the failure output includes `invalid output was: SUDO-SUCCESS-hsuyekhqxevdebwqiylmgyxgixbozcwz` on a line before the traceback.

Here is the play being run:

```
- hosts: memcaphosts
  user: ec2-user

  tasks:
    - name: Run LiME
    command: /usr/sbin/insmod /home/ec2-user/bin/lime-3.14.20-20.44.amzn1.x86_64.ko "path=/home/ec2-user/{{ inventory_hostname_short }}.lime format=lime"
    sudo: true

  - name: Upload memory capture to S3
    s3: bucket=memory-captures object=/home/ec2-user/{{ inventory_hostname_short }}.lime mode=put aws_access_key='[[[CENSORED]]]' aws_secret_key='[[[CENSORED]]]'
```
##### Expected Results:

I expect the S3 bucket 'memory-captures' to contain the memory images.  Each memory capture (for this instance size) is 4GB.
##### Actual Results:

What actually happens is Ansible crashes, leaving the files on-disk and unuploaded.

Copied from original issue: ansible/ansible-modules-core#214



Discussion:
From @jimi-c on 2014-10-22T14:55:54Z

Hi @mechcozmo, at first glance it appears that you have a quoting typo in your example. You have the quotes starting like this:

<pre>
"path=/home/ec2-user/{{ inventory_hostname_short }}.lime format=lime"
</pre>

When it should be like this:

<pre>
path="/home/ec2-user/{{ inventory_hostname_short }}.lime" format=lime
</pre>

The traceback above is indicating it the path param was not set, so I believe fixing your task as above should get you past this error. Regardless, the module should handle this more gracefully so we will keep this issue open.

Thanks!

From @mechcozmo on 2014-10-22T14:55:54Z

Hi @jimi-c.  I'm attempting to pass the parameter to LiME here.  The quotes are the means to pass that chunk of text to the `lime-3.14.20-20.44.amzn1.x86_64.ko` executable.  If I copy/paste that command directly into a CLI window (EDIT: and change the variable out for something sensible), it will execute properly.  The `path=` param isn't for Ansible, it's for LiME.

You can see an example of the command being executed here: https://code.google.com/p/volatility/wiki/LinuxMemoryForensics#Acquiring_memory

From @jimi-c on 2014-10-22T14:55:54Z

Ahh I see, I got the commands crossed up when scrolling horizontally. 

However, it's basically the same root cause: you must specify the `src=` for the source file, the `object=` param is for the name inside the bucket (unless you're keeping the path the same intentionally).

From @mechcozmo on 2014-10-22T14:55:54Z

OK.  I didn't read the docs correctly, then.  That's a very unfortunate and unfriendly way of saying "You didn't give me the parameters I was expecting!"  This ticket can probably be closed, as uploads work now.  Is there a place I should log an improvement request for a better error message?

From @jimi-c on 2014-10-22T14:55:54Z

@mechcozmo we'll hold this issue open for that. We consider all tracebacks to be a bug, so this should definitely be handled better.

cc @lwade
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
cc @s-hertel
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
From @jimi-c on 2014-10-22T14:55:54Z

Hi @mechcozmo, at first glance it appears that you have a quoting typo in your example. You have the quotes starting like this:

<pre>
"path=/home/ec2-user/{{ inventory_hostname_short }}.lime format=lime"
</pre>

When it should be like this:

<pre>
path="/home/ec2-user/{{ inventory_hostname_short }}.lime" format=lime
</pre>

The traceback above is indicating it the path param was not set, so I believe fixing your task as above should get you past this error. Regardless, the module should handle this more gracefully so we will keep this issue open.

Thanks!

From @mechcozmo on 2014-10-22T14:55:54Z

Hi @jimi-c.  I'm attempting to pass the parameter to LiME here.  The quotes are the means to pass that chunk of text to the `lime-3.14.20-20.44.amzn1.x86_64.ko` executable.  If I copy/paste that command directly into a CLI window (EDIT: and change the variable out for something sensible), it will execute properly.  The `path=` param isn't for Ansible, it's for LiME.

You can see an example of the command being executed here: https://code.google.com/p/volatility/wiki/LinuxMemoryForensics#Acquiring_memory

From @jimi-c on 2014-10-22T14:55:54Z

Ahh I see, I got the commands crossed up when scrolling horizontally. 

However, it's basically the same root cause: you must specify the `src=` for the source file, the `object=` param is for the name inside the bucket (unless you're keeping the path the same intentionally).

From @mechcozmo on 2014-10-22T14:55:54Z

OK.  I didn't read the docs correctly, then.  That's a very unfortunate and unfriendly way of saying "You didn't give me the parameters I was expecting!"  This ticket can probably be closed, as uploads work now.  Is there a place I should log an improvement request for a better error message?

From @jimi-c on 2014-10-22T14:55:54Z

@mechcozmo we'll hold this issue open for that. We consider all tracebacks to be a bug, so this should definitely be handled better.

cc @lwade
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
cc @s-hertel
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/aws_s3.py b/lib/ansible/modules/cloud/amazon/aws_s3.py
--- a/lib/ansible/modules/cloud/amazon/aws_s3.py
+++ b/lib/ansible/modules/cloud/amazon/aws_s3.py
@@ -598,6 +598,7 @@ def main():
     module = AnsibleModule(
         argument_spec=argument_spec,
         supports_check_mode=True,
+        required_if=[('mode', 'put', ('src',))],
     )
 
     if module._name == 's3':


Test patch: