Repository: ansible/ansible

Issue:
Cannot download S3 file that was uploaded via multipart upload
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
aws s3

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.3.0.0
  config file = /home/tyler/git/sync/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]
```

##### CONFIGURATION
Using ansible.cfg

```
[defaults]
inventory         = ansible/inventory
remote_tmp        = $HOME/.ansible/tmp
pattern           = *
forks             = 10
poll_interval     = 15
host_key_checking = False
remote_user       = csats
sudo_user         = root
# SSH timeout
timeout           = 20
retry_files_enabled = False
```

##### OS / ENVIRONMENT
Ubuntu 14.04

##### SUMMARY
As of Ansible version 2.3.0.0, when trying to download a tar .tgz file from s3, I am told: `Files uploaded with multipart of s3 are not supported with checksum, unable to compute checksum.` This does not happen with ansible v 2.1.0.0

Interestingly, the s3 module succeeds when trying to download a much smaller file that was not uploaded using multi part uploads.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
local_action: 's3 aws_access_key={{ aws_access_key }} aws_secret_key={{ aws_secret_key }} bucket=csync.csats.com object={{ s3_archive }} region=us-west-2 mode=get dest={{ vidcap_archive }}'
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
The file gets downloaded 

##### ACTUAL RESULTS
Error thrown

<!--- Paste verbatim command output between quotes below -->
```
"Files uploaded with multipart of s3 are not supported with checksum, unable to compute checksum."
```

Using -vvv

```
fatal: [CCAP1 -> localhost]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "aws_access_key": "something", 
            "aws_secret_key": "something", 
            "bucket": "csync.csats.com", 
            "dest": "something.tgz", 
            "ec2_url": null, 
            "encrypt": true, 
            "expiry": "600", 
            "headers": null, 
            "ignore_nonexistent_bucket": false, 
            "marker": null, 
            "max_keys": "1000", 
            "metadata": null, 
            "mode": "get", 
            "object": "something.tgz", 
            "overwrite": "always", 
            "permission": [
                "private"
            ], 
            "prefix": null, 
            "profile": null, 
            "region": "us-west-2", 
            "retries": 0, 
            "rgw": false, 
            "s3_url": null, 
            "security_token": null, 
            "src": null, 
            "validate_certs": true, 
            "version": null
        }
    }, 
    "msg": "Files uploaded with multipart of s3 are not supported with checksum, unable to compute checksum."
}
```



Discussion:
+1
+1
+1
+1
+1
I can "+1" this also, but to add a bit to this:
* https://github.com/ansible/ansible/issues/5442
* https://github.com/ansible/ansible/issues/12646

Not the first time this has been seen and it apparently is re-appearing in 2.3.0.0 after some time of being ... fixed? (That's unclear to me from the above).
+1
+1
+1
seeing this same behavior on 2.0.2.0, but only when using the `overwrite: different` option
```shell
$ ansible --version
ansible 2.0.2.0
  config file = /Users/mconlon/git/hyperwave_relay/ansible.cfg
  configured module search path = custom_modules
```
This is baffling. What business does the module have trying to compute a checksum even when the `overwrite: never` module parameter is set? It's obviously doing it anyway, and worse, it's failing hard when it can't compute the checksum it doesn't need to compute -- instead of doing the idempotent thing and just unconditionally leaving the locally present file alone.
Disappointing issue. I'm looking to switch from Chef to Ansible and this is one in a series of things blocking me. If I wanted to use aws cli shell commands to do things I wouldn't need a CM tool. 
The problem is this: https://github.com/ansible/ansible/issues/5442#issue-24829689,
Or more directly, this: https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/cloud/amazon/s3.py#L308,
Likely because @jimi-c said this: https://groups.google.com/d/msg/ansible-project/Dg7aAmpoS2A/iWkJ6Yp2L3YJ
Hence this issue is being ignored.

...respectfully, I disagree with the claim that the s3 module, quote, must "remain dumb" if the operator can _tell_ the module how to resolve the error that occurs when the local file is already present and a checksum comparison is impossible. The module does already have a parameter, `overwrite`, that can (should?) be used for this purpose. In that branch simply do this (incoming pseudocode!):

```pseudocode
if multipart, then
  case overwrite=always|true) download the remote file, clobber the local file, ignore checksum
  case overwrite=never|false) do not download the remote file, preserve the local file, ignore checksum
  case overwrite=different|is not defined) error as before, maybe hint at defining overwrite
```

As the module, in all cases of the multipart branch, you ignore the checksum. You just throw it out, and either fetch the bits or don't, as you're told. If and only if you're not told which to do, _then_ you play dumb.
Digged a little further into this issue, here are my findings:

Compared s3.py between two releases v2.3.1.0-1 and v2.2.3.0-1
In 2.3:
https://github.com/ansible/ansible/blob/ecb38fdf73a0beb89e9505157a2ea20c96f666d7/lib/ansible/modules/cloud/amazon/s3.py#L567
In 2.2:
https://github.com/ansible/ansible-modules-core/blob/ebdd66c2b6cb15e2364b2ecc41a9da9c1d02d64f/cloud/amazon/s3.py#L517

somehow during the merge from ansible-core to ansible, below code has been removed
```python
if pathrtn is False or overwrite == 'always':
    download_s3file(module, s3, bucket, obj, dest, retries, version=version)
```
which says when ```overwrite == 'always'```(default), download the file without checksum.
I think this explains why this issue re-appearing in 2.3 @Illydth, and this code itself shows why we see this issue when ```overwrite``` was set to other values @conlon @nethershaw 

In terms of why error was thrown for checksum:
```python
etag_multipart = '-' in md5_remote # Check for multipart, etag is not md5
```
In my case, I use aws s3 cp to upload my small file to s3, but ```cp``` is using multipart upload by default and no ```disable-multipart``` option atm, so my s3 file key always with ```-``` in it. one work around is https://github.com/aws/aws-cli/issues/946
  
Also hitting this issue on Ansible 2.3.1.0.  One way to get around this is by stat-ing the file you want to overwrite and removing it if it exists before doing the S3 Get.  This isn't idempotent, but at least keeps your sanity in check.
Still hitting this issue 2.3.2.0
cc @lwade @willthames
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
+1
@s-hertel do we have any tests for this in the new aws_s3 test suite?

Resolving patch:
diff --git a/lib/ansible/modules/cloud/amazon/aws_s3.py b/lib/ansible/modules/cloud/amazon/aws_s3.py
--- a/lib/ansible/modules/cloud/amazon/aws_s3.py
+++ b/lib/ansible/modules/cloud/amazon/aws_s3.py
@@ -271,8 +271,9 @@
   - prefix1/key2
 '''
 
-import os
+import hashlib
 import mimetypes
+import os
 import traceback
 from ansible.module_utils.six.moves.urllib.parse import urlparse
 from ssl import SSLError
@@ -306,7 +307,34 @@ def key_check(module, s3, bucket, obj, version=None, validate=True):
     return exists
 
 
-def keysum(module, s3, bucket, obj, version=None):
+def keysum_compare(module, local_file, s3, bucket, obj, version=None):
+    s3_keysum = keysum(s3, bucket, obj, version=version)
+    if '-' in s3_keysum:  # Check for multipart, ETag is not a proper MD5 sum
+        parts = int(s3_keysum.split('-')[1])
+        md5s = []
+
+        with open(local_file, 'rb') as f:
+            for part_num in range(1, parts + 1):
+                # Get the part size for every part of the multipart uploaded object
+                if version:
+                    key_head = s3.head_object(Bucket=bucket, Key=obj, VersionId=version, PartNumber=part_num)
+                else:
+                    key_head = s3.head_object(Bucket=bucket, Key=obj, PartNumber=part_num)
+                part_size = int(key_head['ContentLength'])
+                data = f.read(part_size)
+                hash = hashlib.md5(data)
+                md5s.append(hash)
+
+        digests = b''.join(m.digest() for m in md5s)
+        digests_md5 = hashlib.md5(digests)
+        local_keysum = '{0}-{1}'.format(digests_md5.hexdigest(), len(md5s))
+    else:  # Compute the MD5 sum normally
+        local_keysum = module.md5(local_file)
+
+    return s3_keysum == local_keysum
+
+
+def keysum(s3, bucket, obj, version=None):
     if version:
         key_check = s3.head_object(Bucket=bucket, Key=obj, VersionId=version)
     else:
@@ -314,8 +342,6 @@ def keysum(module, s3, bucket, obj, version=None):
     if not key_check:
         return None
     md5_remote = key_check['ETag'][1:-1]
-    if '-' in md5_remote:  # Check for multipart, etag is not md5
-        return None
     return md5_remote
 
 
@@ -706,11 +732,11 @@ def main():
             else:
                 module.fail_json(msg="Key %s does not exist." % obj)
 
-        # If the destination path doesn't exist or overwrite is True, no need to do the md5um etag check, so just download.
+        # If the destination path doesn't exist or overwrite is True, no need to do the md5sum ETag check, so just download.
         # Compare the remote MD5 sum of the object with the local dest md5sum, if it already exists.
         if path_check(dest):
             # Determine if the remote and local object are identical
-            if keysum(module, s3, bucket, obj, version=version) == module.md5(dest):
+            if keysum_compare(module, dest, s3, bucket, obj, version=version):
                 sum_matches = True
                 if overwrite == 'always':
                     download_s3file(module, s3, bucket, obj, dest, retries, version=version)
@@ -740,10 +766,10 @@ def main():
         if bucketrtn:
             keyrtn = key_check(module, s3, bucket, obj, version=version, validate=validate)
 
-        # Lets check key state. Does it exist and if it does, compute the etag md5sum.
+        # Lets check key state. Does it exist and if it does, compute the ETag md5sum.
         if bucketrtn and keyrtn:
             # Compare the local and remote object
-            if module.md5(src) == keysum(module, s3, bucket, obj):
+            if keysum_compare(module, src, s3, bucket, obj):
                 sum_matches = True
                 if overwrite == 'always':
                     # only use valid object acls for the upload_s3file function


Test patch: