Repository: ansible/ansible-modules-core

Issue:
sha256sum in get_url doesn't check hash of files that already exist
If a sha256sum is specified, I would assume that the file's digest will be checked even if the file already exists on the server prior to downloading. At present, get_url makes no attempt to check if the digest of the pre-existing file matches the provided digest or not and will only re-download if force is specified.

Using force=yes isn't really an acceptable workaround because force=yes will cause extra network traffic in every ansible run, when we should only need to go to the network if the expected sha256sum differs from the current sha256sum.

This is a migration of ansible/ansible#9097.



Discussion:
# Can You Help Us Out?

Thanks for filing a ticket! I am the friendly GitHub Ansibot. 

It looks like you might not have filled out the issue description based on our standard issue template.  You might not have known about that, and that's ok too, we'll tell you how to do it.

We have a standard template because Ansible is a really busy project and it helps to have some standard information in each ticket, and GitHub doesn't yet provide a standard facility to do this like some other bug trackers.  We hope you understand as this is really valuable to us!. 

Solving this is simple: please copy the contents of this [template](https://raw.githubusercontent.com/ansible/ansible/devel/ISSUE_TEMPLATE.md) and **paste it into the description** of your ticket.  That's it!

# If You Had A Question To Ask Instead

If you happened to have a "how do I do this in Ansible" type of question, that's probably more of a user-list question than a bug report, and you should probably ask this question on the project mailing list instead.  

However, if you think you have a bug, the report is the way to go!  We definitely want all the bugs filed :)  Just trying to help!

# About Priority Tags

Since you're here, we'll also share some useful information at this time.

In general tickets will be assigned a priority between P1 (highest) and P5, and then worked in priority order. We may also have some follow up questions along the way, so keeping up with follow up comments via GitHub notifications is a good idea.

Due to large interest in Ansible, humans may not comment on your ticket immediately.  

# Mailing Lists

If you have concerns or questions, you're welcome to stop by the ansible-project or ansible-development mailing lists, as appropriate.  Here are the links:
- https://groups.google.com/forum/#!forum/ansible-project - for discussion of bugs and how-to type questions
- https://groups.google.com/forum/#!forum/ansible-devel - for discussion on how to implement a code change, or feature brainstorming among developers

Thanks again for the interest in Ansible!

##### Issue Type:

Bug Report

##### Ansible Version:

ansible 1.7.1

##### Environment:

ubuntu 12.04

##### Summary:

sha256sum in get_url doesn't check the hash of files that already exist.  So if you specify the sha256sum of a file, and the file already exists but has a mismatching sha256sum, ansible won't download a new version.

##### Steps To Reproduce:

For an example, see the resource in al3x/sovereign#340 -- in that PR, I wanted to update the file `/opt/check-rbl.pl` to a new version, without using `force=yes` to force a redownload on every ansible run.  I expected that if I provided the sha256sum, ansible would download a new version if the existing file didn't match the sha256sum.  However, ansible does not do this -- if the file already exists, and `force=no`, ansible won't change it, even if it has the wrong sha256sum.

This can be trivially reproduced by first running `touch /opt/check-rbl.pl` to create the file, then running the ansible resource from the above PR (quoted below for convenience) to see if ansible will update it when it detects the sha256sum mismatch.  I expect ansible to download and replace the file, but instead ansible leaves the original file in place.

``` yaml
- name: Download check-rbl
  get_url: url=https://raw.github.com/lukecyca/check-rbl/38006028bc1865e9c/check-rbl.pl dest=/opt/check-rbl.pl sha256sum=59a8e15ee702e8c056426ebdd155fdd7a651716f40c58b7a4fef2a1c312f07da
```

To work around this problem, I manually deleted the file, and reran ansible.  It seems that sha256sum is only used to validate the download, not to decide whether or not to download in the first place.


Resolving patch:
diff --git a/network/basics/get_url.py b/network/basics/get_url.py
--- a/network/basics/get_url.py
+++ b/network/basics/get_url.py
@@ -217,8 +217,29 @@ def main():
     dest_is_dir = os.path.isdir(dest)
     last_mod_time = None
 
+    # Remove any non-alphanumeric characters, including the infamous
+    # Unicode zero-width space
+    stripped_sha256sum = re.sub(r'\W+', '', sha256sum)
+
+    # Fail early if sha256 is not supported
+    if sha256sum != '' and not HAS_HASHLIB:
+        module.fail_json(msg="The sha256sum parameter requires hashlib, which is available in Python 2.5 and higher")
+
     if not dest_is_dir and os.path.exists(dest):
-        if not force:
+        checksum_mismatch = False
+
+        # If the download is not forced and there is a checksum, allow
+        # checksum match to skip the download.
+        if not force and sha256sum != '':
+            destination_checksum = module.sha256(dest)
+
+            if stripped_sha256sum.lower() == destination_checksum:
+                module.exit_json(msg="file already exists", dest=dest, url=url, changed=False)
+
+            checksum_mismatch = True
+
+        # Not forcing redownload, unless sha256sum has already failed
+        if not force and not checksum_mismatch:
             module.exit_json(msg="file already exists", dest=dest, url=url, changed=False)
 
         # If the file already exists, prepare the last modified time for the
@@ -281,15 +302,7 @@ def main():
     # Check the digest of the destination file and ensure that it matches the
     # sha256sum parameter if it is present
     if sha256sum != '':
-        # Remove any non-alphanumeric characters, including the infamous
-        # Unicode zero-width space
-        stripped_sha256sum = re.sub(r'\W+', '', sha256sum)
-
-        if not HAS_HASHLIB:
-            os.remove(dest)
-            module.fail_json(msg="The sha256sum parameter requires hashlib, which is available in Python 2.5 and higher")
-        else:
-            destination_checksum = module.sha256(dest)
+        destination_checksum = module.sha256(dest)
 
         if stripped_sha256sum.lower() != destination_checksum:
             os.remove(dest)


Test patch: