Repository: ansible/ansible

Issue:
maven_artifact should download artifact before hash
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
maven_artifact

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.0.0
  config file = /redacted/ansible.cfg
  configured module search path = [u'/Users/turb/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.4.0.0/libexec/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.14 (default, Sep 25 2017, 09:53:22) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)]
```

##### CONFIGURATION
<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"

Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).

-->
```
ANSIBLE_NOCOWS(/redacted/ansible.cfg) = False
ANSIBLE_SSH_PIPELINING(/redacted/ansible.cfg) = False
DEFAULT_HOST_LIST(/redacted/ansible.cfg) = [u'/redacted/hosts']
```

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->

- From: MacOS X High Sierra
- To: Debian mixed 8.2 and 8.9

##### SUMMARY
<!--- Explain the problem briefly -->

As far as I can observe, `maven_artifact` downloads the MD5 file, *then* downloads the actual artifact.

The python code seems consistent with this (comments are mine):
```python
    def download(self, artifact, filename=None):
        filename = artifact.get_filename(filename)
        if not artifact.version or artifact.version == "latest":
            artifact = Artifact(artifact.group_id, artifact.artifact_id, self.find_latest_version_available(artifact),
                                artifact.classifier, artifact.extension)

        url = self.find_uri_for_artifact(artifact)
        result = True
        if not self.verify_md5(filename, url + ".md5"): # checks MD5 against already downloaded file, if it exists
            response = self._request(url, "Failed to download artifact " + str(artifact), lambda r: r) # if no match, download it (for the first time or again)
            if response:
                f = open(filename, 'w')
                # f.write(response.read())
                self._write_chunks(response, f, report_hook=self.chunk_report)
                f.close()
            else:
                result = False
        return result
```

ansible/lib/ansible/modules/packaging/language/maven_artifact.py line 339

It is an issue for two reasons:
- the hash is checked only if the file is *already* present, and will not check a newly downloaded file
- some maven repositories (Artifactory 4.15.0 for us), when configured as a proxy/cache, returns 404 on http://artifactory/path/artifact.jar.md5 when http://artifactory/path/artifact.jar has not been loaded yet in the cache.

I can provide a PR if needed.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
-  maven_artifact:
    group_id: asm
    artifact_id: asm
    version: 3.2
    repository_url: http://artifactory.url/maven
    dest: /tmp/theartifact.jar
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS

- OK when the downloaded artifact matches the MD5 file, even if not loaded yet in the Artifactory cache
- KO when the newly downloaded artifact does not match the MD5 file

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
failed: [myhost] (item={u'version': 3.2, u'group': u'asm', u'name': u'asm'}) => {"changed": false, "failed": true, "item": {"group": "asm", "name": "asm", "version": 3.2}, "module_stderr": "Shared connection to myhost closed.
", "module_stdout": "Traceback (most recent call last):
  File \"/tmp/ansible_bQHxvE/ansible_module_maven_artifact.py\", line 499, in <module>
    main()
  File \"/tmp/ansible_bQHxvE/ansible_module_maven_artifact.py\", line 476, in main
    if os.path.lexists(b_dest) and downloader.verify_md5(dest, downloader.find_uri_for_artifact(artifact) + '.md5'):
  File \"/tmp/ansible_bQHxvE/ansible_module_maven_artifact.py\", line 390, in verify_md5\
    remote = self._request(remote_md5, \"Failed to download MD5\", lambda r: r.read())
  File \"/tmp/ansible_bQHxvE/ansible_module_maven_artifact.py\", line 333, in _request
    raise ValueError(failmsg + \" because of \" + info['msg'] + \"for URL \" + url_to_use)
ValueError: Failed to download MD5 because of HTTP Error 404: Not Foundfor URL http://artifactory.url/maven/asm/asm/3.2/asm-3.2.jar.md5
", "msg": "MODULE FAILURE", "rc": 0}

```

Workaround: manually do a GET on http://artifactory.url/maven/asm/asm/3.2/asm-3.2.jar and start again 



Discussion:
cc @chrisisbeef @tumbl3w33d
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Looking forward to your PR. You are right, the current implementation only uses the hash to check whether the correct artifact is already present. An initial integrity check was apparently not the intention but it should be done.

Not sure if I'm overlooking something but it seems to me the current implementation checks the md5 twice if the file already exists. Once in the `download(...)` and once in the `main()`:
``` python
    if os.path.lexists(b_dest) and downloader.verify_md5(dest, downloader.find_uri_for_artifact(artifact) + '.md5'):
        prev_state = "present"
```
Seems redundant to me. `download(...)` only gets called when the file doesn't exist or its integrity isn't right so in what constellation would it make sense to check there again?
@tumbl3w33d I think this md5 is only intended for integrity checks (and is not always provided, AFAIK). For a repository, it is be a very bad practice to change the content of a non-snapshot artifact without changing its version; even for snapshot artifacts, if the version does not change, the final filename will be different.

In addition to the 404 error I get, request the repository on each run for each maven_artifact item unnecessary 1) slows the run 2) loads the repository, where (I think) it should be a no-op if the file exists.

I would say *either*:
- create a parameter on how to resolve if the file should change, with "check the md5" or a "check if the file exists" (for those like us who store the version in the name)
- *or* (somewhat better but I don't know how it would be possible) store somewhere the current artifact-name-version information, and evaluate accordingly
Note: Artifactory 5.5.1 returns a 404 too on md5 URL if the actual artifact has not been loaded yet in the cache.
Pull requested: #32745

I finally added a "force" option in the same fashion than "force" in the "template" module.
@tumbl3w33d do you think the PR can be merged?
I'm a little hesitant about the `force` default. Even if that's the current default (I didn't verify that yet), I'd rather see this set to `False`, because it seems a little counter-intuitive to me and I can't remember any other API where `force` defaults to `True`. The module is in state `preview` so it would be fine to change something about defaults. The new default would be more conservative anyway (and that's how it should be).

Yeah uhm... this is mostly what kept me from confirming right away. And the fact that I finally wanted to finalize the integration test suite that I started writing for this module. But I don't wanna be the bottleneck here. :) What do you think about inverting the `force` default? Did I overlook possible consequences?
Well, I am really confortable with setting `force` default to `False` for myself (it would prevent me to setting it everywhere), but I don't know if many people rely on the contrary.

However the `template` module does set `force` to `True` by default, see [its documentation](https://docs.ansible.com/ansible/latest/template_module.html).

To be bold, if the module is in `preview` state and backward compatibility can be broken, I would even remove the whole md5 thing (for reasons I explained above). Not all modules check for the content: for instance, the `unarchive` module does not check the target content: it only checks if the destination exist.
I'd be fine with making that md5 check optional (but checking by default seems a sane default, doesn't it?) by having some attribute like `verify_checksum` (or the like, in case some other module has that optional, we should try to stay consistent with the naming).
The md5 check may be performed two ways:

- check the file integrity on download
- compare the already existing file with the repository - what is currently done

I definitely agree on the first one. The PR and all my comments are about the second one.

A `verify_checksum` option could have four values:
- `never` (never verify it)
- `download` (verify it on download) <= should be the default IMHO
- something like `change` (verify if the destination file is different from the repository) <= what happens currently
- `always` (combines `download` and `change`)
That sounds like a plan. It reduces the voodoo under the hood and would be well documented. 👍 
Hi @turb,

I'd like to add that Sonatype Nexus OSS 3.2.1-01 returns a 404 on the md5 URL with the following conditions:

- Repository is not configured as a proxy
- The md5 checksum is present
- Artifact is locally cached.

Can you let me know if the PR is affected by these observations?
Hello @edwinyumakiuchi,

I don't know how Nexus deals with this kind of situation, not having used it for years. By all means, yes, the current PR may fix a 404 error in this scenario by allowing to remove the md5 check.

You can use it directly, however it will likely never be merged, as we just agreed on another option that will allow to fix it as well.

I'll try to request a new PR, but I can't plan when I will have the time to work on it.

Resolving patch:
diff --git a/lib/ansible/modules/packaging/language/maven_artifact.py b/lib/ansible/modules/packaging/language/maven_artifact.py
--- a/lib/ansible/modules/packaging/language/maven_artifact.py
+++ b/lib/ansible/modules/packaging/language/maven_artifact.py
@@ -7,16 +7,13 @@
 # as a reference and starting point.
 #
 # GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
-
 from __future__ import absolute_import, division, print_function
 __metaclass__ = type
 
-
 ANSIBLE_METADATA = {'metadata_version': '1.1',
                     'status': ['preview'],
                     'supported_by': 'community'}
 
-
 DOCUMENTATION = '''
 ---
 module: maven_artifact
@@ -105,6 +102,21 @@
         default: 'no'
         choices: ['yes', 'no']
         version_added: "2.4"
+    verify_checksum:
+        description:
+            - If C(never), the md5 checksum will never be downloaded and verified.
+            - If C(download), the md5 checksum will be downloaded and verified only after artifact download. This is the default.
+            - If C(change), the md5 checksum will be downloaded and verified if the destination already exist,
+              to verify if they are identical. This was the behaviour before 2.6. Since it downloads the md5 before (maybe)
+              downloading the artifact, and since some repository software, when acting as a proxy/cache, return a 404 error
+              if the artifact has not been cached yet, it may fail unexpectedly.
+              If you still need it, you should consider using C(always) instead - if you deal with a checksum, it is better to
+              use it to verify integrity after download.
+            - C(always) combines C(download) and C(change).
+        required: false
+        default: 'download'
+        choices: ['never', 'download', 'change', 'always']
+        version_added: "2.6"
 extends_documentation_fragment:
     - files
 '''
@@ -176,7 +188,6 @@ def split_pre_existing_dir(dirname):
     '''
     Return the first pre-existing directory and a list of the new directories that will be created.
     '''
-
     head, tail = os.path.split(dirname)
     b_head = to_bytes(head, errors='surrogate_or_strict')
     if not os.path.exists(b_head):
@@ -191,7 +202,6 @@ def adjust_recursive_directory_permissions(pre_existing_dir, new_directory_list,
     '''
     Walk the new directories list and make sure that permissions are as we would expect
     '''
-
     if new_directory_list:
         working_dir = os.path.join(pre_existing_dir, new_directory_list.pop(0))
         directory_args['path'] = working_dir
@@ -338,30 +348,36 @@ def _request(self, url, failmsg, f):
         else:
             return f(response)
 
-    def download(self, artifact, filename=None):
+    def download(self, artifact, verify_download, filename=None):
         filename = artifact.get_filename(filename)
         if not artifact.version or artifact.version == "latest":
             artifact = Artifact(artifact.group_id, artifact.artifact_id, self.find_latest_version_available(artifact),
                                 artifact.classifier, artifact.extension)
-
         url = self.find_uri_for_artifact(artifact)
-        result = True
-        if not self.verify_md5(filename, url + ".md5"):
-            response = self._request(url, "Failed to download artifact " + str(artifact), lambda r: r)
-            if response:
-                f = open(filename, 'wb')
+        error = None
+        response = self._request(url, "Failed to download artifact " + str(artifact), lambda r: r)
+        if response:
+            f = open(filename, 'wb')
+            self._write_chunks(response, f, report_hook=self.chunk_report)
+            f.close()
+            with open(filename, 'wb') as f:
                 self._write_chunks(response, f, report_hook=self.chunk_report)
-                f.close()
+
+            if verify_download and not self.verify_md5(filename, url + ".md5"):
+                # if verify_change was set, the previous file would be deleted
+                os.remove(filename)
+                error = "Checksum verification failed"
             else:
-                result = False
-        return result
+                error = None
+        else:
+            error = "Error downloading artifact " + str(artifact)
+        return error
 
     def chunk_report(self, bytes_so_far, chunk_size, total_size):
         percent = float(bytes_so_far) / total_size
         percent = round(percent * 100, 2)
         sys.stdout.write("Downloaded %d of %d bytes (%0.2f%%)\r" %
                          (bytes_so_far, total_size, percent))
-
         if bytes_so_far >= total_size:
             sys.stdout.write('\n')
 
@@ -401,7 +417,6 @@ def _local_md5(self, file):
 
 
 def main():
-
     module = AnsibleModule(
         argument_spec=dict(
             group_id=dict(default=None),
@@ -417,6 +432,7 @@ def main():
             dest=dict(type="path", default=None),
             validate_certs=dict(required=False, default=True, type='bool'),
             keep_name=dict(required=False, default=False, type='bool'),
+            verify_checksum=dict(required=False, default='download', choices=['never', 'download', 'change', 'always']),
         ),
         add_file_common_args=True
     )
@@ -445,8 +461,10 @@ def main():
     dest = module.params["dest"]
     b_dest = to_bytes(dest, errors='surrogate_or_strict')
     keep_name = module.params["keep_name"]
+    verify_checksum = module.params["verify_checksum"]
+    verify_download = verify_checksum in ['download', 'always']
+    verify_change = verify_checksum in ['change', 'always']
 
-    # downloader = MavenDownloader(module, repository_url, repository_username, repository_password)
     downloader = MavenDownloader(module, repository_url)
 
     try:
@@ -481,15 +499,16 @@ def main():
             dest = posixpath.join(dest, "%s-%s.%s" % (artifact_id, version_part, extension))
         b_dest = to_bytes(dest, errors='surrogate_or_strict')
 
-    if os.path.lexists(b_dest) and downloader.verify_md5(dest, downloader.find_uri_for_artifact(artifact) + '.md5'):
+    if os.path.lexists(b_dest) and ((not verify_change) or downloader.verify_md5(dest, downloader.find_uri_for_artifact(artifact) + '.md5')):
         prev_state = "present"
 
     if prev_state == "absent":
         try:
-            if downloader.download(artifact, b_dest):
+            download_error = downloader.download(artifact, verify_download, b_dest)
+            if download_error is None:
                 changed = True
             else:
-                module.fail_json(msg="Unable to download the artifact")
+                module.fail_json(msg="Cannot download the artifact to destination: " + download_error)
         except ValueError as e:
             module.fail_json(msg=e.args[0])
 
@@ -502,6 +521,5 @@ def main():
     else:
         module.exit_json(state=state, dest=dest, changed=changed)
 
-
 if __name__ == '__main__':
     main()


Test patch: