Repository: ansible/ansible

Issue:
maven_artifact can't get artifact from local repo.
From @chchrist on 2016-07-11T11:55:05Z
<!--- Verify first that your issue/request is not already reported in GitHub -->
##### ISSUE TYPE

<!--- Pick one below and delete the rest: -->
- Bug Report
##### COMPONENT NAME

<!--- Name of the plugin/module/task -->

maven_artifact
##### ANSIBLE VERSION

<!--- Paste verbatim output from “ansible --version” between quotes below -->

```
ansible 2.0.2.0
```
##### CONFIGURATION

<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
##### OS / ENVIRONMENT

<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->

N/A
##### SUMMARY

<!--- Explain the problem briefly -->

maven_artifact can't get artifact from local repo as is looking for maven-metadata.xml and locally is maven-metadata-local.xml
##### STEPS TO REPRODUCE

<!---
For bugs, show exactly how to reproduce the problem.
For new features, show how the feature would be used.
-->

Just try to point to your local maven repository.

<!--- Paste example playbooks or commands between quotes below -->

```
- name: get artifact from local repo
  local_action: maven_artifact
  args:
    group_id: acme
    artifact_id: cms
    extension: zip
    classifier: foo
    version: "{{ artifact_version }}"
    repository_url: "~/.m2/repository"
    state: present
    dest: /tmp
  register: artifact
```

<!--- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS

<!--- What did you expect to happen when running the steps above? -->

It should be able to find the maven-metadata-local.xml
##### ACTUAL RESULTS

<!--- What actually happened? If possible run with high verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->

```
"msg": "unknown url type: ~/.m2/repository/acme/foo/cms/1.10-SNAPSHOT/maven-metadata.xml"}
```

Copied from original issue: ansible/ansible-modules-extras#2547



Discussion:
From @alikins on 2016-07-11T11:55:05Z

Is the behavior any different if repository_url is a file url? like:

```
- name: get artifact from local repo
  local_action: maven_artifact
  args:
    group_id: acme
    artifact_id: cms
    extension: zip
    classifier: foo
    version: "{{ artifact_version }}"
    repository_url: "file:///home/username/.m2/repository"
    state: present
    dest: /tmp
  register: artifact
```

From @ansibot on 2016-07-11T11:55:05Z

@chchrist ping, this issue is still waiting on your feedback. We will close the issue if you do not respond.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_closure --->

From @ansibot on 2016-07-11T11:55:05Z

@chchrist ping, this issue is still waiting on your feedback. We will close the issue if you do not respond.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_closure --->

From @ansibot on 2016-07-11T11:55:05Z

@chchrist ping, this issue is still waiting on your feedback. We will close the issue if you do not respond.
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)

<!--- boilerplate: issue_closure --->

From @chchrist on 2016-07-11T11:55:05Z

@alikins I haven' tried it. I will try and come back to you.

From @ktakashi on 2016-07-11T11:55:05Z

I've tried the `file://` prefix and got the error like this:

```
fatal: [machine]: FAILED! => {"changed": false, "failed": true, "msg": "Failed to download maven-metadata.xml because of Request failed: <urlopen error [Errno 2] No such file or directory: '/home/takashi/.m2/repository/.../maven-metadata.xml'>for URL file:///home/takashi/.m2/repository/.../maven-metadata.xml"}
```

In the repository directory there's `maven-metadata-local.xml` but no `maven-metadata.xml`

From @clabu on 2016-07-11T11:55:05Z

This was working using the `file://` prefix in Ansible 2.0.1.0 but it was broken by this change in `lib/ansible/module_utils/uris.py` introduced in Ansible 2.1.0.0: https://github.com/ansible/ansible/commit/ed35e8bce21180529e72f5bbe1d853b67ddac870

Previously, the `urls.fetch_url` method would always set the response's status code to `200`. Since the change, the actual status code in the response is used. For a "file://" URLs there is no (HTTP) status code in the response, so it gets set to value `NoneType` instead of `200`. The result is that the status code-check in `maven_artifact._request` thinks the URL request failed: https://github.com/ansible/ansible-modules-extras/blob/6c7d63b15c77126b4d6a8a7668545555578469c5/packaging/language/maven_artifact.py#L282

This patch to `maven_artifact.py` line 282 copes with the `NoneType` for `file://` URLs:
```
        if info['status'] == 200 or (info['status'] == None and parsed_url.scheme == 'file'):
            return f(response)
        else:
            raise ValueError(failmsg + " because of " + info['msg'] + "for URL " + url_to_use)
```
If you consider the fix robust enough, I will make a pull request.
From @petenorth on 2016-07-11T11:55:05Z

@clabu The patch to maven_artifact.py didn't resolve the issue for me.

The info['status'] seems to come back with -1, I tried using a value of -1 rather than None but to no avail.

Version of ansible was:

ansible 2.3.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Nov  6 2016, 00:28:07) [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

Error I got when I changed info['status'] == None to info['status'] == -1

failed: [node1.example.com] (item={u'version': u'1.1.0-SNAPSHOT', u'groupId': u'com.redhat.camel', u'artifactId': u'camel-amq-consumer'}) => {"failed": true, "item": {"artifactId": "camel-amq-consumer", "groupId": "com.redhat.camel", "version": "1.1.0-SNAPSHOT"}, "module_stderr": "Shared connection to node1.example.com closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_MMTYaR/ansible_module_maven_artifact.py\", line 424, in <module>\r\n    main()\r\n  File \"/tmp/ansible_MMTYaR/ansible_module_maven_artifact.py\", line 414, in main\r\n    if downloader.download(artifact, dest):\r\n  File \"/tmp/ansible_MMTYaR/ansible_module_maven_artifact.py\", line 294, in download\r\n    url = self.find_uri_for_artifact(artifact)\r\n  File \"/tmp/ansible_MMTYaR/ansible_module_maven_artifact.py\", line 244, in find_uri_for_artifact\r\n    xml = self._request(self.base + path, \"Failed to download maven-metadata.xml\", lambda r: etree.parse(r))\r\n  File \"/tmp/ansible_MMTYaR/ansible_module_maven_artifact.py\", line 283, in _request\r\n    return f(response)\r\n  File \"/tmp/ansible_MMTYaR/ansible_module_maven_artifact.py\", line 244, in <lambda>\r\n    xml = self._request(self.base + path, \"Failed to download maven-metadata.xml\", lambda r: etree.parse(r))\r\n  File \"lxml.etree.pyx\", line 3197, in lxml.etree.parse (src/lxml/lxml.etree.c:64816)\r\n  File \"parser.pxi\", line 1596, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:92968)\r\nTypeError: cannot parse from 'NoneType'\r\n", "msg": "MODULE FAILURE", "rc": 1}
cc @chrisisbeef @tumbl3w33d
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I get the following error with 2.4.0.0

fatal: [localhost -> localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Failed to download artifact com.example:myproject:0.0.525 because of OK (1329468 bytes)for URL file:///home/user/.m2/repository/com/example/myproject/0.0.525/myproject-0.0.525.jar"}



Resolving patch:
diff --git a/lib/ansible/modules/packaging/language/maven_artifact.py b/lib/ansible/modules/packaging/language/maven_artifact.py
--- a/lib/ansible/modules/packaging/language/maven_artifact.py
+++ b/lib/ansible/modules/packaging/language/maven_artifact.py
@@ -51,6 +51,7 @@
         description:
             - The URL of the Maven Repository to download from.
             - Use s3://... if the repository is hosted on Amazon S3, added in version 2.2.
+            - Use file://... if the repository is local, added in version 2.6
         default: http://repo1.maven.org/maven2
     username:
         description:
@@ -145,12 +146,21 @@
     group_id: org.springframework
     dest: /tmp/
     keep_name: yes
+
+# Download the latest version of the JUnit framework artifact from Maven local
+- maven_artifact:
+    group_id: junit
+    artifact_id: junit
+    dest: /tmp/junit-latest.jar
+    repository_url: "file://{{ lookup('env','HOME') }}/.m2/repository"
 '''
 
 import hashlib
 import os
 import posixpath
 import sys
+import shutil
+import io
 
 try:
     from lxml import etree
@@ -263,19 +273,22 @@ def parse(input):
 
 
 class MavenDownloader:
-    def __init__(self, module, base="http://repo1.maven.org/maven2"):
+    def __init__(self, module, base="http://repo1.maven.org/maven2", local=False):
         self.module = module
         if base.endswith("/"):
             base = base.rstrip("/")
         self.base = base
+        self.local = local
         self.user_agent = "Maven Artifact Downloader/1.0"
         self.latest_version_found = None
+        self.metadata_file_name = "maven-metadata-local.xml" if local else "maven-metadata.xml"
 
     def find_latest_version_available(self, artifact):
         if self.latest_version_found:
             return self.latest_version_found
-        path = "/%s/maven-metadata.xml" % (artifact.path(False))
-        xml = self._request(self.base + path, "Failed to download maven-metadata.xml", etree.parse)
+        path = "/%s/%s" % (artifact.path(False), self.metadata_file_name)
+        content = self._getContent(self.base + path, "Failed to retrieve the maven metadata file: " + path)
+        xml = etree.fromstring(content)
         v = xml.xpath("/metadata/versioning/versions/version[last()]/text()")
         if v:
             self.latest_version_found = v[0]
@@ -286,8 +299,11 @@ def find_uri_for_artifact(self, artifact):
             artifact.version = self.find_latest_version_available(artifact)
 
         if artifact.is_snapshot():
-            path = "/%s/maven-metadata.xml" % (artifact.path())
-            xml = self._request(self.base + path, "Failed to download maven-metadata.xml", etree.parse)
+            if self.local:
+                return self._uri_for_artifact(artifact, artifact.version)
+            path = "/%s/%s" % (artifact.path(), self.metadata_file_name)
+            content = self._getContent(self.base + path, "Failed to retrieve the maven metadata file: " + path)
+            xml = etree.fromstring(content)
             timestamp = xml.xpath("/metadata/versioning/snapshot/timestamp/text()")[0]
             buildNumber = xml.xpath("/metadata/versioning/snapshot/buildNumber/text()")[0]
             for snapshotArtifact in xml.xpath("/metadata/versioning/snapshotVersions/snapshotVersion"):
@@ -311,9 +327,26 @@ def _uri_for_artifact(self, artifact, version=None):
 
         return posixpath.join(self.base, artifact.path(), artifact.artifact_id + "-" + version + "." + artifact.extension)
 
-    def _request(self, url, failmsg, f):
+    # for small files, directly get the full content
+    def _getContent(self, url, failmsg, force=True):
+        if self.local:
+            parsed_url = urlparse(url)
+            if os.path.isfile(parsed_url.path):
+                with io.open(parsed_url.path, 'rb') as f:
+                    return f.read()
+            if force:
+                raise ValueError(failmsg + " because can not find file: " + url)
+            return None
+        response = self._request(url, failmsg, force)
+        if response:
+            return response.read()
+        return None
+
+    # only for HTTP request
+    def _request(self, url, failmsg, force=True):
         url_to_use = url
         parsed_url = urlparse(url)
+
         if parsed_url.scheme == 's3':
             parsed_url = urlparse(url)
             bucket_name = parsed_url.netloc
@@ -329,10 +362,11 @@ def _request(self, url, failmsg, f):
         self.module.params['http_agent'] = self.module.params.get('user_agent', None)
 
         response, info = fetch_url(self.module, url_to_use, timeout=req_timeout)
-        if info['status'] != 200:
+        if info['status'] == 200:
+            return response
+        if force:
             raise ValueError(failmsg + " because of " + info['msg'] + "for URL " + url_to_use)
-        else:
-            return f(response)
+        return None
 
     def download(self, artifact, verify_download, filename=None):
         filename = artifact.get_filename(filename)
@@ -340,24 +374,21 @@ def download(self, artifact, verify_download, filename=None):
             artifact = Artifact(artifact.group_id, artifact.artifact_id, self.find_latest_version_available(artifact),
                                 artifact.classifier, artifact.extension)
         url = self.find_uri_for_artifact(artifact)
-        error = None
-        response = self._request(url, "Failed to download artifact " + str(artifact), lambda r: r)
-        if response:
-            f = open(filename, 'wb')
-            self._write_chunks(response, f, report_hook=self.chunk_report)
-            f.close()
-            with open(filename, 'wb') as f:
-                self._write_chunks(response, f, report_hook=self.chunk_report)
-
-            if verify_download and not self.verify_md5(filename, url + ".md5"):
-                # if verify_change was set, the previous file would be deleted
-                os.remove(filename)
-                error = "Checksum verification failed"
+        if self.local:
+            parsed_url = urlparse(url)
+            if os.path.isfile(parsed_url.path):
+                shutil.copy2(parsed_url.path, filename)
             else:
-                error = None
+                return "Can not find local file: " + parsed_url.path
         else:
-            error = "Error downloading artifact " + str(artifact)
-        return error
+            response = self._request(url, "Failed to download artifact " + str(artifact))
+            with io.open(filename, 'wb') as f:
+                self._write_chunks(response, f, report_hook=self.chunk_report)
+        if verify_download and not self.verify_md5(filename, url):
+            # if verify_change was set, the previous file would be deleted
+            os.remove(filename)
+            return "Checksum verification failed"
+        return None
 
     def chunk_report(self, bytes_so_far, chunk_size, total_size):
         percent = float(bytes_so_far) / total_size
@@ -385,20 +416,22 @@ def _write_chunks(self, response, filehandle, chunk_size=8192, report_hook=None)
 
         return bytes_so_far
 
-    def verify_md5(self, file, remote_md5):
-        result = False
+    def verify_md5(self, file, remote_url):
         if os.path.exists(file):
             local_md5 = self._local_md5(file)
-            remote = self._request(remote_md5, "Failed to download MD5", lambda r: r.read())
-            result = local_md5 == remote
-        return result
+            if self.local:
+                parsed_url = urlparse(remote_url)
+                remote_md5 = self._local_md5(parsed_url.path)
+            else:
+                remote_md5 = self._getContent(remote_url + '.md5', "Failed to retrieve MD5", False)
+            return local_md5 == remote_md5
+        return False
 
     def _local_md5(self, file):
         md5 = hashlib.md5()
-        f = open(file, 'rb')
-        for chunk in iter(lambda: f.read(8192), ''):
-            md5.update(chunk)
-        f.close()
+        with io.open(file, 'rb') as f:
+            for chunk in iter(lambda: f.read(8192), ''):
+                md5.update(chunk)
         return md5.hexdigest()
 
 
@@ -418,7 +451,7 @@ def main():
             dest=dict(type="path", default=None),
             validate_certs=dict(required=False, default=True, type='bool'),
             keep_name=dict(required=False, default=False, type='bool'),
-            verify_checksum=dict(required=False, default='download', choices=['never', 'download', 'change', 'always']),
+            verify_checksum=dict(required=False, default='download', choices=['never', 'download', 'change', 'always'])
         ),
         add_file_common_args=True
     )
@@ -429,12 +462,13 @@ def main():
     repository_url = module.params["repository_url"]
     if not repository_url:
         repository_url = "http://repo1.maven.org/maven2"
-
     try:
         parsed_url = urlparse(repository_url)
     except AttributeError as e:
         module.fail_json(msg='url parsing went wrong %s' % e)
 
+    local = parsed_url.scheme == "file"
+
     if parsed_url.scheme == 's3' and not HAS_BOTO:
         module.fail_json(msg='boto3 required for this module, when using s3:// repository URLs')
 
@@ -451,7 +485,7 @@ def main():
     verify_download = verify_checksum in ['download', 'always']
     verify_change = verify_checksum in ['change', 'always']
 
-    downloader = MavenDownloader(module, repository_url)
+    downloader = MavenDownloader(module, repository_url, local)
 
     try:
         artifact = Artifact(group_id, artifact_id, version, classifier, extension)
@@ -485,7 +519,7 @@ def main():
             dest = posixpath.join(dest, "%s-%s.%s" % (artifact_id, version_part, extension))
         b_dest = to_bytes(dest, errors='surrogate_or_strict')
 
-    if os.path.lexists(b_dest) and ((not verify_change) or downloader.verify_md5(dest, downloader.find_uri_for_artifact(artifact) + '.md5')):
+    if os.path.lexists(b_dest) and ((not verify_change) or downloader.verify_md5(dest, downloader.find_uri_for_artifact(artifact))):
         prev_state = "present"
 
     if prev_state == "absent":
@@ -494,7 +528,7 @@ def main():
             if download_error is None:
                 changed = True
             else:
-                module.fail_json(msg="Cannot download the artifact to destination: " + download_error)
+                module.fail_json(msg="Cannot retrieve the artifact to destination: " + download_error)
         except ValueError as e:
             module.fail_json(msg=e.args[0])
 


Test patch: