Repository: ansible/ansible

Issue:
maven_artifact spinning with 100% CPU on checking md5 with python3
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
<!--- Explain the problem briefly -->
When using `ansible_python_interpreter: /usr/bin/python3` maven_artifact hangs on checking the MD5 checksum.

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
maven_artifact

##### ANSIBLE VERSION
```
ansible 2.6.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/michael/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
```
Also affected: 2.5.2

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->
ANSIBLE_HOST_KEY_CHECKING=False

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->

OS running Ansible from: Ubuntu Server 16.04
Target OS: Ubuntu Server 18.04

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->
Using variables for `artifactory_base_url`, `artifactory_user`, `artifactory_password`, and `ansible_python_interpreter: /usr/bin/python3`:

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: Download myservlet.war
  maven_artifact: group_id=com.sample artifact_id=myservlet extension=war dest=/var/tmp/myservlet.war repository_url={{ artifactory_base_url }}/maven username={{ artifactory_user }} password={{ artifactory_password }}
  register: downloaded_myservlet
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Module execution completes.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
Module execution hangs after download, if MD5 checksum is checked, i.e. if file already existed (2.5.2) or if it did not yet exist (2.6). With 2.6 the problem disappears when using `verify_checksum=never`.
<!--- Paste verbatim command output between quotes below -->
```
# ps -fp24825
UID        PID  PPID  C STIME TTY          TIME CMD
root     24825 24824 99 10:14 pts/1    00:01:57 /usr/bin/python3 /tmp/ansible_iet610ng/ansible_module_maven_artifact.py
# strace -p24825 2>&1 | head -20
strace: Process 24825 attached
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
read(8, "", 8192)                       = 0
# lsof -p24825 | grep 8r
python3 24825 root    8r   REG              252,1   151883  783477 /var/tmp/myservlet.war
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/packaging/language/maven_artifact.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/packaging/language/maven_artifact.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @tumbl3w33d @turb
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I can reproduce it outside Ansible, just with this script:

```
import hashlib
import io
import sys

file = sys.argv[0]

md5 = hashlib.md5()
with io.open(file, 'rb') as f:
	for chunk in iter(lambda: f.read(8192), ''):
		md5.update(chunk)
print(md5.hexdigest())
```

Using python2, it works perfectly, but using python3 it hangs forever.

Need to check in [the documentation](https://docs.python.org/3/library/hashlib.html) what is missing.
If you modify the iterator termination to
```
for chunk in iter(lambda: f.read(8192), b''):
```
your script works fine with both python2 and python3.

I couldn't verify the same fix in maven_artifact.py, though. Tried the following:

- Created a copy of `maven_artifact.py` renamed to `python3_maven_artifact.py` in the library directory of my playbook.
- Applied the fix `iter(lambda: f.read(8192), '')` to `iter(lambda: f.read(8192), b'')` in function _local_md5
- Modified the task to use `python3_maven_artifact.py`.
 

Execution no longer hangs, but fails with
```
Traceback (most recent call last):
  File "/tmp/ansible__mx_xgxa/ansible_module_python3_maven_artifact.py", line 553, in <module>
    main()
  File "/tmp/ansible__mx_xgxa/ansible_module_python3_maven_artifact.py", line 535, in main
    download_error = downloader.download(artifact, verify_download, b_dest)
  File "/tmp/ansible__mx_xgxa/ansible_module_python3_maven_artifact.py", line 388, in download
    invalid_md5 = self.is_invalid_md5(filename, url)
  File "/tmp/ansible__mx_xgxa/ansible_module_python3_maven_artifact.py", line 434, in is_invalid_md5
    return "Checksum does not match: we computed " + local_md5 + "but the repository states " + remote_md5
TypeError: must be str, not bytes
```
Now I believe I got it. The TypeError is caused by the server response being bytes rather than string.

https://github.com/ansible/ansible/pull/42737

works fine for me.
Created new pull request for devel branch: https://github.com/ansible/ansible/pull/42878

Resolving patch:
diff --git a/lib/ansible/modules/packaging/language/maven_artifact.py b/lib/ansible/modules/packaging/language/maven_artifact.py
--- a/lib/ansible/modules/packaging/language/maven_artifact.py
+++ b/lib/ansible/modules/packaging/language/maven_artifact.py
@@ -177,7 +177,7 @@
 from ansible.module_utils.basic import AnsibleModule
 from ansible.module_utils.six.moves.urllib.parse import urlparse
 from ansible.module_utils.urls import fetch_url
-from ansible.module_utils._text import to_bytes
+from ansible.module_utils._text import to_bytes, to_native, to_text
 
 
 def split_pre_existing_dir(dirname):
@@ -428,7 +428,10 @@ def is_invalid_md5(self, file, remote_url):
                 parsed_url = urlparse(remote_url)
                 remote_md5 = self._local_md5(parsed_url.path)
             else:
-                remote_md5 = self._getContent(remote_url + '.md5', "Failed to retrieve MD5", False)
+                try:
+                    remote_md5 = to_text(self._getContent(remote_url + '.md5', "Failed to retrieve MD5", False), errors='strict')
+                except UnicodeError as e:
+                    return "Cannot retrieve a valid md5 from %s: %s" % (remote_url, to_native(e))
                 if(not remote_md5):
                     return "Cannot find md5 from " + remote_url
             if local_md5 == remote_md5:
@@ -441,7 +444,7 @@ def is_invalid_md5(self, file, remote_url):
     def _local_md5(self, file):
         md5 = hashlib.md5()
         with io.open(file, 'rb') as f:
-            for chunk in iter(lambda: f.read(8192), ''):
+            for chunk in iter(lambda: f.read(8192), b''):
                 md5.update(chunk)
         return md5.hexdigest()
 


Test patch: