Repository: ansible/ansible

Issue:
apt: unable to install deb from URI with python3 interpreter
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report


##### COMPONENT NAME
<!--- Name of the module/plugin/task/feature -->
```
apt
```
##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
2.2.0.0
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
```
host_key_checking = False
ansible_python_interpreter=/usr/bin/python3
```

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
```
both managed nodes and control node are: Ubuntu 16.04 LTS amd64
```

##### SUMMARY<!--- Explain the problem briefly -->
When declare python3 as the default python, apt module is unable to install deb over the Internet. However, if python2 is the default python, issue doesn't exist.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Assume the inventory file call "hosts" defines one host "localhost".
```
ansible -i hosts localhost -e 'ansible_python_interpreter=/usr/bin/python3' -u username -k  --ask-sudo-pass -m apt -a "deb=http://ftp.twaren.net/Linux/Ubuntu/ubuntu/pool/main/v/vim/vim-common_7.3.547-6ubuntu3_amd64.deb"
```
and it returns:
```
"msg": "Failure downloading http://ftp.twaren.net/Linux/Ubuntu/ubuntu/pool/main/v/vim/vim-common_7.3.547-6ubuntu3_amd64.deb, 'NoneType' object has no attribute 'read'"
```
I tried to install deb from private deb repo, the error message turned:
```
"msg": "Failure downloading http://provision.hct.com/pub/pool/main/p/provision-priv-repo/provision-priv-repo_2016.0809_all.deb, write() argument must be str, not bytes"
```
However if python interpreter is python(2.7.x), there is no problem at all. Below is a part of my playbook:
<!--- Paste example playbooks or commands between quotes below -->
```
- name: <registration> installing private apt repo...
  apt:
      deb: http://provision.hct.com/pub/pool/main/p/provision-priv-repo/provision-priv-repo_{{provision_version}}_all.deb

- name: <registration> installing AndApp8 private keys...
  apt:
      deb: http://provision.hct.com/pub/pool/main/a/andapp-privkey/andapp-privkey_{{privkey_version}}_all.deb
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Both "ansible_python_interpreter=/usr/bin/python3" and "ansible_python_interpreter=/usr/bin/python" should produce the same result.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
Parser seems to behave differently between python2 and python3.
<!--- Paste verbatim command output between quotes below -->




Discussion:
I think I have a fix for the error you're reporting with the private repository.  Trying to figure out what's happening with the public repo before making a PR for this.  (Fix for private repo error is here: https://gist.github.com/abadger/2b25e0e37769e22adf6989d9e6d4dbb4  )

@maxsolar Thanks for the bug report - I've run into the `write() argument must be str, not bytes` error on Python 3 today while trying to install [Dropbox](https://www.dropbox.com/install-linux) from url https://www.dropbox.com/download?dl=packages/ubuntu/dropbox_2015.10.28_amd64.deb

My understanding of the problem is:

* `f = open(package, 'w')` (from https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/packaging/os/apt.py#L719) The file is opened for writing.

* Python 3 sees this file in [text mode, not in binary mode](https://docs.python.org/3/library/functions.html#open)

* Exception is raised when `f.write(data)` happens on the first iteration. https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/packaging/os/apt.py#L727 - Python 3 has opened the file and so `write` is waiting for text and not bytes. When it receives bytes it raises.

@abadger Given what you taught me about Python 2 / 3 compatibility in https://github.com/ansible/ansible/pull/20037 (thanks :pray: ), and hoping that I've got the analysis of the problem right, I've got a working solution for Python 3 which I will open as PR soon.

Summary of my planned fix:

* Open the file in bytes mode for Python 2 and 3 (this is the bit I'm most unsure about).
* Force the output from `rsp.read(BUFSIZE)` to bytes with `to_bytes`.
* Use `len(data)` to check for the end of the loop rather than looking for the empty string / bytes.

Resolving patch:
diff --git a/lib/ansible/modules/packaging/os/apt.py b/lib/ansible/modules/packaging/os/apt.py
--- a/lib/ansible/modules/packaging/os/apt.py
+++ b/lib/ansible/modules/packaging/os/apt.py
@@ -234,7 +234,7 @@
 
 from ansible.module_utils.basic import AnsibleModule
 from ansible.module_utils.pycompat24 import get_exception
-from ansible.module_utils._text import to_native
+from ansible.module_utils._text import to_bytes, to_native
 from ansible.module_utils.urls import fetch_url
 
 # APT related constants
@@ -716,12 +716,14 @@ def download(module, deb):
 
     try:
         rsp, info = fetch_url(module, deb)
-        f = open(package, 'w')
+        # Ensure file is open in binary mode for Python 3
+        f = open(package, 'wb')
         # Read 1kb at a time to save on ram
         while True:
             data = rsp.read(BUFSIZE)
+            data = to_bytes(data, errors='surrogate_or_strict')
 
-            if data == "":
+            if len(data) < 1:
                 break # End of file, break while loop
 
             f.write(data)


Test patch: