Repository: ansible/ansible

Issue:
authorized_key: Crash when loading from Github by URL using Python 3.5
##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
authorized_key core module

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.2.0.0
  config file = /home/james/active/boss/ansible.cfg
  configured module search path = Default w/o overrides
```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
[defaults]
nocows=1
inventory=ansible_hosts
host_key_checking=False

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
-->
Local: Ubuntu with Python 3.5.0+ at `/usr/bin/python3.5`
Remote: Ubuntu 16.04.1 with Python 3.5.2 at `/usr/bin/python3.5`

Python version is specified with:

```
ansible_python_interpreter=/usr/bin/python3.5
```

##### SUMMARY
<!--- Explain the problem briefly -->

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Given that user `james` exists, attempt to install keys from https://github.com/jamescooke.keys using `authorized_key` module.

<!--- Paste example playbooks or commands between quotes below -->

Smallest example playbook I could create `test.yml`:

```yaml
---
- hosts: all
  vars:
    - ansible_ssh_user: root
  tasks:
  - authorized_key:
      user: james
      key: https://github.com/jamescooke.keys
```
<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Two keys would be added to `james` user on remote.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
Python crashes.

<!--- Paste verbatim command output between quotes below -->
```
(venv) james@codebox:~/active/boss$ ansible-playbook test.yml -vvvv
Using /home/james/active/boss/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from /home/james/active/boss/venv/lib/python3.5/site-packages/ansible/plugins/callback/__init__.py

PLAYBOOK: test.yml *************************************************************
1 plays in test.yml

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
Using module file /home/james/active/boss/venv/lib/python3.5/site-packages/ansible/modules/core/system/setup.py
<104.131.17.109> ESTABLISH SSH CONNECTION FOR USER: root
<104.131.17.109> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/james/.ansible/cp/ansible-ssh-%h-%p-%r 104.131.17.109 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1483808135.2190897-90788529272429 `" && echo ansible-tmp-1483808135.2190897-90788529272429="` echo $HOME/.ansible/tmp/ansible-tmp-1483808135.2190897-90788529272429 `" ) && sleep 0'"'"''
<104.131.17.109> PUT /tmp/tmpqi9xuich TO /root/.ansible/tmp/ansible-tmp-1483808135.2190897-90788529272429/setup.py
<104.131.17.109> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/james/.ansible/cp/ansible-ssh-%h-%p-%r '[104.131.17.109]'
<104.131.17.109> ESTABLISH SSH CONNECTION FOR USER: root
<104.131.17.109> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/james/.ansible/cp/ansible-ssh-%h-%p-%r 104.131.17.109 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1483808135.2190897-90788529272429/ /root/.ansible/tmp/ansible-tmp-1483808135.2190897-90788529272429/setup.py && sleep 0'"'"''
<104.131.17.109> ESTABLISH SSH CONNECTION FOR USER: root
<104.131.17.109> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/james/.ansible/cp/ansible-ssh-%h-%p-%r -tt 104.131.17.109 '/bin/sh -c '"'"'/usr/bin/python3.5 /root/.ansible/tmp/ansible-tmp-1483808135.2190897-90788529272429/setup.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1483808135.2190897-90788529272429/" > /dev/null 2>&1 && sleep 0'"'"''
ok: [104.131.17.109]

TASK [authorized_key] **********************************************************
task path: /home/james/active/boss/test.yml:6
Using module file /home/james/active/boss/venv/lib/python3.5/site-packages/ansible/modules/core/system/authorized_key.py
<104.131.17.109> ESTABLISH SSH CONNECTION FOR USER: root
<104.131.17.109> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/james/.ansible/cp/ansible-ssh-%h-%p-%r 104.131.17.109 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1483808139.0550506-81997757222281 `" && echo ansible-tmp-1483808139.0550506-81997757222281="` echo $HOME/.ansible/tmp/ansible-tmp-1483808139.0550506-81997757222281 `" ) && sleep 0'"'"''
<104.131.17.109> PUT /tmp/tmpinuhh2o4 TO /root/.ansible/tmp/ansible-tmp-1483808139.0550506-81997757222281/authorized_key.py
<104.131.17.109> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/james/.ansible/cp/ansible-ssh-%h-%p-%r '[104.131.17.109]'
<104.131.17.109> ESTABLISH SSH CONNECTION FOR USER: root
<104.131.17.109> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/james/.ansible/cp/ansible-ssh-%h-%p-%r 104.131.17.109 '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1483808139.0550506-81997757222281/ /root/.ansible/tmp/ansible-tmp-1483808139.0550506-81997757222281/authorized_key.py && sleep 0'"'"''
<104.131.17.109> ESTABLISH SSH CONNECTION FOR USER: root
<104.131.17.109> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/james/.ansible/cp/ansible-ssh-%h-%p-%r -tt 104.131.17.109 '/bin/sh -c '"'"'/usr/bin/python3.5 /root/.ansible/tmp/ansible-tmp-1483808139.0550506-81997757222281/authorized_key.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1483808139.0550506-81997757222281/" > /dev/null 2>&1 && sleep 0'"'"''
fatal: [104.131.17.109]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_name": "authorized_key"
    },
    "module_stderr": "OpenSSH_6.9p1 Ubuntu-2ubuntu0.2, OpenSSL 1.0.2d 9 Jul 2015\r\ndebug1: Reading configuration data /home/james/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 22341\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to 104.131.17.109 closed.\r\n",
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_grkuvdsh/ansible_module_authorized_key.py\", line 527, in <module>\r\n    main()\r\n  File \"/tmp/ansible_grkuvdsh/ansible_module_authorized_key.py\", line 523, in main\r\n    results = enforce_state(module, module.params)\r\n  File \"/tmp/ansible_grkuvdsh/ansible_module_authorized_key.py\", line 433, in enforce_state\r\n    key = [s for s in key.splitlines() if s and not s.startswith('#')]\r\n  File \"/tmp/ansible_grkuvdsh/ansible_module_authorized_key.py\", line 433, in <listcomp>\r\n    key = [s for s in key.splitlines() if s and not s.startswith('#')]\r\nTypeError: startswith first arg must be bytes or a tuple of bytes, not str\r\n",
    "msg": "MODULE FAILURE"
}
        to retry, use: --limit @/home/james/active/boss/test.retry

PLAY RECAP *********************************************************************
104.131.17.109             : ok=1    changed=0    unreachable=0    failed=1
```
Fails with:
* current master: https://github.com/ansible/ansible/tree/cba66dfedc4b34c622fb61209dbe355d8bcc8503
* 2.2.0.0 from pip: https://pypi.python.org/pypi/ansible/2.2.0.0


Discussion:
The error is produced because the `bytes.startswith` method needs to be called with a `bytes` type argument and likewise for`str.startswith`, with `str` as argument. When fetching the URL contents, the result of `resp.read` is of type `bytes` while the argument `'#'` is a string.

It seems that this [line](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/system/authorized_key.py#L483) should become:

```python
key = str(resp.read())
```

With this change it will probably work for both Python 2 and Python 3.

@jamescooke could you check this patch?
@georgepsarakis I think I checked this already in my venv when I was hacking around with it and it fails further down the flow ... Will try and report back.
@georgepsarakis The line that you've linked to is https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/system/authorized_key.py#L483 and contains...

    new_keys = keys.values()

... within the `serialize(keys)` function.

It doesn't make sense to me to change that to `key = str(resp.read())` - sorry if I'm being slow.

Instead the only instance of `resp.read` in the `authorized_key.py` file at this time is here: https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/system/authorized_key.py#L536 - it makes sense to me to change that to `key = str(resp.read())` as you mention because it should cast the result of `resp.read` to `str` as you mention (this is the change that I mentioned before that I'd tried previously).

Using `master` and changing that line creates a follow on problem where the newlines don't appear to be split and the whole keyfile is treated as a single key... here's the output just with the error at the end:

```
fatal: [45.55.226.156]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "exclusive": false,
            "key": "https://github.com/jamescooke.keys",
            "key_options": null,
            "keyfile": "/home/james/.ssh/authorized_keys",
            "manage_dir": true,
            "path": null,
            "state": "present",
            "unique": false,
            "user": "james",
            "validate_certs": true
        },
        "module_name": "authorized_key"
    },
    "msg": "invalid key specified: b'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgW65xUSkwy1xSrTs3SpJeSMvZOl5EuVg23XyXzH4Q1Tj15SpJJSLOj3JGZF2GwWy5aH4zWnvjG5I1Rpl9QkkbY2Jup9FGZypLzySxUesxTyjRhX7+lMWnTMdVFdl+UuHJVommsGACjScZnDliRiV+qn2iBGe3GnAmikQIlB+3cRXmnqvtgkCJCihxVXwV3BZgM7TAwtgKRIVgOmaJ2H6a+2utUJ1i8+vb1nJeipEdY+p+yLwvgjgXTW0l+Qfsl57+IPenDfSosF6a+DUftoojZVpc2lI4zE89L49Bt/evmb3lbn+6s1jdLDyhvNxwL7TT4BvQN9VaAtigtdsJvCWZ\\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9II+7eZuJopDFgb48dxsAIhlkoDykToBUtsQwot3N7XeiX0SYodD758byveaeHC6aw1OPHL9MpAR8PzYpk919jnbSYSM///M5L9SDHui+d3MWxQC8hM+p+oADJLHgaixxUs+GHQTCrVEKW8LxsvA2Y1VN2NA8uRXhNKMyafXRnWsG2bUD+WpPhFmmdodX1HS6LPB3bqPITUEIAX01QhwauUU5pOJutv2d0qLkSr/qmh0/WIMydQ03QV7UwUaa0vX0bvXsrxg9g5GhXFSs5PtB3/9RnrazvWRNrn5519aPTan8NPWBONlxXyJ3cXNvG97yNlWvtAW+ujfnDc7XbESh\\n'"
}
        to retry, use: --limit @/home/james/active/boss/test.retry

PLAY RECAP **********************************************************************************************
45.55.226.156              : ok=1    changed=0    unreachable=0    failed=1
```

Forgive my innocence - shouldn't there be a failing test somewhere for this in Ansible's test suite? And if not, then one should be created as part of this ticket?
Sorry for that, yes I meant this [line](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/system/authorized_key.py#L536). 

You are right, you cannot simply type-cast from `bytes` to `str`, so you can do this:

```
from ansible.compat.six import PY3

key = resp.read()
if PY3:
    key = key.decode()
```
I couldn't find a compatibility function for converting `bytes` to `str`.

If the above does resolve this, yes you could open a pull request for the fix along with a test perhaps.
Thanks for the guidance George - I should be able to test this and hopefully make a pull request tomorrow.
@jamescooke you are most welcome. Let me know if I can help more on this.

Resolving patch:
diff --git a/lib/ansible/modules/system/authorized_key.py b/lib/ansible/modules/system/authorized_key.py
--- a/lib/ansible/modules/system/authorized_key.py
+++ b/lib/ansible/modules/system/authorized_key.py
@@ -227,6 +227,7 @@
 import shlex
 from operator import itemgetter
 
+from ansible.module_utils._text import to_native
 from ansible.module_utils.basic import AnsibleModule
 from ansible.module_utils.pycompat24 import get_exception
 from ansible.module_utils.urls import fetch_url
@@ -537,6 +538,9 @@ def enforce_state(module, params):
         except Exception:
             module.fail_json(msg=error_msg % key)
 
+        # resp.read gives bytes on python3, convert to native string type
+        key = to_native(key, errors='surrogate_or_strict')
+
     # extract individual keys into an array, skipping blank lines and comments
     new_keys = [s for s in key.splitlines() if s and not s.startswith('#')]
 


Test patch: