Repository: ansible/ansible

Issue:
jenkins_script produces Python 3 error TypeError: the JSON object must be str, not 'bytes'
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
 - Bug Report


##### COMPONENT NAME
jenkins_script

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.4.0.0
  config file = /home/<machine_name>/Projects/ansible.cfg
  configured module search path = [u'/home/<machine_name>/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]

```

##### CONFIGURATION

**"ansible-config dump --only-changed"**
```bash
DEFAULT_ROLES_PATH(/home/<machine_name>/Projects/ansible.cfg) = [u'/home/<machine-name>/Projects/ansible-roles']
```

**ansible.cfg**
[defaults]
roles_path = ../ansible-roles/


**inventory**
[jenkins]
192.168.0.12 ansible_python_interpreter=/usr/bin/python3


<!---
If using Ansible 2.4 or above, paste the results of "ansible-config dump --only-changed"

Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).

-->

##### OS / ENVIRONMENT
Ubuntu 16.04 (Both)
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->

##### SUMMARY
Similar to #24526 jenkins_script will fail when inventory file forces Python3 interpeter i.e.

[jenkins]
192.168.0.12 ansible_python_interpreter=/usr/bin/python3

Error:
```fatal: [192.168.0.10]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Shared connection to 192.168.0.10 closed.\r\n", 
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_grpdvu33/ansible_module_jenkins_script.py\", line 196, in <module>\r\n    main()\r\n  File \"/tmp/ansible_grpdvu33/ansible_module_jenkins_script.py\", line 171, in main\r\n    if is_csrf_protection_enabled(module):\r\n  File \"/tmp/ansible_grpdvu33/ansible_module_jenkins_script.py\", line 129, in is_csrf_protection_enabled\r\n    return json.loads(content).get('useCrumbs', False)\r\n  File \"/usr/lib/python3.5/json/__init__.py\", line 312, in loads\r\n    s.__class__.__name__))\r\nTypeError: the JSON object must be str, not 'bytes'\r\n", 
    "msg": "MODULE FAILURE", 
    "rc": 0
}
```

<!--- Explain the problem briefly -->

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml

- name: Execute Jenkins Plugins Script
  jenkins_script:
    script: "println(Jenkins.instance.pluginManager.plugins)"
    user: admin
    password: admin
    url: http://localhost:8080
    validate_certs: no

```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

List of actual plugins via the output i.e.
```
    "output": "[Plugin:cloudbees-folder, Plugin:bouncycastle-api, Plugin:structs, Plugin:junit, Plugin:antisamy-markup-formatter, Plugin:workflow-step-api, Plugin:scm-api, Plugin:script-security, Plugin:workflow-api, Plugin:workflow-support, Plugin:workflow-job, Plugin:token-macro, Plugin:build-timeout, Plugin:credentials, Plugin:ssh-credentials, Plugin:plain-credentials, Plugin:credentials-binding, Plugin:timestamper, Plugin:durable-task, Plugin:workflow-durable-task-step, Plugin:matrix-project, Plugin:resource-disposer, Plugin:ws-cleanup, Plugin:ant, Plugin:gradle, Plugin:pipeline-milestone-step, Plugin:jquery-detached, Plugin:jackson2-api, Plugin:ace-editor, Plugin:workflow-scm-step, Plugin:workflow-cps, Plugin:pipeline-input-step, Plugin:pipeline-stage-step, Plugin:pipeline-graph-analysis, Plugin:pipeline-rest-api, Plugin:handlebars, Plugin:momentjs, Plugin:pipeline-stage-view, Plugin:pipeline-build-step, Plugin:pipeline-model-api, Plugin:pipeline-model-extensions, Plugin:git-client, Plugin:git-server, Plugin:workflow-cps-global-lib, Plugin:display-url-api, Plugin:mailer, Plugin:branch-api, Plugin:workflow-multibranch, Plugin:authentication-tokens, Plugin:docker-commons, Plugin:docker-workflow, Plugin:pipeline-stage-tags-metadata, Plugin:pipeline-model-declarative-agent, Plugin:workflow-basic-steps, Plugin:pipeline-model-definition, Plugin:workflow-aggregator, Plugin:github-api, Plugin:git, Plugin:github, Plugin:github-branch-source, Plugin:pipeline-github-lib, Plugin:mapdb-api, Plugin:subversion, Plugin:ssh-slaves, Plugin:matrix-auth, Plugin:pam-auth, Plugin:ldap, Plugin:email-ext]\n"


```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
Using module file /usr/lib/python2.7/dist-packages/ansible/modules/web_infrastructure/jenkins_script.py
<192.168.0.10> ESTABLISH SSH CONNECTION FOR USER: vagrant
<192.168.0.10> SSH: EXEC ssh -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -i /home/beaver-root/Projects/node-jenkins/.vagrant/machines/nodeubuntu/virtualbox/private_key -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 -o ControlPath=/home/beaver-root/.ansible/cp/7753a934db 192.168.0.10 '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<192.168.0.10> (0, '/home/vagrant\n', '')
<192.168.0.10> ESTABLISH SSH CONNECTION FOR USER: vagrant
<192.168.0.10> SSH: EXEC ssh -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -i /home/beaver-root/Projects/node-jenkins/.vagrant/machines/nodeubuntu/virtualbox/private_key -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 -o ControlPath=/home/beaver-root/.ansible/cp/7753a934db 192.168.0.10 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/vagrant/.ansible/tmp/ansible-tmp-1507923124.56-15792027556640 `" && echo ansible-tmp-1507923124.56-15792027556640="` echo /home/vagrant/.ansible/tmp/ansible-tmp-1507923124.56-15792027556640 `" ) && sleep 0'"'"''
<192.168.0.10> (0, 'ansible-tmp-1507923124.56-15792027556640=/home/vagrant/.ansible/tmp/ansible-tmp-1507923124.56-15792027556640\n', '')
<192.168.0.10> PUT /tmp/tmpZg3Pfc TO /home/vagrant/.ansible/tmp/ansible-tmp-1507923124.56-15792027556640/jenkins_script.py
<192.168.0.10> SSH: EXEC sftp -b - -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -i /home/beaver-root/Projects/node-jenkins/.vagrant/machines/nodeubuntu/virtualbox/private_key -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 -o ControlPath=/home/beaver-root/.ansible/cp/7753a934db '[192.168.0.10]'
<192.168.0.10> (0, 'sftp> put /tmp/tmpZg3Pfc /home/vagrant/.ansible/tmp/ansible-tmp-1507923124.56-15792027556640/jenkins_script.py\n', '')
<192.168.0.10> ESTABLISH SSH CONNECTION FOR USER: vagrant
<192.168.0.10> SSH: EXEC ssh -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -i /home/beaver-root/Projects/node-jenkins/.vagrant/machines/nodeubuntu/virtualbox/private_key -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 -o ControlPath=/home/beaver-root/.ansible/cp/7753a934db 192.168.0.10 '/bin/sh -c '"'"'chmod u+x /home/vagrant/.ansible/tmp/ansible-tmp-1507923124.56-15792027556640/ /home/vagrant/.ansible/tmp/ansible-tmp-1507923124.56-15792027556640/jenkins_script.py && sleep 0'"'"''
<192.168.0.10> (0, '', '')
<192.168.0.10> ESTABLISH SSH CONNECTION FOR USER: vagrant
<192.168.0.10> SSH: EXEC ssh -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -i /home/beaver-root/Projects/node-jenkins/.vagrant/machines/nodeubuntu/virtualbox/private_key -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=vagrant -o ConnectTimeout=30 -o ControlPath=/home/beaver-root/.ansible/cp/7753a934db -tt 192.168.0.10 '/bin/sh -c '"'"'/usr/bin/python3 /home/vagrant/.ansible/tmp/ansible-tmp-1507923124.56-15792027556640/jenkins_script.py; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1507923124.56-15792027556640/" > /dev/null 2>&1 && sleep 0'"'"''
<192.168.0.10> (0, 'Traceback (most recent call last):\r\n  File "/tmp/ansible_cg6ilms0/ansible_module_jenkins_script.py", line 196, in <module>\r\n    main()\r\n  File "/tmp/ansible_cg6ilms0/ansible_module_jenkins_script.py", line 171, in main\r\n    if is_csrf_protection_enabled(module):\r\n  File "/tmp/ansible_cg6ilms0/ansible_module_jenkins_script.py", line 129, in is_csrf_protection_enabled\r\n    return json.loads(content).get(\'useCrumbs\', False)\r\n  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads\r\n    s.__class__.__name__))\r\nTypeError: the JSON object must be str, not \'bytes\'\r\n', 'Shared connection to 192.168.0.10 closed.\r\n')
fatal: [192.168.0.10]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Shared connection to 192.168.0.10 closed.\r\n", 
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_cg6ilms0/ansible_module_jenkins_script.py\", line 196, in <module>\r\n    main()\r\n  File \"/tmp/ansible_cg6ilms0/ansible_module_jenkins_script.py\", line 171, in main\r\n    if is_csrf_protection_enabled(module):\r\n  File \"/tmp/ansible_cg6ilms0/ansible_module_jenkins_script.py\", line 129, in is_csrf_protection_enabled\r\n    return json.loads(content).get('useCrumbs', False)\r\n  File \"/usr/lib/python3.5/json/__init__.py\", line 312, in loads\r\n    s.__class__.__name__))\r\nTypeError: the JSON object must be str, not 'bytes'\r\n", 
    "msg": "MODULE FAILURE", 
    "rc": 0
}

```

**Workaround**
remove `ansible_python_interpreter=/usr/bin/python3` from inventory file



Discussion:
cc @hogarthj
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/module_utils/urls.py b/lib/ansible/module_utils/urls.py
--- a/lib/ansible/module_utils/urls.py
+++ b/lib/ansible/module_utils/urls.py
@@ -941,7 +941,7 @@ def fetch_url(module, url, data=None, headers=None, method=None,
     :kwarg last_mod_time: Default: None
     :kwarg int timeout:   Default: 10
 
-    :returns: A tuple of (**response**, **info**). Use ``response.body()`` to read the data.
+    :returns: A tuple of (**response**, **info**). Use ``response.read()`` to read the data.
         The **info** contains the 'status' and other meta data. When a HttpError (status > 400)
         occurred then ``info['body']`` contains the error response data::
 
diff --git a/lib/ansible/modules/web_infrastructure/jenkins_script.py b/lib/ansible/modules/web_infrastructure/jenkins_script.py
--- a/lib/ansible/modules/web_infrastructure/jenkins_script.py
+++ b/lib/ansible/modules/web_infrastructure/jenkins_script.py
@@ -116,6 +116,7 @@
 from ansible.module_utils.basic import AnsibleModule
 from ansible.module_utils.six.moves.urllib.parse import urlencode
 from ansible.module_utils.urls import fetch_url
+from ansible.module_utils._text import to_native
 
 
 def is_csrf_protection_enabled(module):
@@ -125,7 +126,7 @@ def is_csrf_protection_enabled(module):
     if info["status"] != 200:
         module.fail_json(msg="HTTP error " + str(info["status"]) + " " + info["msg"])
 
-    content = resp.read()
+    content = to_native(resp.read())
     return json.loads(content).get('useCrumbs', False)
 
 
@@ -136,7 +137,7 @@ def get_crumb(module):
     if info["status"] != 200:
         module.fail_json(msg="HTTP error " + str(info["status"]) + " " + info["msg"])
 
-    content = resp.read()
+    content = to_native(resp.read())
     return json.loads(content)
 
 
@@ -182,7 +183,7 @@ def main():
     if info["status"] != 200:
         module.fail_json(msg="HTTP error " + str(info["status"]) + " " + info["msg"])
 
-    result = resp.read()
+    result = to_native(resp.read())
 
     if 'Exception:' in result and 'at java.lang.Thread' in result:
         module.fail_json(msg="script failed with stacktrace:\n " + result)


Test patch: