Repository: ansible/ansible

Issue:
Unicode parsing issue with junos_config
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
junos_config module

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.3.0.0 (stable-2.3 7941f1ede7) last updated 2017/04/05 10:36:38 (GMT -700)
  config file = /home/users/cody.john/virtualenv/ansible-git/playbook/deploy/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.10 (default, Jul 18 2016, 14:18:08) [GCC 4.4.7 20120313 (Red Hat 4.4.7-16)]
```

##### CONFIGURATION
<!---
ANSIBLE_HOME=/home/users/cody.john/virtualenv/ansible-git/ansible
ANSIBLE_LOG_PATH=./debug.log
-->

##### OS / ENVIRONMENT
<!---
Centos 6.8
-->

##### SUMMARY
<!--- Explain the problem briefly -->
When using the junos_config module and setting the 'backup' parameter to 'yes', there appears to be an ascii-to-unicode issue (UnicodeEncodeError).

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Create a playbook with a single task using the junos_config module.   Apply a test ACL configuration, and set the "backup" paramter to "yes".

<!--- Paste example playbooks or commands between quotes below -->
```yaml
---
- name: Test junos_config module
  hosts: test
  gather_facts: no
  connection: local

  vars:
    ansible_python_interpreter: /glide/bin/python27
    cli:
      host: "{{ inventory_hostname }}"

  tasks:
    - name: junos_config ACL test
      junos_config:
        provider: "{{ cli }}"
        backup: yes
        lines:
          - set firewall family inet filter ANSIBLE-TEST term ANSIBLE then accept
          - delete firewall family inet filter ANSIBLE-TEST
          - set firewall family inet filter ANSIBLE-TEST term TEST-TERM-1 from protocol udp
          - set firewall family inet filter ANSIBLE-TEST term TEST-TERM-1 then log
          - set firewall family inet filter ANSIBLE-TEST term TEST-TERM-1 then discard
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
Expect the junos_config module to create a backup of the current running configuration, and then apply the lines in the play to the configuration.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
(ansible-git)[cns04.sea3:/home/users/cody.john/virtualenv/ansible-git/playbook/deploy]$ ansible-playbook junos_config_test.yml -i inventory --limit cor-001a.sea0.service-now.com --check --diff -vvvv
Using /home/users/cody.john/virtualenv/ansible-git/playbook/deploy/ansible.cfg as config file
Loading callback plugin default of type stdout, v2.0 from /home/users/cody.john/virtualenv/ansible-git/ansible/lib/ansible/plugins/callback/__init__.pyc
Loading callback plugin slack of type notification, v2.0 from /home/users/cody.john/virtualenv/ansible-git/ansible/lib/ansible/plugins/callback/__init__.pyc
 [WARNING]: The `prettytable` python module is not installed. Disabling the Slack callback plugin.

 [WARNING]: Slack Webhook URL was not provided. The Slack Webhook URL can be provided using the `SLACK_WEBHOOK_URL` environment variable.


PLAYBOOK: junos_config_test.yml *************************************************************************************************************
1 plays in junos_config_test.yml

PLAY [Test junos_config module] *************************************************************************************************************
META: ran handlers

TASK [junos_config ACL test] ****************************************************************************************************************
task path: /home/users/cody.john/virtualenv/ansible-git/playbook/deploy/junos_config_test.yml:13
<cor-001a.sea0.service-now.com> using connection plugin netconf
<cor-001a.sea0.service-now.com> socket_path: /home/users/cody.john/.ansible/pc/a647017206
Using module file /home/users/cody.john/virtualenv/ansible-git/ansible/lib/ansible/modules/network/junos/junos_config.py
<cor-001a.sea0.service-now.com> ESTABLISH LOCAL CONNECTION FOR USER: cody.john
<cor-001a.sea0.service-now.com> EXEC /bin/sh -c 'echo ~ && sleep 0'
<cor-001a.sea0.service-now.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/users/cody.john/.ansible/tmp/ansible-tmp-1491414351.75-223916974745349 `" && echo ansible-tmp-1491414351.75-223916974745349="` echo /home/users/cody.john/.ansible/tmp/ansible-tmp-1491414351.75-223916974745349 `" ) && sleep 0'
<cor-001a.sea0.service-now.com> PUT /tmp/tmpUR5ngx TO /home/users/cody.john/.ansible/tmp/ansible-tmp-1491414351.75-223916974745349/junos_config.py
<cor-001a.sea0.service-now.com> EXEC /bin/sh -c 'chmod u+x /home/users/cody.john/.ansible/tmp/ansible-tmp-1491414351.75-223916974745349/ /home/users/cody.john/.ansible/tmp/ansible-tmp-1491414351.75-223916974745349/junos_config.py && sleep 0'
<cor-001a.sea0.service-now.com> EXEC /bin/sh -c '/glide/bin/python27 /home/users/cody.john/.ansible/tmp/ansible-tmp-1491414351.75-223916974745349/junos_config.py; rm -rf "/home/users/cody.john/.ansible/tmp/ansible-tmp-1491414351.75-223916974745349/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_R2QaH_/ansible_module_junos_config.py", line 341, in <module>
    main()
  File "/tmp/ansible_R2QaH_/ansible_module_junos_config.py", line 316, in main
    result['__backup__'] = str(match.text).strip()
UnicodeEncodeError: 'ascii' codec can't encode characters in position 76583-76585: ordinal not in range(128)

fatal: [cor-001a.sea0.service-now.com]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_R2QaH_/ansible_module_junos_config.py\", line 341, in <module>\n    main()\n  File \"/tmp/ansible_R2QaH_/ansible_module_junos_config.py\", line 316, in main\n    result['__backup__'] = str(match.text).strip()\nUnicodeEncodeError: 'ascii' codec can't encode characters in position 76583-76585: ordinal not in range(128)\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 0
}
	to retry, use: --limit @/home/users/cody.john/virtualenv/ansible-git/playbook/deploy/junos_config_test.retry

PLAY RECAP **********************************************************************************************************************************
cor-001a.sea0.service-now.com : ok=0    changed=0    unreachable=0    failed=1   

(ansible-git)[cns04.sea3:/home/users/cody.john/virtualenv/ansible-git/playbook/deploy]$
```



Discussion:
cc @Qalthos @ganeshrn
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I can't manage to reproduce this at all... I assume you have some unicode somewhere in your configuration, but nothing I've tried has resulted in a UnicodeEncodeError, though admittedly the non-ASCII text has not been faithfully transcribed.

If you have more details about your particular configuration that might help me reproduce it, or whatever else I might be missing here, that would be very helpful.
I will see if I can track down what it it is in my config that is causing the issue.   My first thought is that it might be my interface descriptions.   Can you try adding a description to any interface in the following format?

`set interfaces xe-0/0/0 unit 0 description "[foo=bar, type=UL, nbr=sw-001:xe-0/0/0] blah"`
I found it.   Looks like I have a few descriptions that contain an "em dash".  Try to reproduce by setting a description such as:

`set interfaces xe-0/0/0 unit 0 description "hyphen(-), em dash(–)"`

Resolving patch:
diff --git a/lib/ansible/module_utils/junos.py b/lib/ansible/module_utils/junos.py
--- a/lib/ansible/module_utils/junos.py
+++ b/lib/ansible/module_utils/junos.py
@@ -24,6 +24,7 @@
 from ansible.module_utils.netconf import send_request, children
 from ansible.module_utils.netconf import discard_changes, validate
 from ansible.module_utils.six import string_types
+from ansible.module_utils._text import to_text
 
 ACTIONS = frozenset(['merge', 'override', 'replace', 'update', 'set'])
 JSON_ACTIONS = frozenset(['merge', 'override', 'update'])
@@ -111,7 +112,7 @@ def load_configuration(module, candidate=None, action='merge', rollback=None, fo
             if format == 'xml':
                 cfg.append(fromstring(candidate))
             else:
-                cfg.text = candidate
+                cfg.text = to_text(candidate, encoding='latin1')
         else:
             cfg.append(candidate)
     return send_request(module, obj)
@@ -163,7 +164,7 @@ def get_diff(module):
     reply = get_configuration(module, compare=True, format='text')
     output = reply.find('.//configuration-output')
     if output is not None:
-        return output.text
+        return to_text(output.text, encoding='latin1').strip()
 
 def load_config(module, candidate, warnings, action='merge', commit=False, format='xml',
                 comment=None, confirm=False, confirm_timeout=None):
@@ -180,7 +181,6 @@ def load_config(module, candidate, warnings, action='merge', commit=False, forma
         diff = get_diff(module)
 
         if diff:
-            diff = str(diff).strip()
             if commit:
                 commit_configuration(module, confirm=confirm, comment=comment,
                                      confirm_timeout=confirm_timeout)
diff --git a/lib/ansible/modules/network/junos/junos_config.py b/lib/ansible/modules/network/junos/junos_config.py
--- a/lib/ansible/modules/network/junos/junos_config.py
+++ b/lib/ansible/modules/network/junos/junos_config.py
@@ -194,6 +194,7 @@
 from ansible.module_utils.junos import check_args as junos_check_args
 from ansible.module_utils.netconf import send_request
 from ansible.module_utils.six import string_types
+from ansible.module_utils._text import to_text, to_native
 
 USE_PERSISTENT_CONNECTION = True
 DEFAULT_COMMENT = 'configured by junos_config'
@@ -231,7 +232,7 @@ def filter_delete_statements(module, candidate):
     if match is None:
         # Could not find configuration-set in reply, perhaps device does not support it?
         return candidate
-    config = str(match.text)
+    config = to_native(match.text, encoding='latin1')
 
     modified_candidate = candidate[:]
     for index, line in enumerate(candidate):
@@ -324,7 +325,7 @@ def main():
         else:
             module.fail_json(msg='unable to retrieve device configuration')
 
-        result['__backup__'] = str(match.text).strip()
+        result['__backup__'] = match.text.strip()
 
     if module.params['rollback']:
         if not module.check_mode:
diff --git a/lib/ansible/plugins/action/junos_config.py b/lib/ansible/plugins/action/junos_config.py
--- a/lib/ansible/plugins/action/junos_config.py
+++ b/lib/ansible/plugins/action/junos_config.py
@@ -27,6 +27,7 @@
 from ansible.plugins.action.junos import ActionModule as _ActionModule
 from ansible.module_utils._text import to_text
 from ansible.module_utils.six.moves.urllib.parse import urlsplit
+from ansible.module_utils._text import to_native
 from ansible.utils.vars import merge_hash
 
 PRIVATE_KEYS_RE = re.compile('__.+__')
@@ -74,7 +75,7 @@ def _write_backup(self, host, contents):
             os.remove(fn)
         tstamp = time.strftime("%Y-%m-%d@%H:%M:%S", time.localtime(time.time()))
         filename = '%s/%s_config.%s' % (backup_path, host, tstamp)
-        open(filename, 'w').write(contents)
+        open(filename, 'w').write(to_native(contents, encoding='latin1'))
         return filename
 
     def _handle_template(self):
@@ -110,4 +111,3 @@ def _handle_template(self):
         searchpath.append(os.path.dirname(source))
         self._templar.environment.loader.searchpath = searchpath
         self._task.args['src'] = self._templar.template(template_data)
-


Test patch:
diff --git a/test/sanity/pep8/legacy-files.txt b/test/sanity/pep8/legacy-files.txt
--- a/test/sanity/pep8/legacy-files.txt
+++ b/test/sanity/pep8/legacy-files.txt
@@ -851,7 +851,6 @@ lib/ansible/plugins/action/group_by.py
 lib/ansible/plugins/action/ios_template.py
 lib/ansible/plugins/action/iosxr_template.py
 lib/ansible/plugins/action/junos.py
-lib/ansible/plugins/action/junos_config.py
 lib/ansible/plugins/action/junos_template.py
 lib/ansible/plugins/action/normal.py
 lib/ansible/plugins/action/nxos.py