Repository: ansible/ansible

Issue:
junos_config: unable to delete interfaces (UnicodeEncodeError)
<!---
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release, and master branch are affected too.
-->

##### ISSUE TYPE
<!--- Pick one below and delete the rest: -->
 - Bug Report

##### COMPONENT NAME
<!---  -->
junos_config

##### ANSIBLE VERSION
<!--- Paste verbatim output from “ansible --version” between quotes below -->
```
ansible 2.3.2.0
  config file = /*****/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.13 (default, Jan 19 2017, 14:48:08) [GCC 6.3.0 20170118]

```

##### CONFIGURATION
<!---
Mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).
-->
Ansible.cfg:
[defaults]
host_key_checking=False
timeout = 10

##### OS / ENVIRONMENT
<!---
Mention the OS you are running Ansible from, and the OS you are
managing, or say “N/A” for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.
-->
N/A
##### SUMMARY
<!--- Explain the problem briefly -->

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->
Easy to reproduce, just launch the following play, the workaround found is to replace line 231 `config = str(match.text)` by `config = match.text.encode('utf-8')` in the junos network module named: junos_config.py (see output error for more details)

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: delete ipv4 address
  block:
    - junos_config:
        lines:
        - delete interfaces lo0 unit 0 family inet address 7.7.7.3/32
        comment: updated by Ansible
        provider: "{{ netconf }}"
      register: result
      notify: changed_config
  rescue:
    - debug: msg='Error removing ipv4 address'

- debug:
    var: result
    verbosity: 4
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
No error

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_3ikpwv/ansible_module_junos_config.py", line 349, in <module>
    main()
  File "/tmp/ansible_3ikpwv/ansible_module_junos_config.py", line 339, in main
    diff = configure_device(module, warnings)
  File "/tmp/ansible_3ikpwv/ansible_module_junos_config.py", line 271, in configure_device
    candidate = filter_delete_statements(module, candidate)
  File "/tmp/ansible_3ikpwv/ansible_module_junos_config.py", line 231, in filter_delete_statements
    config = str(match.text)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 43581-43583: ordinal not in range(128)

fatal: [jun]: FAILED! => {
    "changed": false,
    "failed": true,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_3ikpwv/ansible_module_junos_config.py\", line 349, in <module>\n    main()\n  File \"/tmp/ansible_3ikpwv/ansible_module_junos_config.py\", line 339, in main\n    diff = configure_device(module, warnings)\n  File \"/tmp/ansible_3ikpwv/ansible_module_junos_config.py\", line 271, in configure_device\n    candidate = filter_delete_statements(module, candidate)\n  File \"/tmp/ansible_3ikpwv/ansible_module_junos_config.py\", line 231, in filter_delete_statements\n    config = str(match.text)\nUnicodeEncodeError: 'ascii' codec can't encode characters in position 43581-43583: ordinal not in range(128)\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}


Discussion:

Resolving patch:
diff --git a/lib/ansible/module_utils/junos.py b/lib/ansible/module_utils/junos.py
--- a/lib/ansible/module_utils/junos.py
+++ b/lib/ansible/module_utils/junos.py
@@ -24,6 +24,7 @@
 from ansible.module_utils.netconf import send_request, children
 from ansible.module_utils.netconf import discard_changes, validate
 from ansible.module_utils.six import string_types
+from ansible.module_utils._text import to_text
 
 ACTIONS = frozenset(['merge', 'override', 'replace', 'update', 'set'])
 JSON_ACTIONS = frozenset(['merge', 'override', 'update'])
@@ -111,7 +112,7 @@ def load_configuration(module, candidate=None, action='merge', rollback=None, fo
             if format == 'xml':
                 cfg.append(fromstring(candidate))
             else:
-                cfg.text = candidate
+                cfg.text = to_text(candidate, encoding='latin1')
         else:
             cfg.append(candidate)
     return send_request(module, obj)
@@ -163,7 +164,7 @@ def get_diff(module):
     reply = get_configuration(module, compare=True, format='text')
     output = reply.find('.//configuration-output')
     if output is not None:
-        return output.text
+        return to_text(output.text, encoding='latin1').strip()
 
 def load_config(module, candidate, warnings, action='merge', commit=False, format='xml',
                 comment=None, confirm=False, confirm_timeout=None):
@@ -183,7 +184,6 @@ def load_config(module, candidate, warnings, action='merge', commit=False, forma
         diff = get_diff(module)
 
         if diff:
-            diff = str(diff).strip()
             if commit:
                 commit_configuration(module, confirm=confirm, comment=comment,
                                      confirm_timeout=confirm_timeout)
diff --git a/lib/ansible/modules/network/junos/junos_config.py b/lib/ansible/modules/network/junos/junos_config.py
--- a/lib/ansible/modules/network/junos/junos_config.py
+++ b/lib/ansible/modules/network/junos/junos_config.py
@@ -185,6 +185,7 @@
 from ansible.module_utils.junos import check_args as junos_check_args
 from ansible.module_utils.netconf import send_request
 from ansible.module_utils.six import string_types
+from ansible.module_utils._text import to_text, to_native
 
 if sys.version_info < (2, 7):
     from xml.parsers.expat import ExpatError
@@ -228,7 +229,7 @@ def filter_delete_statements(module, candidate):
     if match is None:
         # Could not find configuration-set in reply, perhaps device does not support it?
         return candidate
-    config = str(match.text)
+    config = to_native(match.text, encoding='latin1')
 
     modified_candidate = candidate[:]
     for index, line in reversed(list(enumerate(candidate))):
@@ -321,7 +322,7 @@ def main():
         else:
             module.fail_json(msg='unable to retrieve device configuration')
 
-        result['__backup__'] = str(match.text).strip()
+        result['__backup__'] = match.text.strip()
 
     if module.params['rollback']:
         if not module.check_mode:
diff --git a/lib/ansible/plugins/action/junos_config.py b/lib/ansible/plugins/action/junos_config.py
--- a/lib/ansible/plugins/action/junos_config.py
+++ b/lib/ansible/plugins/action/junos_config.py
@@ -27,6 +27,7 @@
 from ansible.plugins.action.junos import ActionModule as _ActionModule
 from ansible.module_utils._text import to_text
 from ansible.module_utils.six.moves.urllib.parse import urlsplit
+from ansible.module_utils._text import to_native
 from ansible.utils.vars import merge_hash
 
 PRIVATE_KEYS_RE = re.compile('__.+__')
@@ -74,7 +75,7 @@ def _write_backup(self, host, contents):
             os.remove(fn)
         tstamp = time.strftime("%Y-%m-%d@%H:%M:%S", time.localtime(time.time()))
         filename = '%s/%s_config.%s' % (backup_path, host, tstamp)
-        open(filename, 'w').write(contents)
+        open(filename, 'w').write(to_native(contents, encoding='latin1'))
         return filename
 
     def _handle_template(self):
@@ -110,4 +111,3 @@ def _handle_template(self):
         searchpath.append(os.path.dirname(source))
         self._templar.environment.loader.searchpath = searchpath
         self._task.args['src'] = self._templar.template(template_data)
-


Test patch: