Repository: ansible/ansible

Issue:
ios_facts: rawunicodeescape codec can't decode bytes in position 4092-4093: truncated \\uXXXX"} with gather_subset: config
<!---
Verify first that your issue/request is not already reported on GitHub.
THIS FORM WILL BE READ BY A MACHINE, COMPLETE ALL SECTIONS AS DESCRIBED.
Also test if the latest release, and devel branch are affected too.
ALWAYS add information AFTER (OUTSIDE) these html comments.
Otherwise it may end up being automatically closed by our bot. -->

##### SUMMARY
`ios_facts` with `gather_subset: config` is returning `{"msg": "'rawunicodeescape' codec can't decode bytes in position 4092-4093: truncated \\uXXXX"}`

##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
<!--- Insert, BELOW THIS COMMENT, the name of the module, plugin, task or feature.
Do not include extra details here, e.g. "vyos_command" not "the network module vyos_command" or the full path-->
ios_facts

##### ANSIBLE VERSION
<!--- Paste, BELOW THIS COMMENT, verbatim output from "ansible --version" between quotes below -->

Tested with 2.6.x and 2.7 devel.

```
ansible 2.6.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/johni/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]


ansible 2.7.0.dev0 (pip_umask_doc 9169daa326) last updated 2018/08/01 22:52:09 (GMT +1100)
  config file = /home/johni/upgrade_ios/ansible.cfg
  configured module search path = [u'/home/johni/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/johni/ansible/hacking/ansible/lib/ansible
  executable location = /home/johni/ansible/hacking/ansible/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

```

##### CONFIGURATION
<!--- If using Ansible 2.4 or above, paste, BELOW THIS COMMENT, the results of "ansible-config dump --only-changed"
Otherwise, mention any settings you have changed/added/removed in ansible.cfg
(or using the ANSIBLE_* environment variables).-->

```
DEFAULT_HOST_LIST(/home/johni/upgrade_ios/ansible.cfg) = [u'/home/johni/upgrade_ios/inventory']

```

##### OS / ENVIRONMENT
<!--- Mention, BELOW THIS COMMENT, the OS you are running Ansible from, and the OS you are
managing, or say "N/A" for anything that is not platform-specific.
Also mention the specific version of what you are trying to control,
e.g. if this is a network bug the version of firmware on the network device.-->

* Ubuntu 16.04 LTS (4) and (5)
* RHEL 7

##### STEPS TO REPRODUCE
<!--- For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used. -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: "Demonstrate connecting to switches"
  hosts: switches
  gather_facts: no

  tasks:
    - name: Gather facts (ops)
      ios_facts:
        gather_subset: config
      when: ansible_network_os == 'ios'
    - name: Display some facts
      debug:
        msg: "The hostname is {{ ansible_net_hostname }} and the OS is {{ ansible_net_version }}"
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->

Would expect to see the hostname and ansible_net_version.  If removing the config from collection:

```
TASK [Display some facts] ****************************************************************************************
ok: [sw1] => {
    "msg": "The hostname is dclabsw03 and the OS is 15.0(2)SE11"
}

```

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->

<!--- Paste verbatim command output between quotes below -->
```
TASK [Display some facts] ****************************************************************************************
task path: /home/johni/upgrade_ios/test.yml:13
<192.168.0.222> attempting to start connection
<192.168.0.222> using connection plugin network_cli
<192.168.0.222>
The full traceback is:
Traceback (most recent call last):
  File "/home/johni/ansible/hacking/ansible/bin/ansible-connection", line 225, in main
    variables = cPickle.loads(vars_data)
UnicodeDecodeError: 'rawunicodeescape' codec can't decode bytes in position 4092-4093: truncated \uXXXX

fatal: [sw1]: FAILED! => {
    "msg": "'rawunicodeescape' codec can't decode bytes in position 4092-4093: truncated \\uXXXX"
}

```



Discussion:
More information.  The switch had crypto pki configuration on it.  Once removed the self signed certificates this `rawunicodeescape` problem went away.

Eg something like:
```
crypto pki trustpoint TP-self-signed-1116070172
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-1116070172
 revocation-check none
 rsakeypair TP-self-signed-1116070172


crypto pki certificate chain TP-self-signed-1116070172
 certificate self-signed 01
  1112022B 11120194 11110201 02110101 300D0609 2A864886 F1110101 05050030
  31312F30 2D060355 04031326 411F532D 11111C66 2D511967 6E65642D 43657274
... omitted ...
  4F20AB3E 523040BC BDBFCDF3 DEDC8860 5CEB96BB 00FA5220 64311759 2D6E3ED0
  FDBEEEBC 2B49E470 FC6D1789 EC58AE
        quit
```

Not a valid solution. 😃 
cc @privateip @rcarrillocruz
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/executor/task_executor.py b/lib/ansible/executor/task_executor.py
--- a/lib/ansible/executor/task_executor.py
+++ b/lib/ansible/executor/task_executor.py
@@ -10,14 +10,15 @@
 import json
 import subprocess
 import sys
+import termios
 import traceback
 
 from ansible import constants as C
 from ansible.errors import AnsibleError, AnsibleParserError, AnsibleUndefinedVariable, AnsibleConnectionFailure, AnsibleActionFail, AnsibleActionSkip
 from ansible.executor.task_result import TaskResult
 from ansible.module_utils.six import iteritems, string_types, binary_type
-from ansible.module_utils.six.moves import cPickle
 from ansible.module_utils._text import to_text, to_native
+from ansible.module_utils.connection import write_to_file_descriptor
 from ansible.playbook.conditional import Conditional
 from ansible.playbook.task import Task
 from ansible.template import Templar
@@ -920,28 +921,24 @@ def find_file_in_path(filename):
             [python, find_file_in_path('ansible-connection'), to_text(os.getppid())],
             stdin=slave, stdout=subprocess.PIPE, stderr=subprocess.PIPE
         )
-        stdin = os.fdopen(master, 'wb', 0)
         os.close(slave)
 
-        # Need to force a protocol that is compatible with both py2 and py3.
-        # That would be protocol=2 or less.
-        # Also need to force a protocol that excludes certain control chars as
-        # stdin in this case is a pty and control chars will cause problems.
-        # that means only protocol=0 will work.
-        src = cPickle.dumps(self._play_context.serialize(), protocol=0)
-        stdin.write(src)
-        stdin.write(b'\n#END_INIT#\n')
-
-        src = cPickle.dumps(variables, protocol=0)
-        # remaining \r fail to round-trip the socket
-        src = src.replace(b'\r', br'\r')
-        stdin.write(src)
-        stdin.write(b'\n#END_VARS#\n')
-
-        stdin.flush()
-
-        (stdout, stderr) = p.communicate()
-        stdin.close()
+        # We need to set the pty into noncanonical mode. This ensures that we
+        # can receive lines longer than 4095 characters (plus newline) without
+        # truncating.
+        old = termios.tcgetattr(master)
+        new = termios.tcgetattr(master)
+        new[3] = new[3] & ~termios.ICANON
+
+        try:
+            termios.tcsetattr(master, termios.TCSANOW, new)
+            write_to_file_descriptor(master, variables)
+            write_to_file_descriptor(master, self._play_context.serialize())
+
+            (stdout, stderr) = p.communicate()
+        finally:
+            termios.tcsetattr(master, termios.TCSANOW, old)
+        os.close(master)
 
         if p.returncode == 0:
             result = json.loads(to_text(stdout, errors='surrogate_then_replace'))
diff --git a/lib/ansible/module_utils/connection.py b/lib/ansible/module_utils/connection.py
--- a/lib/ansible/module_utils/connection.py
+++ b/lib/ansible/module_utils/connection.py
@@ -27,6 +27,7 @@
 # USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
 import os
+import hashlib
 import json
 import socket
 import struct
@@ -36,6 +37,30 @@
 from functools import partial
 from ansible.module_utils._text import to_bytes, to_text
 from ansible.module_utils.six import iteritems
+from ansible.module_utils.six.moves import cPickle
+
+
+def write_to_file_descriptor(fd, obj):
+    """Handles making sure all data is properly written to file descriptor fd.
+
+    In particular, that data is encoded in a character stream-friendly way and
+    that all data gets written before returning.
+    """
+    # Need to force a protocol that is compatible with both py2 and py3.
+    # That would be protocol=2 or less.
+    # Also need to force a protocol that excludes certain control chars as
+    # stdin in this case is a pty and control chars will cause problems.
+    # that means only protocol=0 will work.
+    src = cPickle.dumps(obj, protocol=0)
+
+    # raw \r characters will not survive pty round-trip
+    # They should be rehydrated on the receiving end
+    src = src.replace(b'\r', br'\r')
+    data_hash = to_bytes(hashlib.sha1(src).hexdigest())
+
+    os.write(fd, b'%d\n' % len(src))
+    os.write(fd, src)
+    os.write(fd, b'%s\n' % data_hash)
 
 
 def send_data(s, data):
diff --git a/lib/ansible/plugins/connection/persistent.py b/lib/ansible/plugins/connection/persistent.py
--- a/lib/ansible/plugins/connection/persistent.py
+++ b/lib/ansible/plugins/connection/persistent.py
@@ -34,12 +34,12 @@
 import json
 import subprocess
 import sys
+import termios
 
 from ansible import constants as C
 from ansible.plugins.connection import ConnectionBase
 from ansible.module_utils._text import to_text
-from ansible.module_utils.six.moves import cPickle
-from ansible.module_utils.connection import Connection as SocketConnection
+from ansible.module_utils.connection import Connection as SocketConnection, write_to_file_descriptor
 from ansible.errors import AnsibleError
 
 try:
@@ -109,26 +109,24 @@ def find_file_in_path(filename):
             [python, find_file_in_path('ansible-connection'), to_text(os.getppid())],
             stdin=slave, stdout=subprocess.PIPE, stderr=subprocess.PIPE
         )
-        stdin = os.fdopen(master, 'wb', 0)
         os.close(slave)
 
-        # Need to force a protocol that is compatible with both py2 and py3.
-        # That would be protocol=2 or less.
-        # Also need to force a protocol that excludes certain control chars as
-        # stdin in this case is a pty and control chars will cause problems.
-        # that means only protocol=0 will work.
-        src = cPickle.dumps(self._play_context.serialize(), protocol=0)
-        stdin.write(src)
-        stdin.write(b'\n#END_INIT#\n')
-
-        src = cPickle.dumps({'ansible_command_timeout': self.get_option('persistent_command_timeout')}, protocol=0)
-        stdin.write(src)
-        stdin.write(b'\n#END_VARS#\n')
-
-        stdin.flush()
-
-        (stdout, stderr) = p.communicate()
-        stdin.close()
+        # We need to set the pty into noncanonical mode. This ensures that we
+        # can receive lines longer than 4095 characters (plus newline) without
+        # truncating.
+        old = termios.tcgetattr(master)
+        new = termios.tcgetattr(master)
+        new[3] = new[3] & ~termios.ICANON
+
+        try:
+            termios.tcsetattr(master, termios.TCSANOW, new)
+            write_to_file_descriptor(master, {'ansible_command_timeout': self.get_option('persistent_command_timeout')})
+            write_to_file_descriptor(master, self._play_context.serialize())
+
+            (stdout, stderr) = p.communicate()
+        finally:
+            termios.tcsetattr(master, termios.TCSANOW, old)
+        os.close(master)
 
         if p.returncode == 0:
             result = json.loads(to_text(stdout, errors='surrogate_then_replace'))


Test patch: