Repository: StackStorm/st2

Issue:
CLI ERROR: 'unicodeescape' codec can't decode bytes in position 238-239: truncated \UXXXXXXXX escape
##### SUMMARY

Running a command from the vSphere pack that executes a command on a windows guest and returns a string is successful, and I can see it in the st2 GUI, but the command line response indicates a problem handling the data.

##### ISSUE TYPE
 - Bug Report

##### STACKSTORM VERSION
st2 3.0dev (5377ebb), on Python 2.7.6

##### OS / ENVIRONMENT / INSTALL METHOD
st2-docker with ST2_IMAGE_TAG=dev

##### STEPS TO REPRODUCE
Requires a working vSphere environment.
Install the vSphere pack with guest extensions from my branch:
```
# st2 pack install https://github.com/jeking3/stackstorm-vsphere.git=guestops
```
Run a command against a guest, for example:
```
root@b9bab223e4fa:/# st2 run vsphere.guest_dir_create vm_id=vm-45 username=Administrator password=******** prefix=issue_ suffix=_test
.
ERROR: 'unicodeescape' codec can't decode bytes in position 238-239: truncated \UXXXXXXXX escape
```

##### EXPECTED RESULTS
The action output in the GUI is reasonable:
```
{
  "result": "C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\jking_vmware20_test",
  "exit_code": 0,
  "stderr": "/opt/stackstorm/st2/lib/python2.7/site-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.\n  utils.DeprecatedIn23,\n",
  "stdout": ""
}
```
However the CLI choked.

##### ACTUAL RESULTS
```
ERROR: 'unicodeescape' codec can't decode bytes in position 238-239: truncated \UXXXXXXXX escape
```



Discussion:
What does `st2 --debug execution get <ID>` return?
_possibly_ related:
https://github.com/StackStorm/st2-docker/issues/129
https://github.com/StackStorm/st2/issues/4120
https://github.com/StackStorm/st2/pull/4127
The tail end with the stack trace is probably what you are looking for.  If you need more let me know.
```
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/shell.py", line 402, in run
    func(args)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/commands/resource.py", line 47, in decorate
    return func(*args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/commands/action.py", line 1229, in run_and_print
    return self._print_execution_details(execution=execution, args=args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/commands/action.py", line 366, in _print_execution_details
    self.print_output(instance, formatter, **options)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/commands/__init__.py", line 90, in print_output
    output = self.format_output(subject, formatter, *args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/commands/__init__.py", line 86, in format_output
    return func(subject, *args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/formatters/execution.py", line 122, in format
    return strutil.unescape(str(output)).decode('unicode_escape').encode('utf-8')
UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 251-252: truncated \UXXXXXXXX escape
```
I think it has something to do with backslashes.  I like the debug command - thanks. :)
In this case I'd like to see the whole debug output if possible. I'm specifically interested in seeing what the API returns, since here the exception is coming from the client, not the server
```
root@361bd439738c:/# st2 --debug execution get 5bbcd335668fd4010c01332e
2018-10-12 12:22:32,043  DEBUG - Using cached token from file "/root/.st2/token-st2admin"
# -------- begin 139909321114640 request ----------
curl -X GET -H  'Connection: keep-alive' -H  'Accept-Encoding: gzip, deflate' -H  'Accept: */*' -H  'User-Agent: python-requests/2.14.2' -H  'X-Auth-Token: fe38ee19f27a46d286a61d2f8c20e797' 'http://127.0.0.1:9101/v1/executions/5bbcd335668fd4010c01332e?exclude_attributes=trigger_instance'
# -------- begin 139909321114640 response ----------
{
    "status": "succeeded",
    "start_timestamp": "2018-10-09T16:11:33.664146Z",
    "log": [
        {
            "status": "requested",
            "timestamp": "2018-10-09T16:11:33.000000Z"
        },
        {
            "status": "scheduled",
            "timestamp": "2018-10-09T16:11:33.000000Z"
        },
        {
            "status": "running",
            "timestamp": "2018-10-09T16:11:33.000000Z"
        },
        {
            "status": "succeeded",
            "timestamp": "2018-10-09T16:11:35.000000Z"
        }
    ],
    "parameters": {
        "username": "Administrator",
        "prefix": "jking_",
        "password": "********",
        "vm_id": "vm-45",
        "suffix": "_test"
    },
    "runner": {
        "runner_module": "python_runner",
        "uid": "runner_type:python-script",
        "runner_package": "python_runner",
        "description": "A runner for launching python actions.",
        "enabled": true,
        "output_key": "result",
        "output_schema": {
            "result": {
                "anyOf": [
                    {
                        "type": "object"
                    },
                    {
                        "type": "string"
                    },
                    {
                        "type": "integer"
                    },
                    {
                        "type": "number"
                    },
                    {
                        "type": "boolean"
                    },
                    {
                        "type": "array"
                    },
                    {
                        "type": "null"
                    }
                ]
            },
            "exit_code": {
                "required": true,
                "type": "integer"
            },
            "stderr": {
                "required": true,
                "type": "string"
            },
            "stdout": {
                "required": true,
                "type": "string"
            }
        },
        "runner_parameters": {
            "debug": {
                "default": false,
                "required": false,
                "type": "boolean",
                "description": "Enable runner debug mode."
            },
            "content_version": {
                "required": false,
                "type": "string",
                "description": "Git revision of the pack content to use for this action execution (git commit sha / tag / branch). Only applies to packs which are git repositories."
            },
            "log_level": {
                "default": "DEBUG",
                "enum": [
                    "AUDIT",
                    "CRITICAL",
                    "ERROR",
                    "WARNING",
                    "INFO",
                    "DEBUG"
                ],
                "type": "string",
                "description": "Default log level for Python runner actions."
            },
            "timeout": {
                "default": 600,
                "type": "integer",
                "description": "Action timeout in seconds. Action will get killed if it doesn't finish in timeout seconds."
            },
            "env": {
                "type": "object",
                "description": "Environment variables which will be available to the script."
            }
        },
        "id": "5bacf15ae8db14010c9985d7",
        "name": "python-script"
    },
    "elapsed_seconds": 1.473827,
    "web_url": "https://f34287ba9aa8/#/history/5bbcd335668fd4010c01332e/general",
    "result": {
        "result": "C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\jking_vmware108_test",
        "exit_code": 0,
        "stderr": "/opt/stackstorm/st2/lib/python2.7/site-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.\n  utils.DeprecatedIn23,\n",
        "stdout": ""
    },
    "context": {
        "user": "st2admin",
        "pack": "vsphere"
    },
    "action": {
        "runner_type": "python-script",
        "name": "guest_dir_create",
        "parameters": {
            "username": {
                "position": 1,
                "required": true,
                "type": "string",
                "description": "Username within the guest to perform the action."
            },
            "vsphere": {
                "default": null,
                "position": 5,
                "required": false,
                "type": "string",
                "description": "Pre-configured vSphere connection details."
            },
            "suffix": {
                "position": 4,
                "required": true,
                "type": "string",
                "description": "Suffix for the directory name."
            },
            "prefix": {
                "position": 3,
                "required": true,
                "type": "string",
                "description": "Prefix for the directory name."
            },
            "password": {
                "position": 2,
                "secret": true,
                "required": true,
                "type": "string",
                "description": "Password for the given username."
            },
            "vm_id": {
                "position": 0,
                "required": true,
                "type": "string",
                "description": "VM to modify."
            }
        },
        "tags": [],
        "enabled": true,
        "entry_point": "guest_dir_create.py",
        "notify": {},
        "output_schema": {},
        "uid": "action:vsphere:guest_dir_create",
        "pack": "vsphere",
        "ref": "vsphere.guest_dir_create",
        "id": "5baede55b07a020e256cdb1b",
        "description": "Creates a temporary directory inside the guest."
    },
    "liveaction": {
        "runner_info": {
            "hostname": "f34287ba9aa8",
            "pid": 103
        },
        "parameters": {
            "username": "Administrator",
            "prefix": "jking_",
            "password": "********",
            "vm_id": "vm-45",
            "suffix": "_test"
        },
        "action_is_workflow": false,
        "callback": {},
        "action": "vsphere.guest_dir_create",
        "id": "5bbcd335668fd4010c01332d"
    },
    "id": "5bbcd335668fd4010c01332e",
    "end_timestamp": "2018-10-09T16:11:35.137973Z"
}
# -------- end 139909321114640 response ------------

ERROR: 'unicodeescape' codec can't decode bytes in position 251-252: truncated \UXXXXXXXX escape

CLI settings:
----------------
Config file path: /root/.st2/config
Client settings:
----------------
ST2_BASE_URL: http://127.0.0.1
ST2_AUTH_URL: http://127.0.0.1:9100
ST2_API_URL: http://127.0.0.1:9101/v1
ST2_STREAM_URL: http://127.0.0.1:9102/v1
ST2_AUTH_TOKEN: fe38ee19f27a46d286a61d2f8c20e797

Proxy settings:
---------------
HTTP_PROXY:
HTTPS_PROXY:

Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/shell.py", line 402, in run
    func(args)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/commands/resource.py", line 47, in decorate
    return func(*args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/commands/action.py", line 1229, in run_and_print
    return self._print_execution_details(execution=execution, args=args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/commands/action.py", line 366, in _print_execution_details
    self.print_output(instance, formatter, **options)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/commands/__init__.py", line 90, in print_output
    output = self.format_output(subject, formatter, *args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/commands/__init__.py", line 86, in format_output
    return func(subject, *args, **kwargs)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2client/formatters/execution.py", line 122, in format
    return strutil.unescape(str(output)).decode('unicode_escape').encode('utf-8')
UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 251-252: truncated \UXXXXXXXX escape
```
I can reproduce this issue. Will push a fix shortly.

Resolving patch:
diff --git a/st2client/st2client/formatters/execution.py b/st2client/st2client/formatters/execution.py
--- a/st2client/st2client/formatters/execution.py
+++ b/st2client/st2client/formatters/execution.py
@@ -119,4 +119,9 @@ def format(cls, entry, *args, **kwargs):
             return strutil.unescape(str(output))
         else:
             # Assume Python 2
-            return strutil.unescape(str(output)).decode('unicode_escape').encode('utf-8')
+            try:
+                result = strutil.unescape(str(output)).decode('unicode_escape').encode('utf-8')
+            except UnicodeDecodeError:
+                # String contains a value which is not an unicode escape sequence, ignore the error
+                result = strutil.unescape(str(output))
+            return result


Test patch:
diff --git a/st2client/tests/fixtures/execution_double_backslash.json b/st2client/tests/fixtures/execution_double_backslash.json
new file mode 100644
--- /dev/null
+++ b/st2client/tests/fixtures/execution_double_backslash.json
@@ -0,0 +1,27 @@
+{
+    "id": "547e19561e2e2417d3dde333",
+    "parameters": {
+        "cmd": "echo 'C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\jking_vmware20_test'"
+    },
+    "callback": {},
+    "context": {
+        "user": "stanley"
+    },
+    "result": {
+        "failed": false,
+        "stderr": "",
+        "return_code": 0,
+        "succeeded": true,
+        "stdout": "C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\jking_vmware20_test"
+    },
+    "status": "succeeded",
+    "start_timestamp": "2014-12-02T19:56:06.900000Z",
+    "end_timestamp": "2014-12-02T19:56:07.000000Z",
+    "action": {
+        "ref": "core.local"
+    },
+    "liveaction": {
+        "callback": {},
+        "id": "1"
+    }
+}
diff --git a/st2client/tests/fixtures/execution_double_backslash.txt b/st2client/tests/fixtures/execution_double_backslash.txt
new file mode 100644
--- /dev/null
+++ b/st2client/tests/fixtures/execution_double_backslash.txt
@@ -0,0 +1,10 @@
+id: 547e19561e2e2417d3dde333
+status: succeeded (1s elapsed)
+parameters: 
+  cmd: echo 'C:\Users\ADMINI~1\AppData\Local\Temp\jking_vmware20_test'
+result: 
+  failed: false
+  return_code: 0
+  stderr: ''
+  stdout: C:\Users\ADMINI~1\AppData\Local\Temp\jking_vmware20_test
+  succeeded: true
diff --git a/st2client/tests/unit/test_formatters.py b/st2client/tests/unit/test_formatters.py
--- a/st2client/tests/unit/test_formatters.py
+++ b/st2client/tests/unit/test_formatters.py
@@ -42,6 +42,7 @@
     'executions': ['execution.json',
                    'execution_result_has_carriage_return.json',
                    'execution_unicode.json',
+                   'execution_double_backslash.json',
                    'execution_with_stack_trace.json',
                    'execution_with_schema.json'],
     'results': ['execution_get_default.txt',
@@ -54,6 +55,7 @@
                 'execution_list_empty_response_start_timestamp_attr.txt',
                 'execution_unescape_newline.txt',
                 'execution_unicode.txt',
+                'execution_double_backslash.txt',
                 'execution_unicode_py3.txt',
                 'execution_get_has_schema.txt']
 }
@@ -61,6 +63,7 @@
 FIXTURES = loader.load_fixtures(fixtures_dict=FIXTURES_MANIFEST)
 EXECUTION = FIXTURES['executions']['execution.json']
 UNICODE = FIXTURES['executions']['execution_unicode.json']
+DOUBLE_BACKSLASH = FIXTURES['executions']['execution_double_backslash.json']
 OUTPUT_SCHEMA = FIXTURES['executions']['execution_with_schema.json']
 NEWLINE = FIXTURES['executions']['execution_with_stack_trace.json']
 HAS_CARRIAGE_RETURN = FIXTURES['executions']['execution_result_has_carriage_return.json']
@@ -160,6 +163,18 @@ def test_execution_unicode(self):
             content = content.replace(r'\xE2\x80\xA1', r'\u2021')
             self.assertEqual(content, FIXTURES['results']['execution_unicode_py3.txt'])
 
+    @mock.patch.object(
+        httpclient.HTTPClient, 'get',
+        mock.MagicMock(return_value=base.FakeResponse(json.dumps(DOUBLE_BACKSLASH), 200, 'OK', {})))
+    def test_execution_double_backslash_not_unicode_escape_sequence(self):
+        argv = ['execution', 'get', DOUBLE_BACKSLASH['id']]
+        self.assertEqual(self.shell.run(argv), 0)
+        self._undo_console_redirect()
+        with open(self.path, 'r') as fd:
+            content = fd.read()
+
+        self.assertEqual(content, FIXTURES['results']['execution_double_backslash.txt'])
+
     def test_execution_get_detail_in_json(self):
         argv = ['execution', 'get', EXECUTION['id'], '-d', '-j']
         content = self._get_execution(argv)