Repository: StackStorm/st2

Issue:
Orquesta - Unable to serialize list objects
##### SUMMARY

When running an Orquesta workflow and trying to use the `to_yaml_string` filter with a list object, an error is raised even though the object is valid JSON.

##### ISSUE TYPE
 - Bug Report


##### STACKSTORM VERSION

```
$ st2 --version
st2 3.0.0, on Python 2.7.5
```

##### OS / ENVIRONMENT / INSTALL METHOD
```
OS = Centos 7.6
Install = puppet-st2
```


##### STEPS TO REPRODUCE
`/opt/stackstorm/packs/default/actions/orquesta_list_yaml_error.yaml`
```yaml
---
name: orquesta_list_yaml_error
description: A basic sequential workflow.
runner_type: orquesta
entry_point: workflows/orquesta_list_yaml_error.yaml
enabled: true
parameters: {}
```

`/opt/stackstorm/packs/default/actions/workflows/orquesta_list_yaml_error.yaml`
```yaml
version: 1.0
  
description: A basic sequential workflow.

vars:
  - blah:
      - a
      - b
      - c

tasks:
  demo:
    action: core.local
    input:
      cmd: "echo '{{ ctx().blah | to_yaml_string }}'"
```

##### EXPECTED RESULTS

Workflow runs successfully

##### ACTUAL RESULTS

```
$ st2 run default.orquesta_list_yaml_error
.
id: 5cf5df6b9387ef5a16b63f71
action.ref: default.orquesta_list_yaml_error
parameters: None
status: failed
start_timestamp: Tue, 04 Jun 2019 03:03:07 UTC
end_timestamp: Tue, 04 Jun 2019 03:03:07 UTC
result: 
  errors:
  - message: 'JinjaEvaluationException: Unable to evaluate expression ''{{ ctx().blah | to_yaml_string }}''. RepresenterError: (''cannot represent an object'', [u''a'', u''b'', u''c''])'
    route: 0
    task_id: demo                                                                                                            
    type: error                                                                                                              
  output: null 
```



Discussion:
After doing LOTS of digging (had to add debug statements to the YAML module) that the data type of the list is not actually a `list` but instead a `mongoengine.base.datastructures.BaseList`. For some reason these lists aren't be deserialized properly.

```
$ st2 run default.orquesta_list_yaml_error                                                        
.                                                                                                                            
id: 5cf5df929387ef5a16b63f74                                                                                                 
action.ref: default.orquesta_list_yaml_error                                                                                 
parameters: None                                                                                                             
status: failed                                                                                                               
start_timestamp: Tue, 04 Jun 2019 03:03:46 UTC                                                                               
end_timestamp: Tue, 04 Jun 2019 03:03:46 UTC                                                                                 
result:                                                                                                                      
  errors:                                                                                                                    
  - message: 'JinjaEvaluationException: Unable to evaluate expression ''{{ ctx().blah | to_yaml_string }}''. RepresenterError: ("cannot represent an object data is type=<class ''mongoengine.base.datastructures.BaseList''>", [u''a'', u''b'', u''c''])'
    route: 0
    task_id: demo
    type: error
  output: null
```

I added the debug statement to `/opt/stackstorm/st2/lib/python2.7/site-packages/yaml/representer.py` line 251

```
    def represent_undefined(self, data):
        raise RepresenterError("cannot represent an object data is type={}".format(type(data)), data)
```


@m4dcoder as another helpful thing, it would be great if errors raised during execution  (or even Jinja rendering) actually returned stack traces or at least printed them in the log files, it took me forever to hunt down where this was being thrown from
@nmaludy Please open a separate issue on logging stack traces. This needs to be addressed at the orquesta library and workflow service. We should not mix it w/ this bug fix.

Resolving patch:
diff --git a/st2common/st2common/expressions/functions/data.py b/st2common/st2common/expressions/functions/data.py
--- a/st2common/st2common/expressions/functions/data.py
+++ b/st2common/st2common/expressions/functions/data.py
@@ -16,10 +16,11 @@
 
 import json
 import jsonpath_rw
-import mongoengine
 import six
 import yaml
 
+from st2common.util import db as db_util
+
 
 __all__ = [
     'from_json_string',
@@ -56,8 +57,7 @@ def to_json_string(value, indent=None, sort_keys=False, separators=(',', ': ')):
 
 
 def to_yaml_string(value, indent=None, allow_unicode=True):
-    if isinstance(value, mongoengine.base.datastructures.BaseDict):
-        value = dict(value)
+    value = db_util.mongodb_to_python_types(value)
 
     options = {'default_flow_style': False}
 
diff --git a/st2common/st2common/util/db.py b/st2common/st2common/util/db.py
new file mode 100644
--- /dev/null
+++ b/st2common/st2common/util/db.py
@@ -0,0 +1,34 @@
+# Copyright 2019 Extreme Networks, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+
+import mongoengine
+import six
+
+
+def mongodb_to_python_types(value):
+    # Convert MongoDB BaseDict and BaseList types to python dict and list types.
+    if isinstance(value, mongoengine.base.datastructures.BaseDict):
+        value = dict(value)
+    elif isinstance(value, mongoengine.base.datastructures.BaseList):
+        value = list(value)
+
+    # Recursively traverse the dict and list to convert values.
+    if isinstance(value, dict):
+        value = {k: mongodb_to_python_types(v) for k, v in six.iteritems(value)}
+    elif isinstance(value, list):
+        value = [mongodb_to_python_types(v) for v in value]
+
+    return value


Test patch:
diff --git a/contrib/examples/actions/workflows/tests/orquesta-test-jinja-data-functions.yaml b/contrib/examples/actions/workflows/tests/orquesta-test-jinja-data-functions.yaml
--- a/contrib/examples/actions/workflows/tests/orquesta-test-jinja-data-functions.yaml
+++ b/contrib/examples/actions/workflows/tests/orquesta-test-jinja-data-functions.yaml
@@ -5,6 +5,11 @@ description: A basic workflow testing data related functions.
 vars:
   - data_json_str_1: '{"foo": {"bar": "foobar"}}'
   - none: null
+  - data_list:
+    - a: 1
+      b: 2
+    - x: 3
+      y: 4
 
 tasks:
   task1:
@@ -30,3 +35,4 @@ output:
   - data_json_str_3: '{{ json_dump(ctx("data_json_obj_4")) }}'
   - data_none_str: '{{ use_none(ctx("none")) }}'
   - data_str: '{{ use_none("foobar") }}'
+  - data_list_str: '{{ to_yaml_string(ctx("data_list")) }}'
diff --git a/contrib/runners/orquesta_runner/tests/unit/test_functions_common.py b/contrib/runners/orquesta_runner/tests/unit/test_functions_common.py
--- a/contrib/runners/orquesta_runner/tests/unit/test_functions_common.py
+++ b/contrib/runners/orquesta_runner/tests/unit/test_functions_common.py
@@ -163,7 +163,8 @@ def test_data_functions_in_jinja(self):
             'data_query_1': ['foobar'],
             'data_pipe_str_1': '{"foo": {"bar": "foobar"}}',
             'data_none_str': data_funcs.NONE_MAGIC_VALUE,
-            'data_str': 'foobar'
+            'data_str': 'foobar',
+            'data_list_str': '- a: 1\n  b: 2\n- x: 3\n  y: 4\n'
         }
 
         self._execute_workflow(wf_name, expected_output)
diff --git a/st2common/tests/unit/test_util_db.py b/st2common/tests/unit/test_util_db.py
new file mode 100644
--- /dev/null
+++ b/st2common/tests/unit/test_util_db.py
@@ -0,0 +1,108 @@
+# Copyright 2019 Extreme Networks, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+
+import mongoengine
+import unittest2
+
+from st2common.util import db as db_util
+
+
+class DatabaseUtilTestCase(unittest2.TestCase):
+
+    def test_noop_mongodb_to_python_types(self):
+        data = [
+            123,
+            999.99,
+            True,
+            [10, 20, 30],
+            {'a': 1, 'b': 2},
+            None
+        ]
+
+        for item in data:
+            self.assertEqual(db_util.mongodb_to_python_types(item), item)
+
+    def test_mongodb_basedict_to_dict(self):
+        data = {'a': 1, 'b': 2}
+
+        obj = mongoengine.base.datastructures.BaseDict(data, None, 'foobar')
+
+        self.assertDictEqual(db_util.mongodb_to_python_types(obj), data)
+
+    def test_mongodb_baselist_to_list(self):
+        data = [2, 4, 6]
+
+        obj = mongoengine.base.datastructures.BaseList(data, None, 'foobar')
+
+        self.assertListEqual(db_util.mongodb_to_python_types(obj), data)
+
+    def test_nested_mongdb_to_python_types(self):
+        data = {
+            'a': mongoengine.base.datastructures.BaseList([1, 2, 3], None, 'a'),
+            'b': mongoengine.base.datastructures.BaseDict({'a': 1, 'b': 2}, None, 'b'),
+            'c': {
+                'd': mongoengine.base.datastructures.BaseList([4, 5, 6], None, 'd'),
+                'e': mongoengine.base.datastructures.BaseDict({'c': 3, 'd': 4}, None, 'e')
+            },
+            'f': mongoengine.base.datastructures.BaseList(
+                [
+                    mongoengine.base.datastructures.BaseDict({'e': 5}, None, 'f1'),
+                    mongoengine.base.datastructures.BaseDict({'f': 6}, None, 'f2')
+                ],
+                None,
+                'f'
+            ),
+            'g': mongoengine.base.datastructures.BaseDict(
+                {
+                    'h': mongoengine.base.datastructures.BaseList(
+                        [
+                            mongoengine.base.datastructures.BaseDict({'g': 7}, None, 'h1'),
+                            mongoengine.base.datastructures.BaseDict({'h': 8}, None, 'h2')
+                        ],
+                        None,
+                        'h'
+                    ),
+                    'i': mongoengine.base.datastructures.BaseDict({'j': 9, 'k': 10}, None, 'i')
+                },
+                None,
+                'g'
+            ),
+        }
+
+        expected = {
+            'a': [1, 2, 3],
+            'b': {'a': 1, 'b': 2},
+            'c': {
+                'd': [4, 5, 6],
+                'e': {'c': 3, 'd': 4}
+            },
+            'f': [
+                {'e': 5},
+                {'f': 6}
+            ],
+            'g': {
+                'h': [
+                    {'g': 7},
+                    {'h': 8}
+                ],
+                'i': {
+                    'j': 9,
+                    'k': 10
+                }
+            }
+        }
+
+        self.assertDictEqual(db_util.mongodb_to_python_types(data), expected)
diff --git a/st2tests/integration/orquesta/test_wiring_functions.py b/st2tests/integration/orquesta/test_wiring_functions.py
--- a/st2tests/integration/orquesta/test_wiring_functions.py
+++ b/st2tests/integration/orquesta/test_wiring_functions.py
@@ -57,7 +57,8 @@ def test_data_functions_in_jinja(self):
             'data_query_1': ['foobar'],
             'data_pipe_str_1': '{"foo": {"bar": "foobar"}}',
             'data_none_str': '%*****__%NONE%__*****%',
-            'data_str': 'foobar'
+            'data_str': 'foobar',
+            'data_list_str': '- a: 1\n  b: 2\n- x: 3\n  y: 4\n'
         }
 
         expected_result = {'output': expected_output}
diff --git a/st2tests/st2tests/fixtures/packs/orquesta_tests/actions/workflows/jinja-data-functions.yaml b/st2tests/st2tests/fixtures/packs/orquesta_tests/actions/workflows/jinja-data-functions.yaml
--- a/st2tests/st2tests/fixtures/packs/orquesta_tests/actions/workflows/jinja-data-functions.yaml
+++ b/st2tests/st2tests/fixtures/packs/orquesta_tests/actions/workflows/jinja-data-functions.yaml
@@ -5,6 +5,11 @@ description: A basic workflow testing data related functions.
 vars:
   - data_json_str_1: '{"foo": {"bar": "foobar"}}'
   - none: null
+  - data_list:
+    - a: 1
+      b: 2
+    - x: 3
+      y: 4
 
 tasks:
   task1:
@@ -30,3 +35,4 @@ output:
   - data_json_str_3: '{{ json_dump(ctx("data_json_obj_4")) }}'
   - data_none_str: '{{ use_none(ctx("none")) }}'
   - data_str: '{{ use_none("foobar") }}'
+  - data_list_str: '{{ to_yaml_string(ctx("data_list")) }}'