Repository: StackStorm/st2

Issue:
Orquesta vars can no longer contain '.' in 3.2
## SUMMARY

In StackStorm 3.2, if an Orquesta workflow contains a `.` in any of its `vars` (ie context) then the workflow will fail to run with an error:
```shell
$ st2 run default.orquesta_vars_with_periods.
id: 5eac3faeca07c849e9b4f280
action.ref: default.orquesta_vars_with_periods
parameters: None
status: failed
start_timestamp: Fri, 01 May 2020 15:26:38 UTC
end_timestamp: Fri, 01 May 2020 15:26:38 UTC
result: 
  errors:
  - message: key 'with.period' must not contain '.'
  output: null
```

### STACKSTORM VERSION

```shell
$ st2 --version
st2 3.2.0, on Python 2.7.5
```

##### OS, environment, install method

```shell
$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.8 (Maipo)

# install method = puppet-st2
```

## Steps to reproduce the problem

/opt/stackstorm/packs/default/actions/orquesta_vars_with_periods.yaml 
```yaml
---
description: "Demos bug with orquesta vars with periods"
enabled: true
runner_type: orquesta
entry_point: workflows/orquesta_vars_with_periods.yaml
name: orquesta_vars_with_periods
pack: default
parameters:
  demo:
    type: string
    default: "demo"
```

/opt/stackstorm/packs/default/actions/workflows/orquesta_vars_with_periods.yaml 
```
---
version: 1.0

description: "Demos bug with orquesta vars with periods"

input:
  - demo

vars:
  - myvar.with.period: 'this line is going to cause an error'

tasks:
  dispatch:
    action: core.noop
```

Register and run the action
```shell
$ st2 action create /opt/stackstorm/packs/default/actions/orquesta_vars_with_periods.yaml 
$ st2 run default.orquesta_vars_with_periods
.
id: 5eac47b9ca07c849e9b4f289
action.ref: default.orquesta_vars_with_periods
parameters: None
status: failed
start_timestamp: Fri, 01 May 2020 16:00:57 UTC
end_timestamp: Fri, 01 May 2020 16:00:58 UTC
result: 
  errors:
  - message: key 'myvar.with.period' must not contain '.'
  output: null
```

## Expected Results

Workflow runs without error

## Actual Results

Workflow errors trying to write state to the database



Discussion:
**Investigation**

Rolled back to `v3.1` and this same workflow executes successfully.

Looks like this problem was introduced in this PR: https://github.com/StackStorm/st2/pull/4854

It appears there were some other problems with `'.'` in different places for workflows and subsequent rollback PRs were introduced to fix those bugs:
* https://github.com/StackStorm/st2/pull/4875
* https://github.com/StackStorm/st2/pull/4892

Diffing from 3.1 to 3.2 i can see the following fields were changed:
```shell
$ git diff v3.1.0 v3.2.0 -- st2common/st2common/models/db/workflow.py
diff --git a/st2common/st2common/models/db/workflow.py b/st2common/st2common/models/db/workflow.py
index b93b40686..b642df8eb 100644
--- a/st2common/st2common/models/db/workflow.py
+++ b/st2common/st2common/models/db/workflow.py
@@ -36,11 +36,11 @@ class WorkflowExecutionDB(stormbase.StormFoundationDB, stormbase.ChangeRevisionF
     RESOURCE_TYPE = types.ResourceType.EXECUTION     action_execution = me.StringField(required=True)
-    spec = stormbase.EscapedDictField()
-    graph = stormbase.EscapedDictField()
+    spec = me.DictField()
+    graph = me.DictField()
     input = stormbase.EscapedDictField()
-    notify = stormbase.EscapedDictField()
-    context = stormbase.EscapedDictField()
+    notify = me.DictField()
+    context = me.DictField()
     state = stormbase.EscapedDictField()
     status = me.StringField(required=True)
     output = stormbase.EscapedDictField()
@@ -62,7 +62,7 @@ class TaskExecutionDB(stormbase.StormFoundationDB, stormbase.ChangeRevisionField
     task_name = me.StringField(required=True)
     task_id = me.StringField(required=True)
     task_route = me.IntField(required=True, min_value=0)
-    task_spec = stormbase.EscapedDictField()
+    task_spec = me.DictField()
     delay = me.IntField(min_value=0)
     itemized = me.BooleanField(default=False)
     items_count = me.IntField(min_value=0)
```

Not yet sure which of these fields need to be reverted back to `stormbase.EscapedDictField()` in order to fully fix the issue. Working on this now.
Do you use an IP address or domain name as a var name? Why do you have a period in a var name?
@cognifloyd in this case we have keys that represent datastore keys that we pass to `bolt` so it can do an `st2kv` lookup from within the Bolt process in order to securely transfer secrets from st2 to bolt.

example:
```yaml
vars:
  - attempt_count: 0
  - st2kv_config_linux:
      ssh.password: system.linux.password
      ssh.user: system.linux.username
      ssh.tmpdir: system.linux.tmpdir
  - st2kv_config_windows:
      winrm.password: system.ad.password
      winrm.user: system.ad.username
      winrm.port: system.winrm.port
```

Although, yes, it would be the same problem for a key being a domain name or IP address.
If we drop mongo 3.4 and use mongoengine >=0.19.1, then we don't have to use the EscapedDictField() to have periods in keys. That won't work for now (ie dropping mongo 3.4 in a bugfix release is a no-no), but that means we can have hope of re-adding these performance fixes in a future release.
See: MongoEngine/mongoengine#1594 and https://github.com/MongoEngine/mongoengine/pull/2193 (merged in https://github.com/MongoEngine/mongoengine/commit/b2053144241f8f579993eff2c4de3084f074535f )
There are plans to drop MongoDB 3.4 in favor of 3.6 in next release:
See
- https://github.com/StackStorm/st2-packages/issues/643
- https://github.com/StackStorm/st2docs/pull/968/files

Resolving patch:
diff --git a/contrib/examples/actions/python-mock-core-remote.py b/contrib/examples/actions/python-mock-core-remote.py
new file mode 100644
--- /dev/null
+++ b/contrib/examples/actions/python-mock-core-remote.py
@@ -0,0 +1,23 @@
+from st2common.runners.base_action import Action
+
+
+class MockCoreRemoteAction(Action):
+
+    def run(self, cmd, hosts, hosts_dict):
+        if hosts_dict:
+            return hosts_dict
+
+        if not hosts:
+            return None
+
+        host_list = hosts.split(',')
+        results = {}
+        for h in hosts:
+            results[h] = {
+                'failed': False,
+                'return_code': 0,
+                'stderr': '',
+                'succeeded': True,
+                'stdout': cmd,
+            }
+        return results
diff --git a/st2common/st2common/models/db/workflow.py b/st2common/st2common/models/db/workflow.py
--- a/st2common/st2common/models/db/workflow.py
+++ b/st2common/st2common/models/db/workflow.py
@@ -36,11 +36,11 @@ class WorkflowExecutionDB(stormbase.StormFoundationDB, stormbase.ChangeRevisionF
     RESOURCE_TYPE = types.ResourceType.EXECUTION
 
     action_execution = me.StringField(required=True)
-    spec = me.DictField()
+    spec = stormbase.EscapedDictField()
     graph = me.DictField()
     input = stormbase.EscapedDictField()
     notify = me.DictField()
-    context = me.DictField()
+    context = stormbase.EscapedDictField()
     state = stormbase.EscapedDictField()
     status = me.StringField(required=True)
     output = stormbase.EscapedDictField()
@@ -62,7 +62,7 @@ class TaskExecutionDB(stormbase.StormFoundationDB, stormbase.ChangeRevisionField
     task_name = me.StringField(required=True)
     task_id = me.StringField(required=True)
     task_route = me.IntField(required=True, min_value=0)
-    task_spec = me.DictField()
+    task_spec = stormbase.EscapedDictField()
     delay = me.IntField(min_value=0)
     itemized = me.BooleanField(default=False)
     items_count = me.IntField(min_value=0)


Test patch:
diff --git a/contrib/examples/actions/workflows/tests/orquesta-test-field-escaping.yaml b/contrib/examples/actions/workflows/tests/orquesta-test-field-escaping.yaml
new file mode 100644
--- /dev/null
+++ b/contrib/examples/actions/workflows/tests/orquesta-test-field-escaping.yaml
@@ -0,0 +1,26 @@
+version: 1.0
+
+description: A workflow used to test escaping of things like vars, task names, results, outputs, etc.
+
+vars:
+  - vars.key.with.periods: "vars.value.with.periods"
+  - vars.nested.with.periods:
+      nested.periods: "vars.nested.value.with.periods"
+
+tasks:
+  run:
+    action: examples.python-mock-core-remote
+    input:
+      hosts_dict:
+        hostname.domain.tld: "{{ ctx()['vars.key.with.periods'] }}"
+        hostname2.domain.tld:
+          stdout: "{{ ctx()['vars.nested.with.periods']['nested.periods'] }}"
+    next:
+      - when: "{{ succeeded() }}"
+        publish:
+          - published.hosts: "{{ result().result }}"
+          - published.field: "{{ result().result['hostname2.domain.tld']['stdout'] }}"
+
+output:
+  - wf.hostname.with.periods: "{{ ctx()['published.hosts'] }}"
+  - wf.output.with.periods: "{{ ctx()['published.field'] }}"
diff --git a/st2tests/integration/orquesta/test_wiring.py b/st2tests/integration/orquesta/test_wiring.py
--- a/st2tests/integration/orquesta/test_wiring.py
+++ b/st2tests/integration/orquesta/test_wiring.py
@@ -156,3 +156,23 @@ def test_config_context_renders(self):
 
         self.assertEqual(ex.status, ac_const.LIVEACTION_STATUS_SUCCEEDED)
         self.assertDictEqual(ex.result, expected_result)
+
+    def test_field_escaping(self):
+        wf_name = 'examples.orquesta-test-field-escaping'
+
+        ex = self._execute_workflow(wf_name)
+        ex = self._wait_for_completion(ex)
+
+        expected_output = {
+            'wf.hostname.with.periods': {
+                'hostname.domain.tld': 'vars.value.with.periods',
+                'hostname2.domain.tld': {
+                    'stdout': 'vars.nested.value.with.periods',
+                },
+            },
+            'wf.output.with.periods': 'vars.nested.value.with.periods',
+        }
+        expected_result = {'output': expected_output}
+
+        self.assertEqual(ex.status, ac_const.LIVEACTION_STATUS_SUCCEEDED)
+        self.assertDictEqual(ex.result, expected_result)