Repository: StackStorm/st2

Issue:
st2 run --tail stops tailing workflows after first sub-workflow completes
When tailing a Mistral workflow that has tasks that are also Mistral workflows (mistral-v2 actions) the tailing stops when the first sub-workflow completes instead of waiting for the root workflow to complete.

Example:
```shell
[root@stackstorm ~]# st2 run --tail examples.mistral-workbook-subflows
Tailing execution "5a67e8f9a814c0349e2fa1f9"
Child execution (task=print_subject) 5a67e8f9a814c0349e2fa1fc has started.

Child execution (task=print_subject) 5a67e8f9a814c0349e2fa1fc has started.

5a67e8f9a814c0349e2fa1fc

Execution 5a67e8f9a814c0349e2fa1f9 has completed (status=succeeded).

id: 5a67e8f9a814c0349e2fa1f9
action.ref: examples.mistral-workbook-subflows
parameters: None
status: running
start_timestamp: Wed, 24 Jan 2018 02:01:29 UTC
end_timestamp: 
+-----------------------------+------------------------+---------------+--------------------+-----------------+
| id                          | status                 | task          | action             | start_timestamp |
+-----------------------------+------------------------+---------------+--------------------+-----------------+
| + 5a67e8f9a814c0349e2fa1fc  | running (1s elapsed)   | print_subject | examples.mistral-  | Wed, 24 Jan     |
|                             |                        |               | basic              | 2018 02:01:29   |
|                             |                        |               |                    | UTC             |
|    5a67e8faa814c0349e2fa1fe | succeeded (0s elapsed) | task1         | core.local         | Wed, 24 Jan     |
|                             |                        |               |                    | 2018 02:01:30   |
|                             |                        |               |                    | UTC             |
+-----------------------------+------------------------+---------------+--------------------+-----------------+
```

You can see this stops after the first task `print_subject` action  `examples.mistral-basic` completes.

Looking at the full output afterwards you can see there is another action (workflow) that is executed after.

``` shell
[root@stackstorm ~]# st2 execution get 5a67e8f9a814c0349e2fa1f9
id: 5a67e8f9a814c0349e2fa1f9
action.ref: examples.mistral-workbook-subflows
parameters: None
status: succeeded (7s elapsed)
result_task: print_adjective
result: 
  extra:
    state: SUCCESS
    state_info: null
  stdout: 5a67e8fca814c0349e2fa200
  tasks:
  - created_at: '2018-01-24 02:01:32'
    id: b02a0230-5283-4945-a830-fcb730abf105
    input: null
    name: task1
    published:
      stdout: 5a67e8fca814c0349e2fa200
    result:
      failed: false
      return_code: 0
      stderr: ''
      stdout: 5a67e8fca814c0349e2fa200
      succeeded: true
    state: SUCCESS
    state_info: null
    updated_at: '2018-01-24 02:01:33'
    workflow_execution_id: 8c9fcedf-404b-4ca9-ab2f-a25a24d0e06e
    workflow_name: examples.mistral-basic
start_timestamp: Wed, 24 Jan 2018 02:01:29 UTC
end_timestamp: Wed, 24 Jan 2018 02:01:36 UTC
+-----------------------------+------------------------+-----------------+--------------------+-----------------+
| id                          | status                 | task            | action             | start_timestamp |
+-----------------------------+------------------------+-----------------+--------------------+-----------------+
| + 5a67e8f9a814c0349e2fa1fc  | succeeded (3s elapsed) | print_subject   | examples.mistral-  | Wed, 24 Jan     |
|                             |                        |                 | basic              | 2018 02:01:29   |
|                             |                        |                 |                    | UTC             |
|    5a67e8faa814c0349e2fa1fe | succeeded (0s elapsed) | task1           | core.local         | Wed, 24 Jan     |
|                             |                        |                 |                    | 2018 02:01:30   |
|                             |                        |                 |                    | UTC             |
| + 5a67e8fca814c0349e2fa200  | succeeded (2s elapsed) | print_adjective | examples.mistral-  | Wed, 24 Jan     |
|                             |                        |                 | basic              | 2018 02:01:32   |
|                             |                        |                 |                    | UTC             |
|    5a67e8fca814c0349e2fa202 | succeeded (1s elapsed) | task1           | core.local         | Wed, 24 Jan     |
|                             |                        |                 |                    | 2018 02:01:32   |
|                             |                        |                 |                    | UTC             |
+-----------------------------+------------------------+-----------------+--------------------+-----------------+
```
 


Discussion:
Thanks for reporting this.

Do you have a simple sample workflow which reproduces this problem?

In the past I tested it with action chain and it should work fine for single level nested workflows (but not more), but there could also be some edge case which is not handled correctly.

The CLI code should be workflow engine independent, since it just uses execution ids, but could be something else going on.
@Kami the following action in the examples pack reproduces this `examples.mistral-workbook-subflows`
https://github.com/StackStorm/st2/blob/master/contrib/examples/actions/mistral-workbook-subflows.yaml
Thanks.

I'm looking into it and yeah, it looks like it's indent related to known / documented limitation of only supporting single level nested executions (in this case it's workflow -> workflow -> execution, but only workflow -> execution is supported right now).
I will look into a small change to also support two levels of nested executions (such as in this case, workflow -> workflow -> execution), but supporting arbitrary nested executions will require more work in the future.

Resolving patch:
diff --git a/st2client/st2client/commands/action.py b/st2client/st2client/commands/action.py
--- a/st2client/st2client/commands/action.py
+++ b/st2client/st2client/commands/action.py
@@ -1450,10 +1450,11 @@ def run_and_print(self, args, **kwargs):
     def tail_execution(cls, execution_manager, stream_manager, execution, output_type=None,
                        include_metadata=False, **kwargs):
         execution_id = str(execution.id)
+
         # Note: For non-workflow actions child_execution_id always matches parent_execution_id so
         # we don't need to do any other checks to determine if executions represents a workflow
         # action
-        parent_execution_id = execution_id
+        parent_execution_id = execution_id  # noqa
 
         # Execution has already finished
         if execution.status in LIVEACTION_COMPLETED_STATES:
@@ -1475,7 +1476,9 @@ def tail_execution(cls, execution_manager, stream_manager, execution, output_typ
                 task_execution_id = context['execution_id']
                 task_name = context['task_name']
                 task_parent_execution_id = context['parent_execution_id']
-                is_child_execution = (task_parent_execution_id == parent_execution_id)
+
+                # An execution is considered a child execution if it has parent execution id
+                is_child_execution = bool(task_parent_execution_id)
 
                 if is_child_execution:
                     if status == LIVEACTION_STATUS_RUNNING:
@@ -1492,7 +1495,11 @@ def tail_execution(cls, execution_manager, stream_manager, execution, output_typ
                         # We don't care about other child events so we simply skip then
                         continue
                 else:
-                    if status in LIVEACTION_COMPLETED_STATES:
+                    if status == LIVEACTION_STATUS_RUNNING:
+                        print('Execution %s has started.' % (execution_id))
+                        print('')
+                        continue
+                    elif status in LIVEACTION_COMPLETED_STATES:
                         # Bail out once parent execution has finished
                         print('')
                         print('Execution %s has completed (status=%s).' % (execution_id, status))


Test patch:
diff --git a/st2client/tests/unit/test_execution_tail_command.py b/st2client/tests/unit/test_execution_tail_command.py
--- a/st2client/tests/unit/test_execution_tail_command.py
+++ b/st2client/tests/unit/test_execution_tail_command.py
@@ -150,6 +150,19 @@
     'status': LIVEACTION_STATUS_RUNNING
 }
 
+MOCK_LIVEACTION_4_CHILD_1_1_RUNNING = {
+    'id': 'idmistralchild1_1',
+    'context': {
+        'mistral': {
+            'task_name': 'task_1'
+        },
+        'parent': {
+            'execution_id': 'idmistralchild1'
+        }
+    },
+    'status': LIVEACTION_STATUS_RUNNING
+}
+
 MOCK_LIVEACTION_4_CHILD_1_SUCCEEDED = {
     'id': 'idmistralchild1',
     'context': {
@@ -163,6 +176,19 @@
     'status': LIVEACTION_STATUS_SUCCEEDED
 }
 
+MOCK_LIVEACTION_4_CHILD_1_1_SUCCEEDED = {
+    'id': 'idmistralchild1_1',
+    'context': {
+        'mistral': {
+            'task_name': 'task_1',
+        },
+        'parent': {
+            'execution_id': 'idmistralchild1'
+        }
+    },
+    'status': LIVEACTION_STATUS_SUCCEEDED
+}
+
 MOCK_LIVEACTION_4_CHILD_1_OUTPUT_1 = {
     'execution_id': 'idmistralchild1',
     'timestamp': '1505732598',
@@ -177,6 +203,20 @@
     'data': 'line mistral 5\n'
 }
 
+MOCK_LIVEACTION_4_CHILD_1_1_OUTPUT_1 = {
+    'execution_id': 'idmistralchild1_1',
+    'timestamp': '1505732598',
+    'output_type': 'stdout',
+    'data': 'line mistral 4\n'
+}
+
+MOCK_LIVEACTION_4_CHILD_1_1_OUTPUT_2 = {
+    'execution_id': 'idmistralchild1_1',
+    'timestamp': '1505732598',
+    'output_type': 'stderr',
+    'data': 'line mistral 5\n'
+}
+
 MOCK_LIVEACTION_4_CHILD_2_RUNNING = {
     'id': 'idmistralchild2',
     'context': {
@@ -324,6 +364,8 @@ def test_tail_simple_execution_running_with_data(self, mock_stream_manager):
         stderr = self.stderr.getvalue()
 
         expected_result = """
+Execution idfoo3 has started.
+
 line 1
 line 2
 
@@ -381,6 +423,8 @@ def test_tail_action_chain_workflow_execution(self, mock_stream_manager):
         stderr = self.stderr.getvalue()
 
         expected_result = """
+Execution idfoo3 has started.
+
 Child execution (task=task_1) idchild1 has started.
 
 line ac 4
@@ -447,11 +491,91 @@ def test_tail_mistral_workflow_execution(self, mock_stream_manager):
         stderr = self.stderr.getvalue()
 
         expected_result = """
+Execution idfoo4 has started.
+
+Child execution (task=task_1) idmistralchild1 has started.
+
+line mistral 4
+line mistral 5
+
+Child execution (task=task_1) idmistralchild1 has finished (status=succeeded).
+Child execution (task=task_2) idmistralchild2 has started.
+
+line mistral 100
+
+Child execution (task=task_2) idmistralchild2 has finished (status=timeout).
+
+Execution idfoo4 has completed (status=succeeded).
+""".lstrip()
+        self.assertEqual(stdout, expected_result)
+        self.assertEqual(stderr, '')
+
+    @mock.patch.object(
+        httpclient.HTTPClient, 'get',
+        mock.MagicMock(return_value=base.FakeResponse(json.dumps(MOCK_LIVEACTION_4_RUNNING),
+                                                     200, 'OK')))
+    @mock.patch('st2client.client.StreamManager', autospec=True)
+    def test_tail_double_nested_mistral_workflow_execution(self, mock_stream_manager):
+        argv = ['execution', 'tail', 'idfoo4']
+
+        MOCK_EVENTS = [
+            # Workflow started running
+            MOCK_LIVEACTION_4_RUNNING,
+
+            # Child task 1 started running (sub workflow)
+            MOCK_LIVEACTION_4_CHILD_1_RUNNING,
+
+            # Child task 1 started running
+            MOCK_LIVEACTION_4_CHILD_1_1_RUNNING,
+
+            # Output produced by the child task
+            MOCK_LIVEACTION_4_CHILD_1_1_OUTPUT_1,
+            MOCK_LIVEACTION_4_CHILD_1_1_OUTPUT_2,
+
+            # Child task 1 has finished
+            MOCK_LIVEACTION_4_CHILD_1_1_SUCCEEDED,
+
+            # Child task 1 finished (sub workflow)
+            MOCK_LIVEACTION_4_CHILD_1_SUCCEEDED,
+
+            # Child task 2 started running
+            MOCK_LIVEACTION_4_CHILD_2_RUNNING,
+
+            # Output produced by child task
+            MOCK_LIVEACTION_4_CHILD_2_OUTPUT_1,
+
+            # Child task 2 finished
+            MOCK_LIVEACTION_4_CHILD_2_TIMED_OUT,
+
+            # Parent workflow task finished
+            MOCK_LIVEACTION_4_SUCCEDED
+        ]
+
+        mock_cls = mock.Mock()
+        mock_cls.listen = mock.Mock()
+        mock_listen_generator = mock.Mock()
+        mock_listen_generator.return_value = MOCK_EVENTS
+        mock_cls.listen.side_effect = mock_listen_generator
+        mock_stream_manager.return_value = mock_cls
+
+        self.assertEqual(self.shell.run(argv), 0)
+        self.assertEqual(mock_listen_generator.call_count, 1)
+
+        stdout = self.stdout.getvalue()
+        stderr = self.stderr.getvalue()
+
+        expected_result = """
+Execution idfoo4 has started.
+
 Child execution (task=task_1) idmistralchild1 has started.
 
+Child execution (task=task_1) idmistralchild1_1 has started.
+
 line mistral 4
 line mistral 5
 
+Child execution (task=task_1) idmistralchild1_1 has finished (status=succeeded).
+
 Child execution (task=task_1) idmistralchild1 has finished (status=succeeded).
 Child execution (task=task_2) idmistralchild2 has started.