Repository: StackStorm/st2

Issue:
Inquiry Responses Do Not Mask Secrets
# Problem

When responding to an inquiry with `st2 inquiry respond xxx` the response data displayed does not mask out properties marked as `secret: true`.

# Steps to reproduce

## Action Metadata
``` yaml
---
description: "Restarts a service on the system (with duo integrated 2FA)"
enabled: true
runner_type: "mistral-v2"
entry_point: workflows/service_restart_v3.yaml
name: service_restart_v3
pack: demo_inquiries
parameters:
  service:
    type: string
    description: "Name of the service to restart."
    required: true

```

## Workflow
``` yaml
version: '2.0'

demo_inquiries.service_restart_v3:
  description: Restarts a service on the system (with duo integrated 2FA)
  type: direct
  input:
    - service

  output:
    result: "{{ _.result }}"
    
  tasks:
    inquiry_2fa:
       action: core.ask
       input:
         route: developers
         schema:
           type: object
           properties:
             username:
               type: string
               description: "Please enter Duo username for restarting the '{{ _.service }}' service"
               required: true
             passcode:
               type: string
               description: "Please enter Duo passcode (generated on the mobile app) for restarting the '{{ _.service }}' service"
               required: true
               secret: true
       on-success:
         - duo_2fa

    duo_2fa:
      action: duo.auth_auth
      input:
        username: "{{ task('inquiry_2fa').result.response.username }}"
        factor: passcode
        passcode: "{{ task('inquiry_2fa').result.response.passcode }}"
      on-success:
        - service_restart
              
    service_restart:
      action: core.local_sudo
      input:
        cmd: "systemctl restart {{ _.service }}"
      publish:
        result: "{{ task('service_restart').result }}"
```

## Commands

``` shell
$ st2 run demo_inquiries.service_restart_v3 service=crond
.
id: 59f772a1032fb64ea8a99bd0
action.ref: demo_inquiries.service_restart_v3
parameters: 
  service: crond
status: pausing
start_timestamp: 2017-10-30T18:42:41.554471Z
end_timestamp: None
+--------------------------+----------------------+-------------+----------+-----------------+
| id                       | status               | task        | action   | start_timestamp |
+--------------------------+----------------------+-------------+----------+-----------------+
| 59f772a2032fb64ea8a99bd3 | running (2s elapsed) | inquiry_2fa | core.ask | Mon, 30 Oct     |
|                          |                      |             |          | 2017 18:42:42   |
|                          |                      |             |          | UTC             |
+--------------------------+----------------------+-------------+----------+-----------------+


# note the secret is masked out when receiving input, but not when displaying the response
$ st2 inquiry respond 59f772a2032fb64ea8a99bd3
passcode (secret): ******
username: testuser

 Response accepted. Successful response data to follow...
+----------+----------------------------+
| Property | Value                      |
+----------+----------------------------+
| id       | 59f772a2032fb64ea8a99bd3   |
| response | {                          |
|          |     "passcode": "123456",  |
|          |     "username": "testuser" |
|          | }                          |
+----------+----------------------------+
```


Discussion:

Resolving patch:
diff --git a/st2client/st2client/commands/inquiry.py b/st2client/st2client/commands/inquiry.py
--- a/st2client/st2client/commands/inquiry.py
+++ b/st2client/st2client/commands/inquiry.py
@@ -24,6 +24,7 @@
 LOG = logging.getLogger(__name__)
 
 DEFAULT_SCOPE = 'system'
+MASKED_ATTRIBUTE_VALUE = '******'
 
 
 class InquiryBranch(resource.ResourceBranch):
@@ -127,15 +128,18 @@ def __init__(self, resource, *args, **kwargs):
     def run(self, args, **kwargs):
         instance = Inquiry()
         instance.id = args.id
+        inquiry = self.get_resource_by_id(id=args.id, **kwargs)
         if args.response:
             instance.response = json.loads(args.response)
         else:
-            inquiry = self.get_resource_by_id(id=args.id, **kwargs)
             response = InteractiveForm(
                 inquiry.schema.get('properties')).initiate_dialog()
             instance.response = response
 
-        return self.manager.update(instance, **kwargs)
+        # Send the response to the API first, then sanitize secrets
+        instance = self.manager.update(instance, **kwargs)
+        instance = self.sanitize_secrets(instance, inquiry.schema)
+        return instance
 
     def run_and_print(self, args, **kwargs):
         instance = self.run(args, **kwargs)
@@ -143,3 +147,10 @@ def run_and_print(self, args, **kwargs):
         self.print_output(instance, table.PropertyValueTable,
                           attributes=self.display_attributes, json=args.json,
                           yaml=args.yaml)
+
+    def sanitize_secrets(self, instance, schema):
+        for prop_name, prop_attrs in schema['properties'].items():
+            if prop_attrs.get('secret') is True:
+                if prop_name in instance.response:
+                    instance.response[prop_name] = MASKED_ATTRIBUTE_VALUE
+        return instance
diff --git a/st2common/st2common/models/db/execution.py b/st2common/st2common/models/db/execution.py
--- a/st2common/st2common/models/db/execution.py
+++ b/st2common/st2common/models/db/execution.py
@@ -22,9 +22,9 @@
 from st2common.fields import ComplexDateTimeField
 from st2common.util import date as date_utils
 from st2common.util.secrets import get_secret_parameters
+from st2common.util.secrets import mask_inquiry_response
 from st2common.util.secrets import mask_secret_parameters
 from st2common.constants.types import ResourceType
-
 __all__ = [
     'ActionExecutionDB',
     'ActionExecutionOutputDB'
@@ -110,6 +110,18 @@ def mask_secrets(self, value):
         if 'parameters' in liveaction:
             liveaction['parameters'] = mask_secret_parameters(parameters=liveaction['parameters'],
                                                               secret_parameters=secret_parameters)
+
+        # TODO(mierdin): This logic should be moved to the dedicated Inquiry
+        # data model once it exists.
+        if self.runner.get('name') == "inquirer":
+
+            schema = result['result'].get('schema', {})
+            response = result['result'].get('response', {})
+
+            # We can only mask response secrets if response and schema exist and are
+            # not empty
+            if response and schema:
+                result['result']['response'] = mask_inquiry_response(response, schema)
         return result
 
     def get_masked_parameters(self):
diff --git a/st2common/st2common/util/secrets.py b/st2common/st2common/util/secrets.py
--- a/st2common/st2common/util/secrets.py
+++ b/st2common/st2common/util/secrets.py
@@ -58,3 +58,24 @@ def mask_secret_parameters(parameters, secret_parameters):
             result[parameter] = MASKED_ATTRIBUTE_VALUE
 
     return result
+
+
+def mask_inquiry_response(response, schema):
+    """
+    Introspect an Inquiry's response dict and return a new dict with masked secret
+    values.
+
+    :param response: Inquiry response to process.
+    :type response: ``dict``
+
+    :param schema: Inquiry response schema
+    :type schema: ``dict``
+    """
+    result = copy.deepcopy(response)
+
+    for prop_name, prop_attrs in schema['properties'].items():
+        if prop_attrs.get('secret') is True:
+            if prop_name in response:
+                result[prop_name] = MASKED_ATTRIBUTE_VALUE
+
+    return result


Test patch:
diff --git a/st2api/tests/unit/controllers/v1/test_executions.py b/st2api/tests/unit/controllers/v1/test_executions.py
--- a/st2api/tests/unit/controllers/v1/test_executions.py
+++ b/st2api/tests/unit/controllers/v1/test_executions.py
@@ -26,6 +26,7 @@
 from six.moves import filter
 
 from st2common.constants import action as action_constants
+from st2common.constants.secrets import MASKED_ATTRIBUTE_VALUE
 from st2common.models.db.execution import ActionExecutionDB
 from st2common.models.db.execution import ActionExecutionOutputDB
 from st2common.persistence.execution import ActionExecution
@@ -133,6 +134,14 @@
     }
 }
 
+ACTION_INQUIRY = {
+    'name': 'st2.dummy.ask',
+    'description': 'another test description',
+    'enabled': True,
+    'pack': 'wolfpack',
+    'runner_type': 'inquirer',
+}
+
 LIVE_ACTION_1 = {
     'action': 'sixpack.st2.dummy.action1',
     'parameters': {
@@ -164,6 +173,44 @@
     'action': 'starterpack.st2.dummy.action4',
 }
 
+LIVE_ACTION_INQUIRY = {
+    'parameters': {
+        'route': 'developers',
+        'schema': {
+            'type': 'object',
+            'properties': {
+                'secondfactor': {
+                    'secret': True,
+                    'required': True,
+                    'type': u'string',
+                    'description': 'Please enter second factor for authenticating to "foo" service'
+                }
+            }
+        }
+    },
+    'action': 'wolfpack.st2.dummy.ask',
+    'result': {
+        'users': [],
+        'roles': [],
+        'route': 'developers',
+        'ttl': 1440,
+        'response': {
+            'secondfactor': 'supersecretvalue'
+        },
+        'schema': {
+            'type': 'object',
+            'properties': {
+                'secondfactor': {
+                    'secret': True,
+                    'required': True,
+                    'type': 'string',
+                    'description': 'Please enter second factor for authenticating to "foo" service'
+                }
+            }
+        }
+    }
+}
+
 FIXTURES_PACK = 'generic'
 TEST_FIXTURES = {
     'runners': ['testrunner1.yaml'],
@@ -196,12 +243,17 @@ def setUpClass(cls):
         post_resp = cls.app.post_json('/v1/actions', cls.action4)
         cls.action4['id'] = post_resp.json['id']
 
+        cls.action_inquiry = copy.deepcopy(ACTION_INQUIRY)
+        post_resp = cls.app.post_json('/v1/actions', cls.action_inquiry)
+        cls.action_inquiry['id'] = post_resp.json['id']
+
     @classmethod
     def tearDownClass(cls):
         cls.app.delete('/v1/actions/%s' % cls.action1['id'])
         cls.app.delete('/v1/actions/%s' % cls.action2['id'])
         cls.app.delete('/v1/actions/%s' % cls.action3['id'])
         cls.app.delete('/v1/actions/%s' % cls.action4['id'])
+        cls.app.delete('/v1/actions/%s' % cls.action_inquiry['id'])
         super(BaseActionExecutionControllerTestCase, cls).tearDownClass()
 
     def test_get_one(self):
@@ -1051,6 +1103,31 @@ def test_put_resume_with_result(self):
         finally:
             action_constants.WORKFLOW_RUNNER_TYPES.remove(ACTION_1['runner_type'])
 
+    def test_get_inquiry_mask(self):
+        """Ensure Inquiry responses are masked when retrieved via ActionExecution GET
+
+        The reason this test is included here is so that we can verify that the ActionExecution
+        GET function properly masks fields within an Inquiry response.
+        TODO(mierdin): This test, and the constants it uses, should not be necessary here
+        once Inquiries get their own data model.
+        """
+
+        post_resp = self._do_post(LIVE_ACTION_INQUIRY)
+        actionexecution_id = self._get_actionexecution_id(post_resp)
+        get_resp = self._do_get_one(actionexecution_id)
+        self.assertEqual(get_resp.status_int, 200)
+
+        resp = json.loads(get_resp.body)
+        self.assertEqual(resp['result']['response']['secondfactor'], MASKED_ATTRIBUTE_VALUE)
+
+        post_resp = self._do_post(LIVE_ACTION_INQUIRY)
+        actionexecution_id = self._get_actionexecution_id(post_resp)
+        get_resp = self._do_get_one(actionexecution_id, params={'show_secrets': True})
+        self.assertEqual(get_resp.status_int, 200)
+
+        resp = json.loads(get_resp.body)
+        self.assertEqual(resp['result']['response']['secondfactor'], "supersecretvalue")
+
 
 class ActionExecutionOutputControllerTestCase(BaseActionExecutionControllerTestCase,
                                               FunctionalTest):
diff --git a/st2client/tests/unit/test_inquiry.py b/st2client/tests/unit/test_inquiry.py
--- a/st2client/tests/unit/test_inquiry.py
+++ b/st2client/tests/unit/test_inquiry.py
@@ -23,6 +23,7 @@
 
 from tests import base
 from st2client import shell
+from st2client.commands.inquiry import MASKED_ATTRIBUTE_VALUE
 
 LOG = logging.getLogger(__name__)
 
@@ -70,6 +71,29 @@ def tearDown(self):
     },
 }
 
+SCHEMA_SECRET = {
+    "title": "response_data",
+    "type": "object",
+    "properties": {
+        "secondfactor": {
+            "type": "string",
+            "description": "Please enter secret second factor password",
+            "required": True,
+            "secret": True
+        },
+        "notsecret": {
+            "type": "string",
+            "description": "Please enter not secret obvious thing",
+            "required": True,
+        }
+    },
+}
+
+RESPONSE_SECRET = {
+    "secondfactor": "asdfoije",
+    "notsecret": "notmasked"
+}
+
 RESPONSE_DEFAULT = {
     "continue": True
 }
@@ -124,6 +148,15 @@ def tearDown(self):
     "ttl": 1440
 }
 
+INQUIRY_SECRET = {
+    "id": "face",
+    "schema": SCHEMA_SECRET,
+    "roles": [],
+    "users": [],
+    "route": "",
+    "ttl": 1440
+}
+
 
 class TestInquirySubcommands(TestInquiryBase):
 
@@ -244,11 +277,6 @@ def test_respond_invalid(self):
         self.assertEqual(retcode, 1)
         self.assertEqual('ERROR: 400 Client Error: Bad Request', self.stdout.getvalue().strip())
 
-    @mock.patch.object(
-        requests, 'put',
-        mock.MagicMock(return_value=(base.FakeResponse(
-            json.dumps({}), 404, '404 Client Error: Not Found'
-        ))))
     def test_respond_nonexistent_inquiry(self):
         """Test responding to an inquiry that doesn't exist
         """
@@ -256,7 +284,7 @@ def test_respond_nonexistent_inquiry(self):
         args = ['inquiry', 'respond', '-r', '"%s"' % RESPONSE_DEFAULT, inquiry_id]
         retcode = self.shell.run(args)
         self.assertEqual(retcode, 1)
-        self.assertEqual('ERROR: 404 Client Error: Not Found',
+        self.assertEqual('ERROR: Resource with id "%s" doesn\'t exist.' % inquiry_id,
                          self.stdout.getvalue().strip())
 
     @mock.patch.object(
@@ -280,3 +308,26 @@ def test_respond_nonexistent_inquiry_interactive(self, mock_form):
         self.assertEqual(retcode, 1)
         self.assertEqual('ERROR: Resource with id "%s" doesn\'t exist.' % inquiry_id,
                          self.stdout.getvalue().strip())
+
+    @mock.patch.object(
+        requests, 'get',
+        mock.MagicMock(return_value=(base.FakeResponse(
+            json.dumps(INQUIRY_SECRET), 200, 'OK'
+        ))))
+    @mock.patch.object(
+        requests, 'put',
+        mock.MagicMock(return_value=(base.FakeResponse(
+            json.dumps({"id": "face", "response": RESPONSE_SECRET}), 200, 'OK'
+        ))))
+    def test_respond_secret_mask(self):
+        """Test masking of values
+        """
+        args = ['inquiry', 'respond', '-r', '"%s"' % RESPONSE_SECRET, 'face']
+        retcode = self.shell.run(args)
+
+        table_out = self.stdout.getvalue().strip()
+        i = table_out.index("secondfactor")
+        y = table_out.index("notsecret")
+        self.assertEqual(table_out[i + 16:i + 22], MASKED_ATTRIBUTE_VALUE)
+        self.assertEqual(table_out[y + 13:y + 22], "notmasked")
+        self.assertEqual(retcode, 0)
diff --git a/st2common/tests/unit/test_db_execution.py b/st2common/tests/unit/test_db_execution.py
new file mode 100644
--- /dev/null
+++ b/st2common/tests/unit/test_db_execution.py
@@ -0,0 +1,121 @@
+# Licensed to the StackStorm, Inc ('StackStorm') under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import mock
+
+from st2common.constants.secrets import MASKED_ATTRIBUTE_VALUE
+from st2common.exceptions.db import StackStormDBObjectNotFoundError
+from st2common.models.db.execution import ActionExecutionDB
+from st2common.persistence.execution import ActionExecution
+from st2common.transport.publishers import PoolPublisher
+from st2common.util import date as date_utils
+
+from st2tests import DbTestCase
+
+
+INQUIRY_RESULT = {
+    'users': [],
+    'roles': [],
+    'route': 'developers',
+    'ttl': 1440,
+    'response': {
+        'secondfactor': 'supersecretvalue'
+    },
+    'schema': {
+        'type': 'object',
+        'properties': {
+            'secondfactor': {
+                'secret': True,
+                'required': True,
+                'type': 'string',
+                'description': 'Please enter second factor for authenticating to "foo" service'
+            }
+        }
+    }
+}
+
+INQUIRY_LIVEACTION = {
+    'parameters': {
+        'route': 'developers',
+        'schema': {
+            'type': 'object',
+            'properties': {
+                'secondfactor': {
+                    'secret': True,
+                    'required': True,
+                    'type': u'string',
+                    'description': 'Please enter second factor for authenticating to "foo" service'
+                }
+            }
+        }
+    },
+    'action': 'core.ask'
+}
+
+
+@mock.patch.object(PoolPublisher, 'publish', mock.MagicMock())
+class ActionExecutionModelTest(DbTestCase):
+
+    def setUp(self):
+        created = ActionExecutionDB()
+        created.action = {'uid': 'action:core:ask'}
+        created.status = 'succeeded'
+        created.runner = {'name': 'inquirer'}
+        created.liveaction = INQUIRY_LIVEACTION
+        created.result = INQUIRY_RESULT
+
+        self.saved = ActionExecutionModelTest._save_execution(created)
+        self.retrieved = ActionExecution.get_by_id(self.saved.id)
+        self.assertEqual(self.saved.action, self.retrieved.action,
+                         'Same action was not returned.')
+
+    def tearDown(self):
+        ActionExecutionModelTest._delete([self.retrieved])
+        try:
+            retrieved = ActionExecution.get_by_id(self.saved.id)
+        except StackStormDBObjectNotFoundError:
+            retrieved = None
+        self.assertIsNone(retrieved, 'managed to retrieve after failure.')
+
+    def test_update_execution(self):
+        """Test ActionExecutionDb update
+        """
+        self.assertTrue(self.retrieved.end_timestamp is None)
+        self.retrieved.end_timestamp = date_utils.get_datetime_utc_now()
+        updated = ActionExecution.add_or_update(self.retrieved)
+        self.assertTrue(updated.end_timestamp == self.retrieved.end_timestamp)
+
+    def test_execution_inquiry_secrets(self):
+        """Corner case test for Inquiry responses that contain secrets.
+
+        Should properly mask these if the Inquiry is being retrieved
+        directly via `execution get` commands.
+
+        TODO(mierdin): Move this once Inquiries get their own data model
+        """
+
+        # Test Inquiry response masking is done properly within this model
+        masked = self.retrieved.mask_secrets(self.retrieved.to_serializable_dict())
+        self.assertEqual(masked['result']['response']['secondfactor'], MASKED_ATTRIBUTE_VALUE)
+        self.assertEqual(self.retrieved.result['response']['secondfactor'], "supersecretvalue")
+
+    @staticmethod
+    def _save_execution(execution):
+        return ActionExecution.add_or_update(execution)
+
+    @staticmethod
+    def _delete(model_objects):
+        for model_object in model_objects:
+            model_object.delete()