Repository: StackStorm/st2

Issue:
Bad rewrite directive in controller nginx HA config
##### SUMMARY

I tried to use [the controller nginx HA config](https://github.com/StackStorm/st2/blob/v3.0.0/conf/HA/st2.conf.sample) for setting up our controller nginx server.

However, I noticed that when logging in to the WebUI, there was an error message in the JS console:
> EventSource's response has a MIME type ("text/html") that is not "text/event-stream". Aborting the connection.

This is because:
1. The WebUI requests `https://controller-ip/stream/stream`
2. Because of the rewrite directive quoted below, it redirects to `https://controller-ip/stream/`
3. The path which gets passed to the proxy is `/` instead of `/stream/` somehow (I don't know exactly why about this, but I can tell that this is what is happening for sure, I checked the nginx logs of the worker)
4. The [worker nginx HA config](https://github.com/StackStorm/st2/blob/master/conf/HA/nginx/st2.conf.blueprint.sample) expects an URI with `/stream/` but instead receives `/`
5. The worker nginx server will then respond with the default nginx page
6. Causing the JS error because it is indeed not the expected response

https://github.com/StackStorm/st2/blob/2c8fec90a8029fdce4e7adf19af683edb84bd006/conf/HA/nginx/st2.conf.controller.sample#L82

When removing/commenting out this directive, the request `https://controller-ip/stream/stream` made by the WebUI is passed to the worker nginx as `/stream`, which correctly responds with the intended response of type `text/event-stream`.

##### ISSUE TYPE
Bug Report

##### STACKSTORM VERSION
st2 3.0.0, on Python 2.7.12

(seems to affect previous versions as well)

##### OS / ENVIRONMENT / INSTALL METHOD
Custom HA install

##### STEPS TO REPRODUCE
N/A

##### EXPECTED RESULTS
N/A

##### ACTUAL RESULTS
N/A


Discussion:
Thanks for reporting this issue.

It looks like https://github.com/StackStorm/st2/blob/master/conf/HA/nginx/st2.conf.blueprint.sample you mentioned is a bit unsync with https://github.com/StackStorm/st2/blob/master/conf/HA/st2.conf.sample

Can you please try replacing all `stream`-related locations with the following nginx rules:
https://github.com/StackStorm/st2/blob/2c8fec90a8029fdce4e7adf19af683edb84bd006/conf/nginx/st2.conf#L72-L102 and verify if the issue still present?

Should I replace it in the controller nginx configuration or in the worker nginx configuration?

Also, did you mean `proxy_pass  https://st2/;` instead of `proxy_pass  http://127.0.0.1:9102/;` (the snippet of code you provided isn't the HA version I think)? C.F. https://github.com/StackStorm/st2/blob/2c8fec90a8029fdce4e7adf19af683edb84bd006/conf/nginx/st2.conf#L86

I replaced it in the controller nginx configuration with `proxy_pass  https://st2/;` and the issue is still present.
Thanks for the report @GuiTeK, I could reproduce it.

Will update the configs soon.

Resolving patch:
diff --git a/st2client/st2client/models/core.py b/st2client/st2client/models/core.py
--- a/st2client/st2client/models/core.py
+++ b/st2client/st2client/models/core.py
@@ -14,6 +14,7 @@
 # limitations under the License.
 
 from __future__ import absolute_import
+
 import os
 import json
 import logging
@@ -22,6 +23,7 @@
 import six
 from six.moves import urllib
 from six.moves import http_client
+import requests
 
 from st2client.utils import httpclient
 
@@ -632,8 +634,10 @@ def listen(self, events=None, **kwargs):
         query_string = '?' + urllib.parse.urlencode(query_params)
         url = url + query_string
 
-        for message in SSEClient(url, **request_params):
+        response = requests.get(url, stream=True, **request_params)
+        client = SSEClient(response)
 
+        for message in client.events():
             # If the execution on the API server takes too long, the message
             # can be empty. In this case, rerun the query.
             if not message.data:
diff --git a/st2common/st2common/models/db/execution_queue.py b/st2common/st2common/models/db/execution_queue.py
--- a/st2common/st2common/models/db/execution_queue.py
+++ b/st2common/st2common/models/db/execution_queue.py
@@ -62,10 +62,13 @@ class ActionExecutionSchedulingQueueItemDB(stormbase.StormFoundationDB,
 
     meta = {
         'indexes': [
-            {'fields': ['action_execution_id']},
-            {'fields': ['liveaction_id']},
-            {'fields': ['original_start_timestamp']},
-            {'fields': ['scheduled_start_timestamp']},
+            # NOTE: We limit index names to 65 characters total for compatibility with AWS
+            # DocumentDB.
+            # See https://github.com/StackStorm/st2/pull/4690 for details.
+            {'fields': ['action_execution_id'], 'name': 'ac_exc_id'},
+            {'fields': ['liveaction_id'], 'name': 'lv_ac_id'},
+            {'fields': ['original_start_timestamp'], 'name': 'orig_s_ts'},
+            {'fields': ['scheduled_start_timestamp'], 'name': 'schd_s_ts'},
         ]
     }
 
diff --git a/st2tests/st2tests/base.py b/st2tests/st2tests/base.py
--- a/st2tests/st2tests/base.py
+++ b/st2tests/st2tests/base.py
@@ -67,6 +67,7 @@
 import st2common.models.db.runner as runner_model
 import st2common.models.db.execution as execution_model
 import st2common.models.db.executionstate as executionstate_model
+import st2common.models.db.execution_queue as execution_queue_model
 import st2common.models.db.liveaction as liveaction_model
 import st2common.models.db.actionalias as actionalias_model
 import st2common.models.db.policy as policy_model
@@ -112,6 +113,7 @@
 ALL_MODELS.extend(runner_model.MODELS)
 ALL_MODELS.extend(execution_model.MODELS)
 ALL_MODELS.extend(executionstate_model.MODELS)
+ALL_MODELS.extend(execution_queue_model.MODELS)
 ALL_MODELS.extend(liveaction_model.MODELS)
 ALL_MODELS.extend(actionalias_model.MODELS)
 ALL_MODELS.extend(policy_model.MODELS)


Test patch:
diff --git a/st2client/tests/unit/test_models.py b/st2client/tests/unit/test_models.py
--- a/st2client/tests/unit/test_models.py
+++ b/st2client/tests/unit/test_models.py
@@ -257,32 +257,41 @@ def test_resource_delete_failed(self):
         instance = mgr.get_by_name('abc')
         self.assertRaises(Exception, mgr.delete, instance)
 
+    @mock.patch('requests.get')
     @mock.patch('sseclient.SSEClient')
-    def test_stream_resource_listen(self, mock):
+    def test_stream_resource_listen(self, mock_sseclient, mock_requests):
         mock_msg = mock.Mock()
         mock_msg.data = json.dumps(base.RESOURCES)
 
         # checking the case to specify valid 'cacert' parameter to the StreamManager
-        def side_effect_checking_verify_parameter_is(endpoint_url, **kwargs):
-            self.assertEqual(endpoint_url, 'https://example.com/stream?events=foo%2Cbar')
-            self.assertEqual(kwargs['verify'], '/path/ca.crt')
+        def side_effect_checking_verify_parameter_is():
             return [mock_msg]
 
-        mock.side_effect = side_effect_checking_verify_parameter_is
+        mock_sseclient.return_value.events.side_effect = side_effect_checking_verify_parameter_is
         mgr = models.StreamManager('https://example.com', cacert='/path/ca.crt')
 
         resp = mgr.listen(events=['foo', 'bar'])
         self.assertEqual(list(resp), [base.RESOURCES])
 
+        call_args = tuple(['https://example.com/stream?events=foo%2Cbar'])
+        call_kwargs = {'stream': True, 'verify': '/path/ca.crt'}
+
+        self.assertEqual(mock_requests.call_args_list[0][0], call_args)
+        self.assertEqual(mock_requests.call_args_list[0][1], call_kwargs)
+
         # checking the case not to specify valid 'cacert' parameter to the StreamManager
-        def side_effect_checking_verify_parameter_is_not(endpoint_url, **kwargs):
-            # checking endpoint_url in case of no event specification
-            self.assertEqual(endpoint_url, 'https://example.com/stream?')
-            self.assertFalse('verify' in kwargs)
+        def side_effect_checking_verify_parameter_is_not():
             return [mock_msg]
 
-        mock.side_effect = side_effect_checking_verify_parameter_is_not
+        mock_sseclient.return_value.events.side_effect = \
+            side_effect_checking_verify_parameter_is_not
         mgr = models.StreamManager('https://example.com')
 
         resp = mgr.listen()
         self.assertEqual(list(resp), [base.RESOURCES])
+
+        call_args = tuple(['https://example.com/stream?'])
+        call_kwargs = {'stream': True}
+
+        self.assertEqual(mock_requests.call_args_list[1][0], call_args)
+        self.assertEqual(mock_requests.call_args_list[1][1], call_kwargs)
diff --git a/st2common/tests/unit/test_db.py b/st2common/tests/unit/test_db.py
--- a/st2common/tests/unit/test_db.py
+++ b/st2common/tests/unit/test_db.py
@@ -39,6 +39,9 @@
 from st2common.persistence.trigger import TriggerType, Trigger, TriggerInstance
 from st2tests import DbTestCase
 
+from unittest2 import TestCase
+from st2tests.base import ALL_MODELS
+
 
 __all__ = [
     'DbConnectionTestCase',
@@ -52,6 +55,38 @@
 DUMMY_DESCRIPTION = 'Sample Description.'
 
 
+class DbIndexNameTestCase(TestCase):
+    """
+    Test which verifies that model index name are not longer than the specified limit.
+    """
+    LIMIT = 65
+
+    def test_index_name_length(self):
+        db_name = 'st2'
+
+        for model in ALL_MODELS:
+            collection_name = model._get_collection_name()
+            model_indexes = model._meta['index_specs']
+
+            for index_specs in model_indexes:
+                index_name = index_specs.get('name', None)
+                if index_name:
+                    # Custom index name defined by the developer
+                    index_field_name = index_name
+                else:
+                    # No explicit index name specified, one is auto-generated using
+                    # <db name>.<collection name>.<index field names> schema
+                    index_fields = dict(index_specs['fields']).keys()
+                    index_field_name = '.'.join(index_fields)
+
+                index_name = '%s.%s.%s' % (db_name, collection_name, index_field_name)
+
+                if len(index_name) > self.LIMIT:
+                    self.fail('Index name "%s" for model "%s" is longer than %s characters. '
+                              'Please manually define name for this index so it\'s shorter than '
+                              'that' % (index_name, model.__name__, self.LIMIT))
+
+
 class DbConnectionTestCase(DbTestCase):
 
     def test_check_connect(self):