Repository: StackStorm/st2

Issue:
Bug: Retrying "register exchanges" causes TypeError exception, using Redis as message queue
OS: Ubuntu 16.04
st2 version:  2.3.2

Problem description:
Testing redis as alternative message queue. Registering exchanges in kombu connection throws a TypeError.  There's a code problem here that I don't think is related to transport protocol.

```
2017-08-01 02:07:26,696 139712428401040 DEBUG bootstrap_utils [-] Registering exchanges...
2017-08-01 02:07:26,705 139712428401040 ERROR st2notifier [-] (PID=17199) Results tracker quit due to exception.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/st2notifier.py", line 56, in main
    _setup()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2actions/cmd/st2notifier.py", line 27, in _setup
    register_signal_handlers=True)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/service_setup.py", line 117, in setup
    register_exchanges_with_retry()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/transport/bootstrap_utils.py", line 83, in register_exchanges_with_retry
    return retrying_obj.call(register_exchanges)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 205, in call
    if not self.should_reject(attempt):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/retrying.py", line 189, in should_reject
    reject |= self._retry_on_exception(attempt.value[1])
TypeError: unsupported operand type(s) for |=: 'bool' and 'error'
```

Ref:
https://github.com/StackStorm/st2/issues/3601


Discussion:
@johnarnold Thanks for the issue but if the service you use is not an officially supported one, we'll not prioritize it over other issues. Also, the bug you are pointing to is in a third party library. If you can help them out with a fix, that'd be great. Since this is not a direct st2 issue, I am going to go ahead and close the issue. Supporting redis for message queue is not supported and not something we'll support in the near future. 
@lakshmi-kannan 

I'm pointing to an issue with register_exchanges in bootstrap_utils.py returning an 'error' instead of a bool -- I don't believe that's proper behavior for the callback.

I believe that could happen regardless of transport.  Also doesn't appear to be an issue with retrying.  Do you want to reactivate the issue?

Also, I'm a bit surprised that you would close an issue out of hand.  Does this project not want community contribution of enhancements?
@johnarnold We didn't want to come out this way - we definitely appreciate and value user contributions.

The problem lakshmi wanted to point out is that we don't officially support Redis as a message bus transport. Even if we managed to fix those couple of issues, who knows how many more issues are hiding underneath.

There are countless of software alternatives (detabases, web servers, etc.) we could officially support, but we simply don't have resources to do it and it's better to focus and officially support one small set which we can test, etc. and we know works well.

Having said that, if you manage to get it to work, we would happily merge the fixes as long as they don't interfere with other code.
@johnarnold #3638 should fix the actual retrying issue - thanks for reporting this and a good catch.

I also tried to see what's the deal with Redis transport and it already breaks very early one - for some reason even`exchange_declare` doesn't work (exchange / key is not created in Redis) which is a very basic functionality so the whole issue of supporting Redis is probably a very deep rabbit hole.
So the actual "exchange not exist issue" I mentioned above is related to queue not being declared yet aka not all workers and services being online yet.

This seems to expose different behavior between RabbitMQ and Redis transports - with RabbitMQ it works just fine if not all the workers are online yet and queues are not yet declared, but with Redis transport it seems like queues need to be declared up front for it to work.
I actually managed to get it to work by pre-declaring all the queue on service setup.

Will open a PR with that change shortly. The feature itself will be behind a feature flag and out official recommendation still is to use RabbitMQ because who knows how many issues are still hiding underneath when using Redis backend.

Resolving patch:
diff --git a/st2common/st2common/config.py b/st2common/st2common/config.py
--- a/st2common/st2common/config.py
+++ b/st2common/st2common/config.py
@@ -155,7 +155,11 @@ def register_opts(ignore_errors=False):
         cfg.IntOpt('connection_retries', default=10,
                    help='How many times should we retry connection before failing.'),
         cfg.IntOpt('connection_retry_wait', default=10000,
-                   help='How long should we wait between connection retries.')
+                   help='How long should we wait between connection retries.'),
+        cfg.BoolOpt('predeclare_queues', default=False,
+                   help=('Set this to True to pre-declare all the AMQP queues on service startup. '
+                         'This is required with some non-standard kombu transports such as the '
+                         'Redis one.'))
     ]
     do_register_opts(messaging_opts, 'messaging', ignore_errors)
 
diff --git a/st2common/st2common/transport/bootstrap_utils.py b/st2common/st2common/transport/bootstrap_utils.py
--- a/st2common/st2common/transport/bootstrap_utils.py
+++ b/st2common/st2common/transport/bootstrap_utils.py
@@ -13,10 +13,12 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import retrying
 import socket
+
+import retrying
 from oslo_config import cfg
 from kombu import Connection
+
 from st2common import log as logging
 from st2common.transport import utils as transport_utils
 from st2common.transport.actionexecutionstate import ACTIONEXECUTIONSTATE_XCHG
@@ -26,16 +28,47 @@
 from st2common.transport.liveaction import LIVEACTION_XCHG, LIVEACTION_STATUS_MGMT_XCHG
 from st2common.transport.reactor import SENSOR_CUD_XCHG
 from st2common.transport.reactor import TRIGGER_CUD_XCHG, TRIGGER_INSTANCE_XCHG
+from st2common.transport import actionexecutionstate
+from st2common.transport import announcement
+from st2common.transport import execution
+from st2common.transport import liveaction
+from st2common.transport import reactor
 
 LOG = logging.getLogger('st2common.transport.bootstrap')
 
 __all__ = [
-    'register_exchanges'
+    'register_exchanges',
+
+    'EXCHANGES',
+    'QUEUES'
 ]
 
-EXCHANGES = [ACTIONEXECUTIONSTATE_XCHG, ANNOUNCEMENT_XCHG, EXECUTION_XCHG, LIVEACTION_XCHG,
-             LIVEACTION_STATUS_MGMT_XCHG, TRIGGER_CUD_XCHG, TRIGGER_INSTANCE_XCHG,
-             SENSOR_CUD_XCHG]
+# List of exchanges which are pre-declared on service set up.
+EXCHANGES = [
+    ACTIONEXECUTIONSTATE_XCHG,
+    ANNOUNCEMENT_XCHG,
+    EXECUTION_XCHG,
+    LIVEACTION_XCHG,
+    LIVEACTION_STATUS_MGMT_XCHG,
+    TRIGGER_CUD_XCHG,
+    TRIGGER_INSTANCE_XCHG,
+    SENSOR_CUD_XCHG
+]
+
+# List of queues which are pre-declared on service set up.
+# Because of the worker model used, this is required with some non-standard transports such as
+# Redis one. Even though kombu requires queues to be pre-declared upfront in such scenarios,
+# RabbitMQ transport doesn't require that.
+QUEUES = [
+    actionexecutionstate.get_queue(name='st2.preinit', routing_key='init'),
+    announcement.get_queue(name='st2.preinit', routing_key='init'),
+    execution.get_queue(name='st2.preinit', routing_key='init'),
+    liveaction.get_queue(name='st2.preinit', routing_key='init'),
+    liveaction.get_status_management_queue(name='st2.preinit', routing_key='init'),
+    reactor.get_trigger_cud_queue(name='st2.preinit', routing_key='init'),
+    reactor.get_trigger_instances_queue(name='st2.preinit', routing_key='init'),
+    reactor.get_sensor_cud_queue(name='st2.preinit', routing_key='init'),
+]
 
 
 def _do_register_exchange(exchange, connection, channel, retry_wrapper):
@@ -59,6 +92,21 @@ def _do_register_exchange(exchange, connection, channel, retry_wrapper):
         LOG.exception('Failed to register exchange: %s.', exchange.name)
 
 
+def _do_predeclare_queue(channel, queue):
+    LOG.debug('Predeclaring queue for exchange "%s"' % (queue.exchange.name))
+
+    bound_queue = None
+
+    try:
+        bound_queue = queue(channel)
+        bound_queue.declare(nowait=False)
+        LOG.debug('Predeclared queue for exchange "%s"' % (queue.exchange.name))
+    except Exception:
+        LOG.exception('Failed to predeclare queue for exchange "%s"' % (queue.exchange.name))
+
+    return bound_queue
+
+
 def register_exchanges():
     LOG.debug('Registering exchanges...')
     connection_urls = transport_utils.get_messaging_urls()
@@ -73,6 +121,13 @@ def wrapped_register_exchanges(connection, channel):
 
         retry_wrapper.run(connection=conn, wrapped_callback=wrapped_register_exchanges)
 
+        def wrapped_predeclare_queues(connection, channel):
+            for queue in QUEUES:
+                _do_predeclare_queue(channel=channel, queue=queue)
+
+        if cfg.CONF.messaging.predeclare_queues:
+            retry_wrapper.run(connection=conn, wrapped_callback=wrapped_predeclare_queues)
+
 
 def register_exchanges_with_retry():
     def retry_if_io_error(exception):


Test patch:
diff --git a/st2common/tests/unit/test_service_setup.py b/st2common/tests/unit/test_service_setup.py
--- a/st2common/tests/unit/test_service_setup.py
+++ b/st2common/tests/unit/test_service_setup.py
@@ -15,7 +15,12 @@
 
 import tempfile
 
+import mock
+from oslo_config import cfg
+
 from st2common import service_setup
+from st2common.transport.bootstrap_utils import register_exchanges
+from st2common.transport.bootstrap_utils import QUEUES
 
 from st2tests.base import CleanFilesTestCase
 from st2tests import config
@@ -87,3 +92,21 @@ def mock_get_logging_config_path():
                                 register_signal_handlers=False,
                                 register_internal_trigger_types=False,
                                 run_migrations=False)
+
+    @mock.patch('kombu.Queue.declare')
+    def test_register_exchanges_predeclare_queues(self, mock_declare):
+        # Verify that queues are correctly pre-declared if the corresponding config option is set
+
+        # Pre-declaration is disabled
+        self.assertEqual(mock_declare.call_count, 0)
+        cfg.CONF.set_override(group='messaging', name='predeclare_queues', override=False)
+
+        register_exchanges()
+        self.assertEqual(mock_declare.call_count, 0)
+
+        # Pre-declaration is enabled
+        self.assertEqual(mock_declare.call_count, 0)
+        cfg.CONF.set_override(group='messaging', name='predeclare_queues', override=True)
+
+        register_exchanges()
+        self.assertEqual(mock_declare.call_count, len(QUEUES))