Repository: StackStorm/st2

Issue:
Clearer Error Logs when RabbitMQ not reachable
If RabbitMQ is not running, users see logs like this in st2api.log:

```
2018-02-06 17:44:10,549 140082567223888 INFO __init__ [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "None".
2018-02-06 17:44:10,654 140082567223888 ERROR connection_retry_wrapper [-] Connection or channel error identified: [Errno 111] ECONNREFUSED.
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/transport/connection_retry_wrapper.py", line 115, in run
    channel = connection.channel()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 266, in channel
    chan = self.transport.create_channel(self.connection)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 802, in connection
    self._connection = self._establish_connection()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/connection.py", line 757, in _establish_connection
    conn = self.transport.establish_connection()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
    conn.connect()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/connection.py", line 282, in connect
    self.transport.connect()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/transport.py", line 109, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/amqp/transport.py", line 150, in _connect
    self.sock.connect(sa)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 256, in connect
    socket_checkerr(fd)
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/eventlet/greenio/base.py", line 46, in socket_checkerr
    raise socket.error(err, errno.errorcode[err])
error: [Errno 111] ECONNREFUSED
```

This is confusing for users for two reasons. One is because it is immediately below a "Connecting to database" message. Issue #3990 addresses that. The other issue is that the ERROR message here does not make it obvious that it is a problem with connecting to RabbitMQ. This should be more obvious in the log message. 


Discussion:

Resolving patch:
diff --git a/st2common/st2common/transport/connection_retry_wrapper.py b/st2common/st2common/transport/connection_retry_wrapper.py
--- a/st2common/st2common/transport/connection_retry_wrapper.py
+++ b/st2common/st2common/transport/connection_retry_wrapper.py
@@ -117,7 +117,7 @@ def run(self, connection, wrapped_callback):
                 wrapped_callback(connection=connection, channel=channel)
                 should_stop = True
             except connection.connection_errors + connection.channel_errors as e:
-                self._logger.exception('Connection or channel error identified: %s.' % (str(e)))
+                self._logger.exception('RabbitMQ connection or channel error: %s.' % (str(e)))
                 should_stop, wait = self._retry_context.test_should_stop()
                 # reset channel to None to avoid any channel closing errors. At this point
                 # in case of an exception there should be no channel but that is better to


Test patch: