Repository: celery/kombu

Issue:
celery/kombu does not endure a mongodb failover
When using a MongoDB replica set as the broker for celery, celery can run regardless of which node is primary. So _that_ feature of 2.5 is definitely helpful. The problem is, celery doesn't come back from the mongodb failover itself. That is, when the current primary is lost or otherwise voted out by the other nodes, the celery processes simply go down. This ultimately means that while the replica set is doing it's job of keeping mongo available, celery has to be monitored and re-started when the switches occur. kombu version is 2.1.4.

Perhaps the thing to do is have kombu wait a little longer for the new primary to be elected on reconnect tries?



Discussion:
There's not specific AutoReconnect handling in the mongodb transport! I'll work on that so it'll be possible to wait(?) for a new primary before raising the exception.

@ask I'm wondering whether it should be celery handling the "kombu reconnection". We could throw a ConnectionError or something like that and let celery do the reconnect to the broker perhaps creating a new BrokerConnection instance ? 

@FlaPer87: Yeah, Celery already does this if it encounters one of `transport.connection_error`

It does so by using `BrokerConnection.ensure` + `ensure_connection`, which retries establishing a connection or other operations.  But it always uses the same BrokerConnection instance, this is why the real connection is an attribute (BrokerConnection.connection).

`BrokerConnection.transport.establish_connection`

The transport instance is the same during the connection lifetime even when a connection is retried, so the mongodb transport could do something like keep track
of the number of failures and

```
 class Transport:
     failures = 0

     def establish_connection(self):
           try:
               connect_to(self.round_robin(host_list))
           except self.connection_errors:
               self.failures += 1
               raise
           else:
               self.failures = 0
```

What would be the best way to work around this in Celery (or Kombu itself) for the moment?

Could I subclass the currently used MongoDB transport and implement the above mentioned hack?

```
Traceback (most recent call last):
  File "/opt/local/python/virtualenvs/app/lib/python2.7/site-packages/celery/task/trace.py", line 258, in trace_task
    store_result(uuid, retval, SUCCESS)
  File "/opt/local/python/virtualenvs/app/lib/python2.7/site-packages/celery/backends/base.py", line 257, in store_result
    self._store_result(task_id, result, status, traceback, **kwargs)
  File "/opt/local/python/virtualenvs/app/lib/python2.7/site-packages/celery/backends/mongodb.py", line 117, in _store_result
    self.collection.save(meta, safe=True)
  File "/opt/local/python/virtualenvs/app/lib/python2.7/site-packages/kombu/utils/__init__.py", line 294, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/opt/local/python/virtualenvs/app/lib/python2.7/site-packages/celery/backends/mongodb.py", line 214, in collection
    collection.ensure_index('date_done', background='true')
  File "/opt/local/python/virtualenvs/app/lib/python2.7/site-packages/pymongo/collection.py", line 783, in ensure_index
    return self.create_index(key_or_list, ttl, **kwargs)
  File "/opt/local/python/virtualenvs/app/lib/python2.7/site-packages/pymongo/collection.py", line 704, in create_index
    safe=True)
  File "/opt/local/python/virtualenvs/app/lib/python2.7/site-packages/pymongo/collection.py", line 306, in insert
    continue_on_error, self.__uuid_subtype), safe)
  File "/opt/local/python/virtualenvs/app/lib/python2.7/site-packages/pymongo/connection.py", line 748, in _send_message
    raise AutoReconnect(str(e))
AutoReconnect: not master
```

@FlaPer87 @ask 

we think, that a much better way to handle that is to use pymongo.connection.ReplicaSetConnection — something like that: http://github.com/mediasite/kombu/commit/c3b415762057a07be681fd692cba5bacdfc32c26

:+1: for `ReplicaSetConnection` use

:+1: for me too!

Did you run mongodb's related unit tests? (There've been some issues with uri parsing and mongodb connections in the past; sorry for the stupid question).

https://travis-ci.org/#!/mediasite/kombu/builds/3093962
is that enough? there are some issues with broker url and replica set name, we've decided to add it after #

that's a fix for that: https://github.com/mediasite/kombu/commit/16f1f8185a441d113d4b299c050cd0a11e9520b7
and travis log: https://travis-ci.org/#!/mediasite/kombu/builds/3094195

should i make pull request? everything works fine for us.

Unfortunately that isn't enough. Mongodb's transport tests are [skipped](https://github.com/celery/kombu/blob/master/kombu/tests/transport/test_mongodb.py#L18) if pymongo isn't installed and they also need a up and running mongodb [instance](https://github.com/celery/kombu/blob/master/kombu/tests/transport/test_mongodb.py#L23) (or 2 in order to test RS).

@FlaPer87 @ramm I forced pymongo into requests on a fork of the branch told Travis to start mongo, it works but fails on py 2.5 due to latest pymongo requiring parse_ql (which was added in 2.6):
https://travis-ci.org/1stvamp/kombu

Hello. I have a similar problem and I've tried solution suggested in
https://travis-ci.org/#!/mediasite/kombu/builds/3093962
But I had no sucess because "AutoReconnect: not master" exception is already handled by kombu. AutoReconnect is an ancestor of ConnectionFailure which is already present in connection_errors. So it is caught and kombu tries to reconnect to another server probing one after another until it finds master. And only then real error occurs. It is "KeyError: '<hostname>.pidbox'" and it is described in following issues https://github.com/celery/kombu/issues/150 https://github.com/celery/kombu/issues/155

@1stvamp  mmhh looks weird since there's no parse_qs neither in mongodb transport nor in pymongo :confused: 

What version are you using?

@kibergus The pidbox error is kind of mistic :stuck_out_tongue: but I'll dedicate some time to it later this week.

@FlaPer87 I literally added `pymongo` unpinned, so `2.4.2` in that travis worker most likely.
Ah, my bad, it was actual parse_qsl for the the URI connection string parsing:
`from urlparse import parse_qsl` (`2.4` branch of kombu)

https://travis-ci.org/1stvamp/kombu/jobs/4835834/#L124

@FlaPer87 can be avoided by doing this for parse_qsl for <2.6:

``` python
try:
    from urlparse import parse_qsl
except ImportError:
    from cgi import parse_qsl
```

There is no need to specially handle AutoReconnect or to use ReplicaSetConnection. ReplicaSetConnection would raise AutoReconnect exceptions too.
The problem is located in global variables in Channel class getting out of sink with class members. Here is a patch that works for me:

```
--- kombu/transport/mongodb.py  2013-03-06 20:10:31.388249993 +0400
+++ kombu/transport/mongodb.py  2013-03-06 20:10:53.838530683 +0400
@@ -49,6 +49,7 @@
     def _get(self, queue):
         try:
             if queue in self._fanout_queues:
+                self._ensure_cursor(queue)
                 msg = self._queue_cursors[queue].next()
                 self._queue_readcounts[queue] += 1
                 return loads(msg['payload'])
@@ -72,6 +73,7 @@

     def _size(self, queue):
         if queue in self._fanout_queues:
+            self._ensure_cursor(queue)
             return (self._queue_cursors[queue].count() -
                     self._queue_readcounts[queue])

@@ -181,12 +183,8 @@

     def _queue_bind(self, exchange, routing_key, pattern, queue):
         if self.typeof(exchange).type == 'fanout':
-            cursor = self.bcast.find(query={'queue': exchange},
-                                     sort=[('$natural', 1)], tailable=True)
-            # Fast forward the cursor past old events
-            self._queue_cursors[queue] = cursor.skip(cursor.count())
-            self._queue_readcounts[queue] = cursor.count()
             self._fanout_queues[queue] = exchange
+            self._ensure_cursor(queue)

         meta = {'exchange': exchange,
                 'queue': queue,
@@ -198,8 +196,9 @@
         self.routing.remove({'queue': queue})
         super(Channel, self).queue_delete(queue, **kwargs)
         if queue in self._fanout_queues:
-            self._queue_cursors[queue].close()
-            self._queue_cursors.pop(queue, None)
+            cursor = self._queue_cursors.pop(queue, None)
+            if cursor is not None:
+                cursor.close()
             self._fanout_queues.pop(queue, None)

     @property
@@ -208,6 +207,17 @@
             self._client = self._open()
         return self._client

+    def _ensure_cursor(self, queue):
+        self.client
+        exchange = self._fanout_queues[queue]
+        if queue not in self._queue_cursors:
+            cursor = self.bcast.find(query={'queue': exchange},
+                             sort=[('$natural', 1)], tailable=True)
+            # Fast forward the cursor past old events
+            count = cursor.count()
+            self._queue_cursors[queue] = cursor.skip(count)
+            self._queue_readcounts[queue] = count
+

 class Transport(virtual.Transport):
     Channel = Channel
```

But global variables are still there and strange things would happen if you try to open two Channels.

But a separate patch is needed for celery mongo result storage because it doesn't handle AutoReconnect exceptions too. I'll post it later.

What does
self.client
line from _ensure_cursor function do??

self.client is a method decorated with @property. It opens connection if it is not opened yet and sets some attributes such as self.db and self.bcast. So if you don't call self.cursor you can't use self.bcast.

Got it. Thanks.

To make things clear. I added kibergus lines and AutoreConnect handling as people mentioned above. So now I do not get errors in log, but also I have nothing else in there. Seems like celery does not work after rs.stepDown()

Sorry, my mistake. Actually, this complex solution doesn't work for me:

AutoReconnect: x.x.x.x:27017 is not primary or master, x.x.x.x:27017 is not primary or master

Errors should be in log. This patch does not remove them, it just fixes their handling. There should be a lot of AutoReconnect errors during 20 seconds after rs.stepDown(). After 20 seconds mongo would elect new master and celery will continue to work.
But if you use mongo result storage you must use this patch too:
https://github.com/celery/celery/issues/1260
Or celery would catch exceptions while writing result to storage. When this happens celery still runs but does not process any new tasks.

Beside that in your application you must wrap all calls to task invocation and result gathering with loops and try blocks. See celery mongo backend patch for a convenient decorator.

Can I wrap such calls to just

```
try:
    statements
except AutoReconnect:
    pass
```

?

If anyone is missing me in this discussion then it's not because I don't care, it's just that I have no experience with mongodb, I trust you to find a good solution.  If you need any help please ping me

Woops

Actually patch proposed in https://github.com/celery/kombu/issues/123#issuecomment-14527395 with a patched celery https://github.com/celery/celery/issues/1260 is working in production for about 4 weeks now and celery works fine, it does not switch off on mongodb failover.

Beside these patches we wrap all calls to celery functions with a special auto_retry decorator. You can find a slightly simplified version of it in https://github.com/celery/celery/issues/1260 . We also had to write MoreReliableMongoConnection class which retries all idempotent queries to mongo and use  it in our code. It does not relate to kombu or celery directly, but there is no much use of stable working celery when your http server has crashed.

There is still a problem with mongos, proposed patches work good when you connect directly to mongod, but would not help when you connect through mongos because error reporting is broken in it. Interested ones can read more in pymongo discussion https://github.com/mongodb/mongo-python-driver/pull/170#issuecomment-16096161

Another issue is that when mongod totally fails, celery and kombu will infinitely continue to try to connect to it. Probably some delays should be added. May be some developers would like a timeout. But I'm fine with this solution because when mongod would be revived my service will continue to work.


Resolving patch:
diff --git a/kombu/transport/mongodb.py b/kombu/transport/mongodb.py
--- a/kombu/transport/mongodb.py
+++ b/kombu/transport/mongodb.py
@@ -25,10 +25,37 @@
 DEFAULT_HOST = '127.0.0.1'
 DEFAULT_PORT = 27017
 
-__author__ = """\
-Flavio [FlaPer87] Percoco Premoli <flaper87@flaper87.org>;\
-Scott Lyons <scottalyons@gmail.com>;\
-"""
+DEFAULT_MESSAGES_COLLECTION = 'messages'
+DEFAULT_ROUTING_COLLECTION = 'messages.routing'
+DEFAULT_BROADCAST_COLLECTION = 'messages.broadcast'
+
+
+class BroadcastCursor(object):
+    '''Cursor for broadcast queues.'''
+
+    def __init__(self, cursor):
+        # Fast forward the cursor past old events
+        self._cursor = cursor.skip(cursor.count())
+        self._offset = cursor.count()
+
+    def get_size(self):
+        return self._cursor.count() - self._offset
+
+    def close(self):
+        self._cursor.close()
+
+    def purge(self):
+        self._cursor.rewind()
+        self._cursor = self._cursor.skip(self._cursor.count())
+
+    def __iter__(self):
+        return self
+
+    def next(self):
+        msg = next(self._cursor)
+        self._offset += 1
+
+        return msg
 
 
 class Channel(virtual.Channel):
@@ -37,58 +64,47 @@ class Channel(virtual.Channel):
     _fanout_queues = {}
 
     def __init__(self, *vargs, **kwargs):
-        super_ = super(Channel, self)
-        super_.__init__(*vargs, **kwargs)
+        super(Channel, self).__init__(*vargs, **kwargs)
 
-        self._queue_cursors = {}
-        self._queue_readcounts = {}
+        self._broadcast_cursors = {}
 
     def _new_queue(self, queue, **kwargs):
         pass
 
     def _get(self, queue):
-        try:
-            if queue in self._fanout_queues:
-                msg = next(self._queue_cursors[queue])
-                self._queue_readcounts[queue] += 1
-                return loads(bytes_to_str(msg['payload']))
-            else:
-                msg = self.client.command(
-                    'findandmodify', 'messages',
-                    query={'queue': queue},
-                    sort={'_id': pymongo.ASCENDING}, remove=True,
-                )
-        except errors.OperationFailure as exc:
-            if 'No matching object found' in exc.args[0]:
-                raise Empty()
-            raise
-        except StopIteration:
-            raise Empty()
+        if queue in self._fanout_queues:
+            try:
+                msg = next(self.get_broadcast_cursor(queue))
+            except StopIteration:
+                msg = None
+        else:
+            msg = self.get_messages().find_and_modify(query={'queue': queue},
+                                                      sort={'_id': pymongo.ASCENDING},
+                                                      remove=True)
 
-        # as of mongo 2.0 empty results won't raise an error
-        if msg['value'] is None:
+        if msg is None:
             raise Empty()
-        return loads(bytes_to_str(msg['value']['payload']))
+
+        return loads(bytes_to_str(msg['payload']))
 
     def _size(self, queue):
         if queue in self._fanout_queues:
-            return (self._queue_cursors[queue].count() -
-                    self._queue_readcounts[queue])
+            return self.get_broadcast_cursor(queue).get_size()
 
-        return self.client.messages.find({'queue': queue}).count()
+        return self.get_messages().find({'queue': queue}).count()
 
     def _put(self, queue, message, **kwargs):
-        self.client.messages.insert({'payload': dumps(message),
-                                     'queue': queue})
+        self.get_messages().insert({'payload': dumps(message),
+                                    'queue': queue})
 
     def _purge(self, queue):
         size = self._size(queue)
+
         if queue in self._fanout_queues:
-            cursor = self._queue_cursors[queue]
-            cursor.rewind()
-            self._queue_cursors[queue] = cursor.skip(cursor.count())
+            self.get_broadcaset_cursor(queue).purge()
         else:
-            self.client.messages.remove({'queue': queue})
+            self.get_messages().remove({'queue': queue})
+
         return size
 
     def _parse_uri(self, scheme='mongodb://'):
@@ -141,7 +157,7 @@ def _open(self, scheme='mongodb://'):
             connectTimeoutMS=options['connectTimeoutMS'],
             use_greenlets=_detect_environment() != 'default',
         )
-        database = getattr(mongoconn, dbname)
+        database = mongoconn[dbname]
 
         version = mongoconn.server_info()['version']
         if tuple(map(int, version.split('.')[:2])) < (1, 3):
@@ -149,27 +165,30 @@ def _open(self, scheme='mongodb://'):
                 'Kombu requires MongoDB version 1.3+ (server is {0})'.format(
                     version))
 
-        self.db = database
-        col = database.messages
-        col.ensure_index([('queue', 1), ('_id', 1)], background=True)
+        self._create_broadcast(database, options)
+
+        self._client = database
 
-        if 'messages.broadcast' not in database.collection_names():
-            capsize = options.get('capped_queue_size') or 100000
-            database.create_collection('messages.broadcast',
-                                       size=capsize, capped=True)
+    def _create_broadcast(self, database, options):
+        '''Create capped collection for broadcast messages.'''
+        if DEFAULT_BROADCAST_COLLECTION in database.collection_names():
+            return
 
-        self.bcast = getattr(database, 'messages.broadcast')
-        self.bcast.ensure_index([('queue', 1)])
+        capsize = options.get('capped_queue_size') or 100000
+        database.create_collection(DEFAULT_BROADCAST_COLLECTION,
+                                   size=capsize, capped=True)
 
-        self.routing = getattr(database, 'messages.routing')
-        self.routing.ensure_index([('queue', 1), ('exchange', 1)])
-        return database
+    def _ensure_indexes(self):
+        '''Ensure indexes on collections.'''
+        self.get_messages().ensure_index([('queue', 1), ('_id', 1)], background=True)
+        self.get_broadcast().ensure_index([('queue', 1)])
+        self.get_routing().ensure_index([('queue', 1), ('exchange', 1)])
 
     #TODO: Store a more complete exchange metatable in the routing collection
     def get_table(self, exchange):
         """Get table of bindings for ``exchange``."""
         localRoutes = frozenset(self.state.exchanges[exchange]['table'])
-        brokerRoutes = self.client.messages.routing.find(
+        brokerRoutes = self.get_messages().routing.find(
             {'exchange': exchange}
         )
 
@@ -179,38 +198,71 @@ def get_table(self, exchange):
 
     def _put_fanout(self, exchange, message, routing_key, **kwargs):
         """Deliver fanout message."""
-        self.client.messages.broadcast.insert({'payload': dumps(message),
-                                               'queue': exchange})
+        self.get_broadcast().insert({'payload': dumps(message),
+                                     'queue': exchange})
 
     def _queue_bind(self, exchange, routing_key, pattern, queue):
         if self.typeof(exchange).type == 'fanout':
-            cursor = self.bcast.find(query={'queue': exchange},
-                                     sort=[('$natural', 1)], tailable=True)
-            # Fast forward the cursor past old events
-            self._queue_cursors[queue] = cursor.skip(cursor.count())
-            self._queue_readcounts[queue] = cursor.count()
+            self.create_broadcast_cursor(exchange, routing_key, pattern, queue)
             self._fanout_queues[queue] = exchange
 
         meta = {'exchange': exchange,
                 'queue': queue,
                 'routing_key': routing_key,
                 'pattern': pattern}
-        self.client.messages.routing.update(meta, meta, upsert=True)
+        self.get_routing().update(meta, meta, upsert=True)
 
     def queue_delete(self, queue, **kwargs):
-        self.routing.remove({'queue': queue})
+        self.get_routing().remove({'queue': queue})
+
         super(Channel, self).queue_delete(queue, **kwargs)
+
         if queue in self._fanout_queues:
-            self._queue_cursors[queue].close()
-            self._queue_cursors.pop(queue, None)
-            self._fanout_queues.pop(queue, None)
+            try:
+                cursor = self._broadcast_cursors.pop(queue)
+            except KeyError:
+                pass
+            else:
+                cursor.close()
+
+                self._fanout_queues.pop(queue)
 
     @property
     def client(self):
         if self._client is None:
-            self._client = self._open()
+            self._open()
+            self._ensure_indexes()
+
         return self._client
 
+    def get_messages(self):
+        return self.client[DEFAULT_MESSAGES_COLLECTION]
+
+    def get_routing(self):
+        return self.client[DEFAULT_ROUTING_COLLECTION]
+
+    def get_broadcast(self):
+        return self.client[DEFAULT_BROADCAST_COLLECTION]
+
+    def get_broadcast_cursor(self, queue):
+        try:
+            return self._broadcast_cursors[queue]
+        except KeyError:
+            # Cursor may be absent when Channel created more than one time.
+            # _fanout_queues is the class-level mutable attrbite so it is shared over all
+            # Channel instances.
+            return self.create_broadcast_cursor(self._fanout_queues[queue], None, None, queue)
+
+    def create_broadcast_cursor(self, exchange, routing_key, pattern, queue):
+        cursor = self.get_broadcast().find(query={'queue': exchange},
+                                           sort=[('$natural', 1)], tailable=True)
+
+        broadcast_cursor = BroadcastCursor(cursor)
+
+        self._broadcast_cursors[queue] = broadcast_cursor
+
+        return broadcast_cursor
+
 
 class Transport(virtual.Transport):
     Channel = Channel


Test patch: