Repository: celery/kombu

Issue:
Mongodb url is broken in Kombu 3.0.1
Upgrading to version 3.0.1, a mongodb url like mongodb://localhost/wq  fails as the // gets converted to / by some code in transport/mongodb.py  - basically it seems to boil down to this code:-

```
    if '/' in hostname[10:]:
        if not client.userid:
            hostname = hostname.replace('/' + client.virtual_host, '/')
        else:
            hostname = hostname.replace('/' + client.virtual_host, '/' + authdb)
```

by default virtual_host == '/' so basically hostname becomes:-

mongodb:/localhost/wq  instead of mongodb://localhost/wq causing the MongoClient to fail wit a port not an integer error.

right now can work around this by setting the url to mongodb:////localhost/wq but its a kludge.

mongodb transport no longer honors configured port.
The mongodb transport in kombu version 2.1.3 (version according to pypi) has changed the BrokerConnection arguments that it uses. We are passing in hostname, port, and transport in as separate BrokerConnection arguments, but the mongodb transport now expects to get all that information out of hostname, which it assumes is a mongodb:// URI.

For some history, version 1.5.0 changed the parse_url() function so that the port was no longer parsed out of mongodb:// URIs. So we were forced to pass the hostname, port, and transport in as separate BrokerConnection arguments. Now, we are being forced back to passing a mongodb:// URI in as the hostname.



Discussion:
Please try this then.  Pretty horrible, maybe there is a better way

Thanks - looking at the original code wouldn't the following code work equally as well and be less horrible :-)

```
if '/' in hostname[10:]:
    if not client.userid:
        hostname = hostname.replace('/' + db_name, '/')
    else:
        hostname = hostname.replace('/' + db_name, '/' + authdb)
```

Since this looks like what the code is trying to do - a bit further up the if the virtual_host = '/' or None, db_name is set to a default name so we never can get the // scenario.  The only gotcha in this is that you will need to specify a virtual_host param that matches the database name in the db url string. (but this is the case anyway with the way the code is now)

sorry I didn't suggest this yesterday, was a bit time pressured.

hope this helps ,

Phillip.

It's not my code and I'm not quite sure why it does like this, it could probably use urlparse and insert the virtual host if there isn't already one, that way you could specify it in the url directly.

yes - thats sounds like a good idea. Anyways its not a super high priority as its pretty easy to workaround, simply by removing the database name from the url and adding the name to the virtual_host field of the connection.

thanks for the speedy reply to my issue.

cheers,

Phillip

On 31/10/2013, at 10:51 AM, Ask Solem Hoel notifications@github.com wrote:

> It's not my code and I'm not quite sure why it does like this, it could probably use urlparse and insert the virtual host if there isn't already one, that way you could specify it in the url directly.
> 
> —
> Reply to this email directly or view it on GitHub.

That's right. I'll work on getting that back asap!

thnx for posting!

Was this fixed?


Resolving patch:
diff --git a/kombu/transport/mongodb.py b/kombu/transport/mongodb.py
--- a/kombu/transport/mongodb.py
+++ b/kombu/transport/mongodb.py
@@ -14,7 +14,7 @@
 
 from pymongo import errors
 from anyjson import loads, dumps
-from pymongo import MongoClient
+from pymongo import MongoClient, uri_parser
 
 from kombu.five import Empty
 from kombu.syn import _detect_environment
@@ -91,38 +91,52 @@ def _purge(self, queue):
             self.client.messages.remove({'queue': queue})
         return size
 
-    def _open(self, scheme='mongodb://'):
+    def _parse_uri(self, scheme='mongodb://'):
         # See mongodb uri documentation:
-        # http://www.mongodb.org/display/DOCS/Connections
+        # http://docs.mongodb.org/manual/reference/connection-string/
         client = self.connection.client
-        options = client.transport_options
-        hostname = client.hostname or DEFAULT_HOST
-        dbname = client.virtual_host
+        hostname = client.hostname
 
-        if dbname in ['/', None]:
-            dbname = "kombu_default"
         if not hostname.startswith(scheme):
             hostname = scheme + hostname
 
         if not hostname[len(scheme):]:
-            hostname += 'localhost'
-
-        # XXX What does this do?  [ask]
-        urest = hostname[len(scheme):]
-        if '/' in urest:
-            if not client.userid:
-                urest = urest.replace('/' + client.virtual_host, '/')
-                hostname = ''.join([scheme, urest])
-
-        # At this point we expect the hostname to be something like
-        # (considering replica set form too):
-        #
-        #   mongodb://[username:password@]host1[:port1][,host2[:port2],
-        #   ...[,hostN[:portN]]][/[?options]]
-        options.setdefault('auto_start_request', True)
+            hostname += DEFAULT_HOST
+
+        if client.userid and '@' not in hostname:
+            head, tail = hostname.split('://')
+
+            credentials = client.userid
+            if client.password:
+                credentials += ':' + client.password
+
+            hostname = head + '://' + credentials + '@' + tail
+
+        port = client.port if client.port is not None else DEFAULT_PORT
+
+        parsed = uri_parser.parse_uri(hostname, port)
+
+        dbname = parsed['database'] or client.virtual_host
+
+        if dbname in ('/', None):
+            dbname = 'kombu_default'
+
+        options = {'auto_start_request': True,
+                   'ssl': client.ssl,
+                   'connectTimeoutMS': int(client.connect_timeout * 1000)
+                                            if client.connect_timeout else None}
+        options.update(client.transport_options)
+        options.update(parsed['options'])
+
+        return hostname, dbname, options
+
+    def _open(self, scheme='mongodb://'):
+        hostname, dbname, options = self._parse_uri(scheme=scheme)
+
         mongoconn = MongoClient(
-            host=hostname, ssl=client.ssl,
+            host=hostname, ssl=options['ssl'],
             auto_start_request=options['auto_start_request'],
+            connectTimeoutMS=options['connectTimeoutMS'],
             use_greenlets=_detect_environment() != 'default',
         )
         database = getattr(mongoconn, dbname)


Test patch:
diff --git a/kombu/tests/transport/test_mongodb.py b/kombu/tests/transport/test_mongodb.py
--- a/kombu/tests/transport/test_mongodb.py
+++ b/kombu/tests/transport/test_mongodb.py
@@ -13,35 +13,67 @@ def __setattr__(self, key, value):
 
 class test_mongodb(Case):
 
-    @skip_if_not_module('pymongo')
-    def test_url_parser(self):
+    def _get_connection(self, url, **kwargs):
         from kombu.transport import mongodb
-        from pymongo.errors import ConfigurationError
-
-        raise SkipTest(
-            'Test is functional: it actually connects to mongod')
 
         class Transport(mongodb.Transport):
             Connection = MockConnection
 
+        return Connection(url, transport=Transport, **kwargs).connect()
+
+    @skip_if_not_module('pymongo')
+    def test_defaults(self):
         url = 'mongodb://'
-        c = Connection(url, transport=Transport).connect()
-        client = c.channels[0].client
-        self.assertEquals(client.name, 'kombu_default')
-        self.assertEquals(client.connection.host, '127.0.0.1')
 
+        c = self._get_connection(url)
+        hostname, dbname, options = c.channels[0]._parse_uri()
+
+        self.assertEquals(dbname, 'kombu_default')
+        self.assertEquals(hostname, 'mongodb://127.0.0.1')
+
+    @skip_if_not_module('pymongo')
+    def test_custom_host(self):
         url = 'mongodb://localhost'
-        c = Connection(url, transport=Transport).connect()
-        client = c.channels[0].client
-        self.assertEquals(client.name, 'kombu_default')
+        c = self._get_connection(url)
+        hostname, dbname, options = c.channels[0]._parse_uri()
 
+        self.assertEquals(dbname, 'kombu_default')
+
+    @skip_if_not_module('pymongo')
+    def test_custom_database(self):
         url = 'mongodb://localhost/dbname'
-        c = Connection(url, transport=Transport).connect()
-        client = c.channels[0].client
-        self.assertEquals(client.name, 'dbname')
+        c = self._get_connection(url)
+        hostname, dbname, options = c.channels[0]._parse_uri()
+
+        self.assertEquals(dbname, 'dbname')
+
+    @skip_if_not_module('pymongo')
+    def test_custom_credentions(self):
+        url = 'mongodb://localhost/dbname'
+        c = self._get_connection(url, userid='foo', password='bar')
+        hostname, dbname, options = c.channels[0]._parse_uri()
+
+        self.assertEquals(hostname, 'mongodb://foo:bar@localhost/dbname')
+        self.assertEquals(dbname, 'dbname')
+
+    @skip_if_not_module('pymongo')
+    def test_options(self):
+        url = 'mongodb://localhost,localhost2:29017/dbname?safe=true'
+        c = self._get_connection(url)
+
+        hostname, dbname, options = c.channels[0]._parse_uri()
+
+        self.assertEqual(options['safe'], True)
+
+    @skip_if_not_module('pymongo')
+    def test_real_connections(self):
+        from pymongo.errors import ConfigurationError
+
+        raise SkipTest(
+            'Test is functional: it actually connects to mongod')
 
         url = 'mongodb://localhost,localhost:29017/dbname'
-        c = Connection(url, transport=Transport).connect()
+        c = self._get_connection(url)
         client = c.channels[0].client
 
         nodes = client.connection.nodes
@@ -51,37 +83,32 @@ class Transport(mongodb.Transport):
             self.assertTrue(('localhost', 29017) in nodes)
             self.assertEquals(client.name, 'dbname')
 
-        # Passing options breaks kombu's _init_params method
-        # url = 'mongodb://localhost,localhost2:29017/dbname?safe=true'
-        # c = Connection(url, transport=Transport).connect()
-        # client = c.channels[0].client
-
         url = 'mongodb://localhost:27017,localhost2:29017/dbname'
-        c = Connection(url, transport=Transport).connect()
+        c = self._get_connection(url)
         client = c.channels[0].client
 
         # Login to admin db since there's no db specified
         url = "mongodb://adminusername:adminpassword@localhost"
-        c = Connection(url, transport=Transport).connect()
+        c = self._get_connection()
         client = c.channels[0].client
         self.assertEquals(client.name, "kombu_default")
 
         # Lets make sure that using admin db doesn't break anything
         # when no user is specified
         url = "mongodb://localhost"
-        c = Connection(url, transport=Transport).connect()
+        c = self._get_connection(url)
         client = c.channels[0].client
 
         # Assuming there's user 'username' with password 'password'
         # configured in mongodb
         url = "mongodb://username:password@localhost/dbname"
-        c = Connection(url, transport=Transport).connect()
+        c = self._get_connection(url)
         client = c.channels[0].client
 
         # Assuming there's no user 'nousername' with password 'nopassword'
         # configured in mongodb
         url = "mongodb://nousername:nopassword@localhost/dbname"
-        c = Connection(url, transport=Transport).connect()
+        c = self._get_connection(url)
 
         # Needed, otherwise the error would be rose before
         # the assertRaises is called