Repository: MongoEngine/mongoengine

Issue:
Can't read from multiple databases
I have an issue accessing multiple databases allocated in the same host one after another.

The thing is that I want to retrieve two different types of documents from two different databases with different authentication settings.

I have set up this example code.
```
from docs.doc1 import Doc1
from docs.doc2 import Doc2
from mongoengine import connect

db1_connection_dict = dict(
    alias='db1',
    db='db1',
    host='',
    port=27017,
    username='db1',
    password='db1',
    authentication_source='admin',
)

db2_connection_dict = dict(
    alias='db2',
    db='db2',
    host='',
    port=27017,
    username='db2',
    password='db2',
    authentication_source='admin',
)

Doc1._meta['db_alias'] = 'db1'
Doc2._meta['db_alias'] = 'db2'

connect(**db1_connection_dict)
connect(**db2_connection_dict)

read1_db1 = Doc1.objects().first()  # This works
read1_db2 = Doc2.objects().first()  # This works

read2_db1 = Doc1.objects().first()  # This crashes
```

I get an error as if the authorization wasn’t right.

I have inquired into this matter and the problem is that when it tries to query db1 for the second time, it is trying to access it with the db2 authentication settings. 
I have realized that by setting up a breakpoit before the second query to the first database and running `connectionStatus` command in the db1 database:
```
import mongoengine
mongoengine.connections._dbs[‘db1’].command(‘connectionStatus’)
```
And I get
`{'authInfo': {'authenticatedUsers': [{'user': 'db2', 'db': 'admin'}], 'authenticatedUserRoles': [{'role': 'read', 'db': 'db2'}]}, 'ok':     
1.0} `
Which are the db2 authentication settings.

If anyone could shed some light over this I would really appreciate it.

Thanks!


Discussion:
Hi @paco-cabrera,
Initially I thought it could be related to mongodb version but I just tried with 3.4 and 3.6 and I get the expected results by executing a code similar to yours.

By first loading 1 empty document in db1 with the following:
```
import pymongo
uri = "mongodb://db1:db1@127.0.0.1/?authSource=admin"
client = pymongo.MongoClient(uri)
client.db1.doc1.drop()
client.db1.doc1.insert({})

uri = "mongodb://db2:db2@127.0.0.1/?authSource=admin"
client = pymongo.MongoClient(uri)
client.db2.doc2.drop()
```

If I then run the following:
```
import mongoengine
db1_connection_dict = dict(
    alias='db1',
    db='db1',
    host='',
    port=27017,
    username='db1',
    password='db1',
    authentication_source='admin',
)

db2_connection_dict = dict(
    alias='db2',
    db='db2',
    host='',
    port=27017,
    username='db2',
    password='db2',
    authentication_source='admin',
)

class Doc1(mongoengine.Document):
    meta = {'db_alias': 'db1'}

class Doc2(mongoengine.Document):
    meta = {'db_alias': 'db2'}

mongoengine.connect(**db1_connection_dict)
mongoengine.connect(**db2_connection_dict)

assert isinstance(Doc1.objects().first(), Doc1)
print(mongoengine.connection._dbs['db1'].command('connectionStatus'))    # {u'authInfo': {u'authenticatedUsers': [{u'db': u'admin', u'user': u'db1'}], u'authenticatedUserRoles': [{u'db': u'admin', u'role': u'readWriteAnyDatabase'}, {u'db': u'admin', u'role': u'dbAdminAnyDatabase'}, {u'db': u'admin', u'role': u'userAdminAnyDatabase'}]}, u'ok': 1.0}
assert Doc2.objects().first() is None
print(mongoengine.connection._dbs['db1'].command('connectionStatus'))    # {u'authInfo': {u'authenticatedUsers': [{u'db': u'admin', u'user': u'db2'}], u'authenticatedUserRoles': [{u'db': u'admin', u'role': u'readWriteAnyDatabase'}, {u'db': u'admin', u'role': u'dbAdminAnyDatabase'}, {u'db': u'admin', u'role': u'userAdminAnyDatabase'}]}, u'ok': 1.0}

assert isinstance(Doc1.objects().first(), Doc1)
assert Doc2.objects().first() is None
```
The assertions don't go in error (thus mongoengine reads at the right place) but its true that the output of `connectionStatus` is strange. What happens behind the scene in mongoengine is that it uses the same MongoClient (as both dbs are in the same host) and then authenticates in the 2 different databases through this same client. This is allowed by pymongo. It looks like the output of `connectionStatus` gives you the latest authinfo but I don't think there is anything wrong there
Hi @bagerard, 
First of all thank you for the reply and the time you spent trying to solve the issue.

But I have run the same exact code you posted in your reply and I'm still getting the same authentication error when trying to do the second query to db1:
`OperationFailure: not authorized on db1 to execute command { find: "doc1", filter: {}, limit: 1, singleBatch: true, $db: "db1", $readPreference: { mode: "primaryPreferred" }  }`

There's a thing I shall comment, mongod security is enabled in my localhost, so you have to be correctly authorized before accessing any database.
By default local host mongo daemons have the security disabled, so maybe yours is deactivated too. I'm saying this because a colleague of mine said it worked, but when he turned on the authorization it crashed. 
If want to give it a try you have to open the config file of the mongo daemon (if you have ubuntu by default it is in `/etc/mongod.config`) and look for this lines, that most likely will be commented:
```
security:
  authorization: "enabled"
```
So if you want to replicate the error, you can follow this steps: 
Create 'db1' and 'db2' users in admin db with 'readWrite' roles in the respective databases, edit the `mongod.config`  uncomment the security lines (enable authorization), stop the mongo daemon and restart it. 

Most surely the code you wrote above will crash, when trying to query 'db1' for the second time.

If you try this, and have any thoughts on the matter I'd love to hear about them.

Thank you. 
Hi, I had security turned on during my test but both my users had `readWriteAnyDatabase` role so that's why it was working :).

If I set up the users' roles properly, I now see the exact same error as you.

The error relates to this, which is +- what's happening behind mongoengine:
```
import pymongo
client = pymongo.MongoClient()

client.db1.authenticate('db1', 'db1', source='admin')
client.db1.doc1.find_one()
client.db2.authenticate('db2', 'db2', source='admin')   # overwrite auth
client.db1.doc1.find_one()
```
When the host is the same, it uses the same MongoClient and then uses the .authenticate method of pymongo.Database to authenticate but it looks like the last .authenticate call overwrite the previous one.

`pymongo.Database.authenticate` method actually got deprecated in pymongo and the doc mentions that  

> .... Instead of using these methods to change credentials, pass credentials for one user to the MongoClient at construction time, and either grant access to several databases to one user account, or use a distinct client object for each user.

To fix this we would need to either:
- use a different client for every `alias`
- make it smart enough to understand better if it should create a new client or if it could reuse the existing one

**WORKAROUND**
mongoengine checks both `host` key of your connection settings to identify if it need to create an additional client. So if both your database are running locally, changing your settings to the following tricks mongoengine:
```
db1_connection_dict = dict(
    alias='db1',
    db='db1',
    host='127.0.0.1',    # changed
    port=27017,
    username='db1',
    password='db1',
    authentication_source='admin',
)

db2_connection_dict = dict(
    alias='db2',
    db='db2',
    host='',
    port=27017,
    username='db2',
    password='db2',
    authentication_source='admin',
)
```
Hello @bagerard!
I'm kind of glad that you found the error as I was beginning to think I might be going crazy or something :).

Thanks for the explanation as I understand now where does the error come from. 

I came up with the error when connecting to remote mongolatlas dbs hosts, but I have been able to implement your workaround by reordering the list of hosts in one connection; so thanks for that too.

Anyway, is this going to be fixed in the next update of the code? 
If so, let me know if I can contribute in some way, because using the workaround is fine for now, but I think would be nicer if it worked in a proper way. 

Again, thank you for everything!

Resolving patch:
diff --git a/mongoengine/connection.py b/mongoengine/connection.py
--- a/mongoengine/connection.py
+++ b/mongoengine/connection.py
@@ -235,7 +235,6 @@ def get_connection(alias=DEFAULT_CONNECTION_NAME, reconnect=False):
         raise MongoEngineConnectionError(msg)
 
     def _clean_settings(settings_dict):
-        # set literal more efficient than calling set function
         irrelevant_fields_set = {
             'name', 'username', 'password',
             'authentication_source', 'authentication_mechanism'
@@ -245,10 +244,12 @@ def _clean_settings(settings_dict):
             if k not in irrelevant_fields_set
         }
 
+    raw_conn_settings = _connection_settings[alias].copy()
+
     # Retrieve a copy of the connection settings associated with the requested
     # alias and remove the database name and authentication info (we don't
     # care about them at this point).
-    conn_settings = _clean_settings(_connection_settings[alias].copy())
+    conn_settings = _clean_settings(raw_conn_settings)
 
     # Determine if we should use PyMongo's or mongomock's MongoClient.
     is_mock = conn_settings.pop('is_mock', False)
@@ -262,35 +263,60 @@ def _clean_settings(settings_dict):
     else:
         connection_class = MongoClient
 
-    # Iterate over all of the connection settings and if a connection with
-    # the same parameters is already established, use it instead of creating
-    # a new one.
-    existing_connection = None
-    connection_settings_iterator = (
-        (db_alias, settings.copy())
-        for db_alias, settings in _connection_settings.items()
-    )
-    for db_alias, connection_settings in connection_settings_iterator:
-        connection_settings = _clean_settings(connection_settings)
-        if conn_settings == connection_settings and _connections.get(db_alias):
-            existing_connection = _connections[db_alias]
-            break
+    # Re-use existing connection if one is suitable
+    existing_connection = _find_existing_connection(raw_conn_settings)
 
     # If an existing connection was found, assign it to the new alias
     if existing_connection:
         _connections[alias] = existing_connection
     else:
-        # Otherwise, create the new connection for this alias. Raise
-        # MongoEngineConnectionError if it can't be established.
-        try:
-            _connections[alias] = connection_class(**conn_settings)
-        except Exception as e:
-            raise MongoEngineConnectionError(
-                'Cannot connect to database %s :\n%s' % (alias, e))
+        _connections[alias] = _create_connection(alias=alias,
+                                                 connection_class=connection_class,
+                                                 **conn_settings)
 
     return _connections[alias]
 
 
+def _create_connection(alias, connection_class, **connection_settings):
+    """
+    Create the new connection for this alias. Raise
+    MongoEngineConnectionError if it can't be established.
+    """
+    try:
+        return connection_class(**connection_settings)
+    except Exception as e:
+        raise MongoEngineConnectionError(
+            'Cannot connect to database %s :\n%s' % (alias, e))
+
+
+def _find_existing_connection(connection_settings):
+    """
+    Check if an existing connection could be reused
+
+    Iterate over all of the connection settings and if an existing connection
+    with the same parameters is suitable, return it
+
+    :param connection_settings: the settings of the new connection
+    :return: An existing connection or None
+    """
+    connection_settings_bis = (
+        (db_alias, settings.copy())
+        for db_alias, settings in _connection_settings.items()
+    )
+
+    def _clean_settings(settings_dict):
+        # Only remove the name but it's important to
+        # keep the username/password/authentication_source/authentication_mechanism
+        # to identify if the connection could be shared (cfr https://github.com/MongoEngine/mongoengine/issues/2047)
+        return {k: v for k, v in settings_dict.items() if k != 'name'}
+
+    cleaned_conn_settings = _clean_settings(connection_settings)
+    for db_alias, connection_settings in connection_settings_bis:
+        db_conn_settings = _clean_settings(connection_settings)
+        if cleaned_conn_settings == db_conn_settings and _connections.get(db_alias):
+            return _connections[db_alias]
+
+
 def get_db(alias=DEFAULT_CONNECTION_NAME, reconnect=False):
     if reconnect:
         disconnect(alias)


Test patch:
diff --git a/tests/test_connection.py b/tests/test_connection.py
--- a/tests/test_connection.py
+++ b/tests/test_connection.py
@@ -611,6 +611,16 @@ def test_multiple_connection_settings(self):
         self.assertEqual(mongo_connections['t1'].address[0], 'localhost')
         self.assertEqual(mongo_connections['t2'].address[0], '127.0.0.1')
 
+    def test_connect_2_databases_uses_same_client_if_only_dbname_differs(self):
+        c1 = connect(alias='testdb1', db='testdb1')
+        c2 = connect(alias='testdb2', db='testdb2')
+        self.assertIs(c1, c2)
+
+    def test_connect_2_databases_uses_different_client_if_different_parameters(self):
+        c1 = connect(alias='testdb1', db='testdb1', username='u1')
+        c2 = connect(alias='testdb2', db='testdb2', username='u2')
+        self.assertIsNot(c1, c2)
+
 
 if __name__ == '__main__':
     unittest.main()