Repository: MongoEngine/mongoengine

Issue:
Connection closing instantly with mongoengine 0.24 and pymongo >= 4.0
I am facing the following issue when trying to connect to my db using the last version of MongoEngine

 I think it is due to [this new behavior of pymongo 4](https://pymongo.readthedocs.io/en/stable/migrate-to-pymongo4.html#mongoclient-cannot-execute-operations-after-close)  that does not automatically reconnects when you try to use a `MongoClient` whose connection was closed

Except I get it directly after calling [connect](https://github.com/MongoEngine/mongoengine/blob/master/mongoengine/connection.py#L399), so the connection is not supposed to be close at this point, unless some code in connect closes the connection and is expecting MongoClient to automatically reconnect, as it would do in `pymongo==3.12.3`

```python
mongo_engine_connection_dict = {
    "db": os.environ["MONGODB_DATABASE"],
    "host": os.environ["MONGODB_NETWORK_IP"],
    "port": int(os.environ["MONGODB_PORT"]),
    "username": os.environ["MONGODB_USERNAME"],
    "password": os.environ["MONGODB_PASSWORD"],
    "tls": os.environ["MONGODB_TLS"] == "true" if "MONGODB_TLS" in os.environ else True,
    "tlsAllowInvalidCertificates": True,
    "authentication_source": "admin",
}
connect(**mongo_engine_connection_dict)

doc=Document.objects.get(id=12)
````

Returns the error : 
```pymongo.errors.InvalidOperation: Cannot use MongoClient after close```





Discussion:
This is weird, I'm not observing this but I'm also not using the tls options, did you find a workaround? Could you also try connecting with a uri (see [here](https://docs.mongoengine.org/guide/connecting.html#connect-with-uri-string)) to see if that makes a difference.
For what it's worth, I use the TLS options in my mongoengine configuration and I didn't observe this issue with pymongo 4, but I reverted back to pymongo 3.12 and haven't done much beyond a simple auth test. I'm never doing an explicit connect anywhere in my code. I think that's being done by flask-mongonegine.
@bagerard  for now my solution was to revert to `mongoengine==0.23.1` and `pymongo==3.12.3`  
I'll give it a try with an uri and report here
@LuisBlanche I think you should also try to understand why the connection gets closed on the first place, perhaps look in the logs of your mongo instance
@bagerard  yes I have tried this using the debugger and I think it is difficult to follow because of all the code around the concept of "alias" .. It is probably linked to the way the parameters are parsed and passed through to the `pymongo.MongoClient` object, but the thing is I don't get an  `Access Denied` error , just is seems that the connection does not live to see the next line of code 

Any updates on this one? Completely killed us out of the serverless game and is just painful to try to work through...@bagerard
It is probably related to the pymongo 4 upgrade, rather than mongoengine. Perhaps @ShaneHarvey (pymongo maintainer) can assist here ?
MongoClient only raises this error once it has been closed by an explicit call to client.close(). From a quick search it looks like the one and only place mongoengine calls close() is in [connection.disconnect()](https://github.com/MongoEngine/mongoengine/blob/v0.24.2/mongoengine/connection.py#L242).

The only way to reproduce this error is for the app to acquire the client from `connect()` and then manually close it (which you should not do):
```python
from mongoengine import *

mongo_engine_connection_dict = {
    "db": "test",
    "host": "localhost",
    "port": 27017,
    "username": "user",
    "password": "password",
    "tls": True,
    "tlsAllowInvalidCertificates": True,
    "authentication_source": "admin",
}
client = connect(**mongo_engine_connection_dict)

class User(Document):
    name = StringField(max_length=50)


User.objects.filter(name="Nunu").first()

client.close()
User.objects.filter(name="Nunu").first()  # Raises pymongo.errors.InvalidOperation: Cannot use MongoClient after close
```

One other way the app might be accidentally closing the client is by entering and exiting a with-statement (don't do this either):
```python
with connect(**mongo_engine_connection_dict):  # Client is automatically closed when exiting the with block.
    User.objects.filter(name="Nunu").first()

User.objects.filter(name="Nunu").first()  # Raises pymongo.errors.InvalidOperation: Cannot use MongoClient after close
```

You can use PDB to set a breakpoint on MongoClient.close to see where this is being called, like this:
```
$ python -m pdb repro-2627.py 
> /Users/shane/git/mongoengine/repro-2627.py(1)<module>()
-> from mongoengine import *
(Pdb) from pymongo import MongoClient
(Pdb) b MongoClient.close
Breakpoint 1 at /Users/shane/mongoengine/.venv3.7/lib/python3.7/site-packages/pymongo/mongo_client.py:1142
(Pdb) c
> /Users/shane/mongoengine/.venv3.7/lib/python3.7/site-packages/pymongo/mongo_client.py(1157)close()
-> session_ids = self._topology.pop_all_sessions()
(Pdb) bt
  /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/bdb.py(578)run()
-> exec(cmd, globals, locals)
  <string>(1)<module>()
  /Users/shane/git/mongoengine/repro-2627.py(20)<module>()
-> client.close()
> /Users/shane/mongoengine/.venv3.7/lib/python3.7/site-packages/pymongo/mongo_client.py(1157)close()
-> session_ids = self._topology.pop_all_sessions()
(Pdb) 
```
Any options how it can work with several calls of context manager?
> Any options how it can work with several calls of context manager?

I don't follow. Can you show a short example of what you want to do?
Aha! I discovered the real problem here while working on another issue. Here's a short repro:
```python
from mongoengine import *
from pymongo.errors import InvalidOperation

client1 = connect(alias='default')
client2 = connect(alias='non-default')
disconnect(alias='default')
try:
    client1.admin.command('ping')
except InvalidOperation:
    print('default client is closed')
try:
    client2.admin.command('ping')
except InvalidOperation:
    print('non-default client is also closed!')
```

The problem is that mongoengine will intentionally share a single MongoClient across multiple aliases if it determines the connection settings are equal (here https://github.com/MongoEngine/mongoengine/blob/4b5dbc7a52f95ae5525c08cee8ff9f743e8e677d/mongoengine/connection.py#L333-L334). The bug is that mongoengine does not remember this fact when disconnect() is called. So calling disconnect on one alias will close the client which might be used for others.

The fix is to skip calling MongoClient.close if this client instance is still in use by another alias. I'm working on the fix.

Resolving patch:
diff --git a/mongoengine/connection.py b/mongoengine/connection.py
--- a/mongoengine/connection.py
+++ b/mongoengine/connection.py
@@ -246,9 +246,13 @@ def disconnect(alias=DEFAULT_CONNECTION_NAME):
     from mongoengine import Document
     from mongoengine.base.common import _get_documents_by_db
 
-    if alias in _connections:
-        get_connection(alias=alias).close()
-        del _connections[alias]
+    connection = _connections.pop(alias, None)
+    if connection:
+        # Only close the client if we're removing the final reference.
+        # Use 'is' instead of 'in' or '==' to ensure the clients are
+        # the same instance.
+        if all(connection is not c for c in _connections.values()):
+            connection.close()
 
     if alias in _dbs:
         # Detach all cached collections in Documents


Test patch:
diff --git a/tests/test_connection.py b/tests/test_connection.py
--- a/tests/test_connection.py
+++ b/tests/test_connection.py
@@ -5,7 +5,11 @@
 import pytest
 from bson.tz_util import utc
 from pymongo import MongoClient, ReadPreference
-from pymongo.errors import InvalidName, OperationFailure
+from pymongo.errors import (
+    InvalidName,
+    InvalidOperation,
+    OperationFailure,
+)
 
 import mongoengine.connection
 from mongoengine import (
@@ -287,6 +291,30 @@ def test_disconnect_silently_pass_if_alias_does_not_exist(self):
         assert len(connections) == 0
         disconnect(alias="not_exist")
 
+    def test_disconnect_does_not_close_client_used_by_another_alias(self):
+        client1 = connect(alias="disconnect_reused_client_test_1")
+        client2 = connect(alias="disconnect_reused_client_test_2")
+        client3 = connect(alias="disconnect_reused_client_test_3", maxPoolSize=10)
+        assert client1 is client2
+        assert client1 is not client3
+        client1.admin.command("ping")
+        disconnect("disconnect_reused_client_test_1")
+        # The client is not closed because the second alias still exists.
+        client2.admin.command("ping")
+        disconnect("disconnect_reused_client_test_2")
+        # The client is now closed:
+        if PYMONGO_VERSION >= (4,):
+            with pytest.raises(InvalidOperation):
+                client2.admin.command("ping")
+        # 3rd client connected to the same cluster with different options
+        # is not closed either.
+        client3.admin.command("ping")
+        disconnect("disconnect_reused_client_test_3")
+        # 3rd client is now closed:
+        if PYMONGO_VERSION >= (4,):
+            with pytest.raises(InvalidOperation):
+                client3.admin.command("ping")
+
     def test_disconnect_all(self):
         connections = mongoengine.connection._connections
         dbs = mongoengine.connection._dbs