Repository: MongoEngine/mongoengine

Issue:
“AttributeError: ‘_thread._local’ object has no attribute ‘no_dereferencing_class’ in Multithreaded Environment”
Hi,

### Details : 

In my project, I’m using ASGI’s **sync_to_async** utility. I believe this utility creates multiple threads using a thread pool executor, and each thread has its own thread-local storage. The attribute **no_dereferencing_class**, set by MongoEngine in **context_managers.py**, is causing an error: ‘_thread._local’ object has no attribute ‘no_dereferencing_class’. This issue is observed in the context_managers.py file.

```python
thread_locals = threading.local()
thread_locals.no_dereferencing_class = {}


def no_dereferencing_active_for_class(cls):
    return cls in thread_locals.no_dereferencing_class
```

Tested on versions:
- Python `3.12.1`
- MongoEngine `0.28.0`
- PyMongo `4.6.2`





Discussion:
Argh this change was brought in the last release.
Could you provide a reproducible snippet containing minimal amount of code? 
I have the same error with eventlet-0.35.2 when running mongoengine-0.28.0 and flask_track_usage-2.0.0.

Fallback to mongoengine-**0.27.0** help remove the **no_dereferencing_class** error:

```
  File "/home/pyuser/app/dist-packages/flask_track_usage/summarization/__init__.py", line 53, in _caller
    method(**kwargs)
  File "/home/pyuser/app/dist-packages/flask_track_usage/summarization/mongoenginestorage.py", line 139, in sumUrl
    increment(sumUrlClasses, src, "url", ["url"])
  File "/home/pyuser/app/dist-packages/flask_track_usage/summarization/mongoenginestorage.py", line 38, in increment
    doc = class_dict[period].objects(date=times[period], **db_args).first()
  File "/home/pyuser/app/dist-packages/mongoengine/queryset/base.py", line 301, in first
    result = queryset[0]
  File "/home/pyuser/app/dist-packages/mongoengine/queryset/base.py", line 206, in __getitem__
    _auto_dereference=self._auto_dereference,
  File "/home/pyuser/app/dist-packages/mongoengine/queryset/base.py", line 1768, in _auto_dereference
    should_deref = not no_dereferencing_active_for_class(self._document)
  File "/home/pyuser/app/dist-packages/mongoengine/context_managers.py", line 28, in no_dereferencing_active_for_class
    return cls in thread_locals.no_dereferencing_class
  File "/home/pyuser/app/dist-packages/eventlet/corolocal.py", line 45, in __getattribute__
    return object.__getattribute__(self, attr)
AttributeError: 'local' object has no attribute 'no_dereferencing_class'
```

Thanks!
```
# activate eventlet
import os
import eventlet
eventlet.monkey_patch()

from flask_track_usage.storage.mongo import MongoEngineStorage
from mongoengine import connect
```
I am facing the same issue after updating to the latest version ` 0.28.0`.  The issue occurs only in the latest version, downgrading to version `0.27.0` seems to solve the issue.

sample code from my project

```python
@auth_bp.route('/login', methods=['POST'])
def login():
    data = request.get_json()
    email = data.get('email')
    password = data.get('password')
    user = User.objects(email=email).first()
    if user and check_password_hash(user.password, password):
        return make_response('Logged in', 200)
    return make_response('Invalid email or password', 401)
```
The issue occurs where ever there is a query in the code.
Here is a simple code snippet to reproduce the said issue using asgiref sync to async with **ThreadPoolExecuter**
```python
import asyncio
from concurrent.futures import ThreadPoolExecutor
from functools import partial
import mongoengine
from asgiref.sync import sync_to_async
from mongoengine import Document, StringField

mongoengine.connect(
    host="mongodb://localhost:27017/",
    db="test_db",
)


class UserModel(Document):
    meta = {
        "collection": "users",
    }
    name = StringField()


user_a = UserModel(name="JothirAdithyan")
UserModel.objects.insert(user_a)


async def get_docs():
    function = partial(UserModel.objects.first)
    user1 = await sync_to_async(func=function, thread_sensitive=False, executor=ThreadPoolExecutor())()
    print(user1.to_json())

if __name__ == "__main__":
    asyncio.run(get_docs())

```


Here is an example using just Flask (no async).

tl;dr: Downgrade to mongoengine 0.27.0 and it works

### requirements.txt
```
Flask==3.0.2
mongoengine==0.28.0
```

### app.py
```from flask import Flask

app = Flask(__name__)

from flask import Flask
from mongoengine import connect
from mongoengine import Document, StringField

app = Flask(__name__)
app.config['MONGODB_SETTINGS'] = {
    'db': 'my_database',
    'host': 'mongodb://localhost:20000/my_database'
}
connect(db='my_database', host='mongodb://localhost:20000/my_database')

class User(Document):
    user_id = StringField(required=True)
    name = StringField()
    email = StringField(required=True)

@app.route('/')
def index():
    # Create a new user
    user = User(user_id='john_doe', email='john@example.com')
    user.save()
    return 'User created successfully!'

@app.route('/users')
def get_users():
    # Retrieve all users
    users = User.objects.all()
    user_list = ', '.join([f"{user.user_id}: {user.email}" for user in users])
    return f'Users: {user_list}'


if __name__ == "__main__":
    app.run(debug=True, port=6001)
```

### Exception
```
Traceback (most recent call last):
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 1488, in __call__
    return self.wsgi_app(environ, start_response)
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 1466, in wsgi_app
    response = self.handle_exception(e)
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  File "/tmp/temp-flask-mongo/app.py", line 32, in get_users
    user_list = ', '.join([f"{user.user_id}: {user.email}" for user in users])
  File "/tmp/temp-flask-mongo/app.py", line 32, in <listcomp>
    user_list = ', '.join([f"{user.user_id}: {user.email}" for user in users])
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/mongoengine/queryset/queryset.py", line 109, in _iter_results
    self._populate_cache()
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/mongoengine/queryset/queryset.py", line 128, in _populate_cache
    self._result_cache.append(next(self))
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/mongoengine/queryset/base.py", line 1636, in __next__
    _auto_dereference=self._auto_dereference,
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/mongoengine/queryset/base.py", line 1768, in _auto_dereference
    should_deref = not no_dereferencing_active_for_class(self._document)
  File "/tmp/temp-flask-mongo/venv/lib/python3.10/site-packages/mongoengine/context_managers.py", line 28, in no_dereferencing_active_for_class
    return cls in thread_locals.no_dereferencing_class
AttributeError: '_thread._local' object has no attribute 'no_dereferencing_class'
```

### Fix
Downgrade mongoengine to `0.27.0`
Recently I'm getting the same error
Thanks for the reproducible snippet, working on a fix right now

Resolving patch:
diff --git a/mongoengine/__init__.py b/mongoengine/__init__.py
--- a/mongoengine/__init__.py
+++ b/mongoengine/__init__.py
@@ -29,7 +29,7 @@
 )
 
 
-VERSION = (0, 28, 0)
+VERSION = (0, 28, 1)
 
 
 def get_version():
diff --git a/mongoengine/context_managers.py b/mongoengine/context_managers.py
--- a/mongoengine/context_managers.py
+++ b/mongoengine/context_managers.py
@@ -20,8 +20,12 @@
 )
 
 
-thread_locals = threading.local()
-thread_locals.no_dereferencing_class = {}
+class MyThreadLocals(threading.local):
+    def __init__(self):
+        self.no_dereferencing_class = {}
+
+
+thread_locals = MyThreadLocals()
 
 
 def no_dereferencing_active_for_class(cls):


Test patch:
diff --git a/tests/test_context_managers.py b/tests/test_context_managers.py
--- a/tests/test_context_managers.py
+++ b/tests/test_context_managers.py
@@ -1,4 +1,7 @@
+import random
+import time
 import unittest
+from threading import Thread
 
 import pytest
 from bson import DBRef
@@ -18,6 +21,29 @@
 from tests.utils import MongoDBTestCase
 
 
+class TestableThread(Thread):
+    """
+    Wrapper around `threading.Thread` that propagates exceptions.
+
+    REF: https://gist.github.com/sbrugman/59b3535ebcd5aa0e2598293cfa58b6ab
+    """
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.exc = None
+
+    def run(self):
+        try:
+            super().run()
+        except BaseException as e:
+            self.exc = e
+
+    def join(self, timeout=None):
+        super().join(timeout)
+        if self.exc:
+            raise self.exc
+
+
 class TestContextManagers(MongoDBTestCase):
     def test_set_write_concern(self):
         class User(Document):
@@ -172,13 +198,27 @@ class Group(Document):
                 group = Group.objects.first()
                 assert isinstance(group.ref, DBRef)
 
-            # make sure its still off here
+            # make sure it's still off here
             group = Group.objects.first()
             assert isinstance(group.ref, DBRef)
 
         group = Group.objects.first()
         assert isinstance(group.ref, User)
 
+        def run_in_thread(id):
+            time.sleep(random.uniform(0.1, 0.5))  # Force desync of threads
+            if id % 2 == 0:
+                with no_dereference(Group):
+                    group = Group.objects.first()
+                    assert isinstance(group.ref, DBRef)
+            else:
+                group = Group.objects.first()
+                assert isinstance(group.ref, User)
+
+        threads = [TestableThread(target=run_in_thread, args=(id,)) for id in range(10)]
+        _ = [th.start() for th in threads]
+        _ = [th.join() for th in threads]
+
     def test_no_dereference_context_manager_dbref(self):
         """Ensure that DBRef items in ListFields aren't dereferenced"""