MongoEngine/mongoengine — pull request #412 by fleat (2013-07-23 01:49:16+00:00)
Title: dereference instance not thread-safe

This is sort of a sequel to hmarr/mongoengine#399 I think. The problem I noticed was in a threaded environment where multiple threads are pulling the same type of document from the database, and the type has a ListField of ReferenceFields defined. When those threads dereferenced the fields on the documents at roughly the same time, sometimes the resulting list would contain some items that were properly dereferenced and some items that were still pymongo DBRefs.

I tracked it down to the ListField's \_dereference instance, which itself is not thread safe as it saves state during \__call__ and then references the state later in the execution path. Since all threads would use the same ListField instance on a given type this causes problems.

I believe the simplest example showing the problem is this:
```python
import threading
import time
from mongoengine import Document, StringField, ReferenceField, ListField

class Sibling(Document):
name = StringField()

class Person(Document):
name = StringField()
siblings = ListField(ReferenceField(Sibling))

def traverse(thread_num):
if thread_num > 0: # let 1st thread get through, avoids race condition
time.sleep(1)
deref = Person.siblings._dereference
print "Thread num %d, siblings dereferencer: %s" % (thread_num, deref)

threads = [
threading.Thread(target=traverse, args=(thread_nr,))
for thread_nr in xrange(2)
]

for thread in threads:
thread.start()
for thread in threads:
thread.join()
```

Note in the print output both threads are holding on to the same \_dereference instance.

Fix attached, just instantiates a DeReference instance during execution of \__get__ which I don't think incurs very much of a penalty as a DeReference is just a basic object. With this I no longer see the behavior I observed previously.

I'm happy to write a test for this but it's not straightforward. The existing tests all still work at least.