Repository: MongoEngine/mongoengine

Issue:
select_related() ConnectionError 
Hi,

I am having an interesting situation. We have been using MongoEngine in our project for a good few years now and this issue appeared out of nowhere (no update, no change to the code).

Calling `select_related()` on the results of a query raises a `mongoengine.connection.ConnectionError: You have not defined a default connection` Exception

The call that causes the issue:
```credentials = Credential.objects(id__in=ids).select_related()```

I noticed that I am getting the exception only the first time time I call `select_related()` and only on this object. Any subsequent calls (on the same collection) work as expected. 

While trying to debug, I got to this:
```
> /usr/local/lib/python2.7/dist-packages/mongoengine/dereference.py(129)_fetch_objects()
    127         object_map = {}
    128         for collection, dbrefs in self.reference_map.iteritems():
--> 129             if hasattr(collection, 'objects'):  # We have a document class for the refs
    130                 col_name = collection._get_collection_name()
    131                 refs = [dbref for dbref in dbrefs

ipdb> collection
<class 'insights.models.company.Company'>
ipdb> hasattr(collection, 'objects')
False
ipdb> hasattr(collection, 'objects')
True
```

Can this be caused by some lazy loading or some configuration issue?

Thank you!


Discussion:
can you provide sample code to reproduce this error?
Hi @erdenezul so sorry it took so long to reply. It took forever to figure out how to reproduce the issue on anything but our live environment and then to trim it down where it could be easily reproduced on other systems.

I am adding a zip file containing the collection dumps and the three models for the respective collections.

- `credentials`: 1 entry in collection
- `companies`: 4 entries in collection
- `users`: 1 entry in collection

Once imported into MongoDB, simply calling the following line will result in the exception being raised. Calling the same line again, will work as expected.
```
Credential.objects(id__in=['553a30b05552290a2d0481e9']).select_related()
``` 

Peculiar aspect:
The `companies` collection contains 4 items with 3 of them not being related to the credential. However, if we remove any of these items from the collection, the error does not show up. 

[files.zip](https://github.com/MongoEngine/mongoengine/files/1515185/files.zip)
I got no error
![screenshot from 2017-11-30 16-39-14](https://user-images.githubusercontent.com/176678/33421453-0addc0fa-d5ed-11e7-9896-9b1e7e525bf3.png)

I was able to reproduce using the files provided by @radulucaciu 
```
>>> from mongoengine import *
>>> connect('meta', 'meta-db')
MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, read_preference=Primary())
>>> from company import Company
>>> from credential import Credential
>>> from users import User
>>> Credential.objects(id__in=['553a30b05552290a2d0481e9']).select_related()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/mongoengine/queryset/base.py", line 756, in select_related
    return queryset._dereference(queryset, max_depth=max_depth)
  File "/Library/Python/2.7/site-packages/mongoengine/dereference.py", line 76, in __call__
    self.object_map = self._fetch_objects(doc_type=doc_type)
  File "/Library/Python/2.7/site-packages/mongoengine/dereference.py", line 149, in _fetch_objects
    references = get_db()[collection].find({'_id': {'$in': refs}})
  File "/Library/Python/2.7/site-packages/mongoengine/connection.py", line 226, in get_db
    conn = get_connection(alias)
  File "/Library/Python/2.7/site-packages/mongoengine/connection.py", line 146, in get_connection
    raise MongoEngineConnectionError(msg)
mongoengine.connection.MongoEngineConnectionError: You have not defined a default connection
>>> Credential.objects(id__in=['553a30b05552290a2d0481e9']).select_related()
[<Credential: Credential object>]
>>> 
```

Fresh install of everything (mongodb, pip, mongoengine), latest stable versions.

I wasn't sure how to import the collection data, so I manually created the database and connections and inserted each document (replacing `{"$oid": "..."}` with `ObjectId("...")`). I created a `mongodump`, attaching.
[meta.zip](https://github.com/MongoEngine/mongoengine/files/1517370/meta.zip)

cc @erdenezul 
Thank you @jozsi 

I wonder if this is influenced by the version of MongoDB - I am using v. 3.2.6

Package versions are:
```
mongoengine==0.14.3
pymongo==3.2.2
```

OK It was a tricky one, actually the problem lies in the Company class definition that conflicts with the existing data in the database.

The model you sent us is:
```
class Company(Document):
    meta = {
        'db_alias': 'meta-db',
        'collection': 'companies',
    }
    name = StringField(unique=True)
```
And the data in the db (restored from the file you attached):
```
In [3]: Company.objects.as_pymongo()
Out[3]: [{'_id': ObjectId('53a44802833be3e812e4ec3f'), 'name': 'WH'}, {'_id': ObjectId('59fc5b1b5998373e8c320ba2'), 'name': 'BMW'}, {'_id': ObjectId('5a02f42e5998373e8c320bc3'), 'name': 'MvP'}, {'_id': ObjectId('59fc55a75998373e8c320b9e'), 'name': 'BMW'}]
```
You actually have 2 companies with the same 'name' (being BMW) ubut your model defines a unique=True, which creates an index behind the scene.

You've debugged it properly until the line:
```
> /usr/local/lib/python2.7/dist-packages/mongoengine/dereference.py(129)_fetch_objects()
    127         object_map = {}
    128         for collection, dbrefs in self.reference_map.iteritems():
--> 129             if hasattr(collection, 'objects'):  # We have a document class for the refs
    130                 col_name = collection._get_collection_name()
    131                 refs = [dbref for dbref in dbrefs
```
What happens here (under python2):
Mongoengine will access .objects, which will call ._get_collection() and then .ensure_indexes() which will finally attempt to create the unique index there https://github.com/MongoEngine/mongoengine/blob/master/mongoengine/document.py#L885. 
The creation of the index fails and throws an exception but under Python2 hasattr swallows any exceptions (https://mail.python.org/pipermail/python-3000/2008-January/011771.html), thus it behaves like if hasattr simply returns False but it is actually swallowing an exception before returning False.

The behavior of hasattr changed in Python3 and if you run the same code in Python3, it fails as followed:
```
~/.pyenv/versions/3.6.5/envs/cmp/lib/python3.6/site-packages/mongoengine/document.py in ensure_indexes(cls)
    876                 if IS_PYMONGO_3:
--> 877                     collection.create_index(fields, background=background, **opts)
~/.pyenv/versions/3.6.5/envs/cmp/lib/python3.6/site-packages/pymongo/collection.py in create_index(self, keys, session, **kwargs)
-> 1958         self.__create_index(keys, kwargs, session, **cmd_options)
...
~/.pyenv/versions/3.6.5/envs/cmp/lib/python3.6/site-packages/pymongo/helpers.py in _check_command_response(response, msg, allowable_errors, parse_write_concern_error)
--> 148                 raise DuplicateKeyError(errmsg, code, response)

DuplicateKeyError: E11000 duplicate key error collection: meta.companies index: name_1 dup key: { : "BMW" }
``` 
This is the root cause of your issue. It sounds like the unique index was added to the Document class at a later stage.
What we can do to improve this is try/except the access to .objects without relying on hasattr, that way any exceptions will get to the user in both Python2 and Python3. I'll push a PR shortly to improve this

Resolving patch:
diff --git a/mongoengine/dereference.py b/mongoengine/dereference.py
--- a/mongoengine/dereference.py
+++ b/mongoengine/dereference.py
@@ -133,7 +133,12 @@ def _fetch_objects(self, doc_type=None):
         """
         object_map = {}
         for collection, dbrefs in self.reference_map.iteritems():
-            if hasattr(collection, 'objects'):  # We have a document class for the refs
+
+            # we use getattr instead of hasattr because as hasattr swallows any exception under python2
+            # so it could hide nasty things without raising exceptions (cfr bug #1688))
+            ref_document_cls_exists = (getattr(collection, 'objects', None) is not None)
+
+            if ref_document_cls_exists:
                 col_name = collection._get_collection_name()
                 refs = [dbref for dbref in dbrefs
                         if (col_name, dbref) not in object_map]


Test patch:
diff --git a/tests/document/instance.py b/tests/document/instance.py
--- a/tests/document/instance.py
+++ b/tests/document/instance.py
@@ -8,9 +8,12 @@
 
 from datetime import datetime
 from bson import DBRef, ObjectId
+from pymongo.errors import DuplicateKeyError
+
 from tests import fixtures
 from tests.fixtures import (PickleEmbedded, PickleTest, PickleSignalsTest,
                             PickleDynamicEmbedded, PickleDynamicTest)
+from tests.utils import MongoDBTestCase
 
 from mongoengine import *
 from mongoengine.base import get_document, _document_registry
@@ -30,12 +33,9 @@
 __all__ = ("InstanceTest",)
 
 
-class InstanceTest(unittest.TestCase):
+class InstanceTest(MongoDBTestCase):
 
     def setUp(self):
-        connect(db='mongoenginetest')
-        self.db = get_db()
-
         class Job(EmbeddedDocument):
             name = StringField()
             years = IntField()
@@ -3248,6 +3248,23 @@ class BlogPost(Document):
         blog.reload()
         self.assertEqual(blog.tags, [["value1", 123]])
 
+    def test_accessing_objects_with_indexes_error(self):
+        insert_result = self.db.company.insert_many([{'name': 'Foo'},
+                                                     {'name': 'Foo'}])  # Force 2 doc with same name
+        REF_OID = insert_result.inserted_ids[0]
+        self.db.user.insert_one({'company': REF_OID})   # Force 2 doc with same name
+
+        class Company(Document):
+            name = StringField(unique=True)
+
+        class User(Document):
+            company = ReferenceField(Company)
+
+
+        # Ensure index creation exception aren't swallowed (#1688)
+        with self.assertRaises(DuplicateKeyError):
+            User.objects().select_related()
+
 
 if __name__ == '__main__':
     unittest.main()