Repository: MongoEngine/mongoengine

Issue:
Can't query a GenericReferenceField using an ObjectId or DBRef?
I can't seem to figure out how to query a `GenericReferenceField` properly, based on (a list of) ObjectIds. I feel like I might be overlooking something very obvious, but the following scenario doesn't work for me?

``` python
    def test_query_genericreferencefield( self ):
        class TestActivity( Document ):
            name = StringField()
            owner = GenericReferenceField( required=True )

        class TestPerson( Document ):
            name = StringField()

        TestActivity.drop_collection()
        TestPerson.drop_collection()

        person = TestPerson( name="owner" )
        person.save()

        a1 = TestActivity( name="a1", owner=person )
        a1.save()

        # Doesn't work; same error as for a DBRef
        # activities = TestActivity.objects( owner=person.id )
        activities = TestActivity.objects( owner=DBRef( 'TestPerson', person.pk ) )

        self.assertEquals( activities[0].owner, person )
```

This gives the following error for both an ObjectId and a DBRef:

```
  File "C:\Python27\lib\site-packages\mongoengine\fields.py", line 1011, in to_mongo
    id_field_name = document.__class__._meta['id_field']
AttributeError: type object 'DBRef' has no attribute '_meta'
```

Edit: as noted below, you can of course work around this by using a raw query; however, I feel this shouldn't be necessary for simple cases like this? In my opinion, a `GenericReferenceField` should behave just like a regular `ReferenceField` does as much as possible.



Discussion:
@PaulUithol I believe you only need to do: `TestActivity.objects(owner=person)`. The deref happers internally.

Alternatively you can do a raw query with no dereferencing:

```
TestActivity.objects(__raw__={'owner._ref.$id': person.id})

# Or, to provide every possible solution ...
personId = str(person.id)
TestActivity.objects(__raw__={'owner._ref.$id': ObjectId(personId)})
```

Thanks, I know that. I don't have a document though; actually this is just a simplified test case from our application. I have a list of about 200 ObjectIds, and I need to fetch corresponding records from a GenericReferenceField on another collection.

What I need to do is more like `activities = TestActivity.objects( owner__in=[ person.id ] )`.

Ok in that case the `__raw__` mechanism _should_ work:

```
activities = TestActivity.objects( __raw__={'owner._ref.$id': {'$in': [ ObjectId(person.id) ]} })
```

Make sure about the person.id being converted into an ObjectID is all

Yeah, I know, thanks anyway :)

The point of this issue is more that, in my opinion, using a GenericReferenceField shouldn't be that different / harder than using a regular ReferenceField - mongoengine should offer the same interface where possible.

Ah apologies for my misunderstanding! I agree :)

I believe this actually broke between 0.8.4 and 0.8.7. I just tested with 0.8.7 and I can reference `owner=person`. The same fails in 0.8.7. Any chance you could confirm? I'm diffing the versions right now to see if I can submit a fix.

The following change is what caused this problem:

```
git diff v0.8.4..v0.8.7 -- mongoengine/fields.py

@@ -1004,7 +1023,10 @@ class GenericReferenceField(BaseField):
         id_ = id_field.to_mongo(id_)
         collection = document._get_collection_name()
         ref = DBRef(collection, id_)
-        return {'_cls': document._class_name, '_ref': ref}
+        return SON((
+            ('_cls', document._class_name),
+            ('_ref', ref)
+        ))
```

The change happened in 6db59a9c314827050b1e8a6ff746c5940b3c22da, which was related to Geo Location fields (#488). 

The problem is that this actually causes the subsequent query() to fail because the query ends up looking like (this is in my DB, which has `Media.media_for = GenericReferenceField()`):

```
dict: {'media_for': SON([('_cls', 'Users'), ('_ref', DBRef('users', ObjectId('4e9f46c26e48caa47f000124')))])}
```

AFAICT the Cursor is unable to process SON() - so the genericReference value is blank.

I don't understand the reason for switching from a dictionary to SON(), so it's possible that there's something going on here that I'm missing.

I'm submitting a pull request to revert this to dict vs SON().

First apologies for spamming this ticket - I've been trying to diagnose why I had this issue on some instances and not others. So finally created #577 since I believe this is a mongoengine bug, and it requires a different solution.

We're on 0.8.4 now (due to #570), but it still fails for us due to this line in `GenericReferenceField.to_mongo`:

``` python
id_field_name = document.__class__._meta['id_field']
```

Which fails for DBRef and ObjectIds, since those clearly don't have a `_meta` attribute.

I see. Looks like this is definitely a bug :(
Seems like this ticket is getting no love from Rozza, and 0.8.4+ won't work
with legacy data.
I'm going to try spend some time on solutions to these issues so that I can
upgrade to 0.8.7 at some point - there's no way I can continue with 0.8.4
with no upgrade path :(
Any suggestions you have would be very helpful (and thanks much for your
test cases - they really helped me to determine that I'm also suffering
from a variant of the problem).

On Wed, Feb 19, 2014 at 5:11 AM, Paul Uithol notifications@github.comwrote:

> We're on 0.8.4 now (due to #570https://github.com/MongoEngine/mongoengine/issues/570),
> but it still fails for us due to this line in
> GenericReferenceField.to_mongo:
> 
> id_field_name = document.**class**._meta['id_field']
> 
> Which fails for DBRef and ObjectIds, since those clearly don't have a
> _meta attribute.
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/MongoEngine/mongoengine/issues/560#issuecomment-35483532
> .

Any updates on this bug? I've just found out I'm also affected by id

I also get no results when querying GenericReferenceFields anymore. +1 on the reversion by @cybertoast as a quick fix. The SON wrapping may have some purpose, but there is a need for changes somewhere else in the code if SON is necessary there. @rozza I've spent the day trying to upgrade to 0.9.0 on my way to adopting the latest mongoengine, and I ran into this issue as well as the upgrade directions missing an important step as I note here: https://groups.google.com/d/msg/mongoengine-users/TG-fokDi_m8/KpQQLyDlCwAJ. 

any progress on this?

I'm planning to work on this soon. The notes below are mostly me thinking through the problem. Feel free to ignore or provide feedback about the implementation details (listed at the bottom).

-----

First, let me give a bit of extra context on how `GenericReferenceField`s are stored and queried:
```
In [21]: class Doc(Document):
        ref = GenericReferenceField()
   ....:

In [22]: doc1 = Doc.objects.create()

In [23]: doc2 = Doc.objects.create(ref=doc1)

In [26]: doc2.to_mongo()
Out[26]: SON([('_id', ObjectId('584494a8c6e47b875d602773')), ('ref', SON([('_cls', 'Doc'), ('_ref', DBRef('doc', ObjectId('584494a1c6e47b875d602772')))]))])

In [28]: Doc._get_collection().find_one({ '_id': doc2.pk })
Out[28]:
{u'_id': ObjectId('584494a8c6e47b875d602773'),
 u'ref': {u'_cls': u'Doc',
  u'_ref': DBRef(u'doc', ObjectId('584494a1c6e47b875d602772'))}}
```

As you can see, a generic reference is stored as an object containing:
```
_cls: "NameOfTheMongoEngineDocClass",  // this is crucial for dereferencing into the right MongoEngine document class
_ref: DBRef("collection_name", idOfTheDoc)  // this is crucial for uniquely identifying the referenced doc
```

When you filter a queryset by a generic reference, we get the following query:
```
In [42]: Doc.objects.filter(ref=doc1)._query
Out[42]: {'ref': SON([('_cls', 'Doc'), ('_ref', DBRef('doc', ObjectId('584494a1c6e47b875d602772')))])}
```

Same structure, again.

This ticket argues that two extra filtering options should be possible:
1. `Doc.objects.filter(ref=doc.pk)` (aka filtering by `ObjectId`).
2. `Doc.objects.filter(ref=DBRef('doc', doc.pk))` (aka filtering by a `DBRef`).

Technically, the 1st option could cause issues, because there's no mechanism in MongoDB that prevents the same ID from existing in multiple collections. That being said, given the implementation details of an `ObjectId` (https://docs.mongodb.com/v3.2/reference/bson-types/#objectid), it's **extremely** unlikely that we'd encounter conflicts.

Given all of the above, my proposed solution is that the query should resolve into:
* `"ref._ref.$id": value` if `value` is an `ObjectId`.
* `"ref._ref": value` if `value` is a `DBRef`.
* `ref: value` if `value` is a `BaseDocument`.

Resolving patch:
diff --git a/mongoengine/fields.py b/mongoengine/fields.py
--- a/mongoengine/fields.py
+++ b/mongoengine/fields.py
@@ -1249,7 +1249,7 @@ def to_mongo(self, document):
         if document is None:
             return None
 
-        if isinstance(document, (dict, SON)):
+        if isinstance(document, (dict, SON, ObjectId, DBRef)):
             return document
 
         id_field_name = document.__class__._meta['id_field']
diff --git a/mongoengine/queryset/transform.py b/mongoengine/queryset/transform.py
--- a/mongoengine/queryset/transform.py
+++ b/mongoengine/queryset/transform.py
@@ -1,6 +1,7 @@
 from collections import defaultdict
 
-from bson import SON
+from bson import ObjectId, SON
+from bson.dbref import DBRef
 import pymongo
 
 from mongoengine.base.fields import UPDATE_OPERATORS
@@ -26,6 +27,7 @@
                    STRING_OPERATORS + CUSTOM_OPERATORS)
 
 
+# TODO make this less complex
 def query(_doc_cls=None, **kwargs):
     """Transform a query from Django-style format to Mongo format.
     """
@@ -62,6 +64,7 @@ def query(_doc_cls=None, **kwargs):
             parts = []
 
             CachedReferenceField = _import_class('CachedReferenceField')
+            GenericReferenceField = _import_class('GenericReferenceField')
 
             cleaned_fields = []
             for field in fields:
@@ -101,6 +104,16 @@ def query(_doc_cls=None, **kwargs):
                 # 'in', 'nin' and 'all' require a list of values
                 value = [field.prepare_query_value(op, v) for v in value]
 
+            # If we're querying a GenericReferenceField, we need to alter the
+            # key depending on the value:
+            # * If the value is a DBRef, the key should be "field_name._ref".
+            # * If the value is an ObjectId, the key should be "field_name._ref.$id".
+            if isinstance(field, GenericReferenceField):
+                if isinstance(value, DBRef):
+                    parts[-1] += '._ref'
+                elif isinstance(value, ObjectId):
+                    parts[-1] += '._ref.$id'
+
         # if op and op not in COMPARISON_OPERATORS:
         if op:
             if op in GEO_OPERATORS:
@@ -128,11 +141,13 @@ def query(_doc_cls=None, **kwargs):
 
         for i, part in indices:
             parts.insert(i, part)
+
         key = '.'.join(parts)
+
         if op is None or key not in mongo_query:
             mongo_query[key] = value
         elif key in mongo_query:
-            if key in mongo_query and isinstance(mongo_query[key], dict):
+            if isinstance(mongo_query[key], dict):
                 mongo_query[key].update(value)
                 # $max/minDistance needs to come last - convert to SON
                 value_dict = mongo_query[key]


Test patch:
diff --git a/tests/fields/fields.py b/tests/fields/fields.py
--- a/tests/fields/fields.py
+++ b/tests/fields/fields.py
@@ -2810,6 +2810,38 @@ class User(Document):
         Post.drop_collection()
         User.drop_collection()
 
+    def test_generic_reference_filter_by_dbref(self):
+        """Ensure we can search for a specific generic reference by
+        providing its ObjectId.
+        """
+        class Doc(Document):
+            ref = GenericReferenceField()
+
+        Doc.drop_collection()
+
+        doc1 = Doc.objects.create()
+        doc2 = Doc.objects.create(ref=doc1)
+
+        doc = Doc.objects.get(ref=DBRef('doc', doc1.pk))
+        self.assertEqual(doc, doc2)
+
+    def test_generic_reference_filter_by_objectid(self):
+        """Ensure we can search for a specific generic reference by
+        providing its DBRef.
+        """
+        class Doc(Document):
+            ref = GenericReferenceField()
+
+        Doc.drop_collection()
+
+        doc1 = Doc.objects.create()
+        doc2 = Doc.objects.create(ref=doc1)
+
+        self.assertTrue(isinstance(doc1.pk, ObjectId))
+
+        doc = Doc.objects.get(ref=doc1.pk)
+        self.assertEqual(doc, doc2)
+
     def test_binary_fields(self):
         """Ensure that binary fields can be stored and retrieved.
         """