Repository: MongoEngine/mongoengine

Issue:
[BUG] EmbeddedDocument.__hash__ is None.
python: 3.6.0
mongoengine: 0.12.0

``` python
from mongoengine import EmbeddedDocument
type(EmbeddedDocument.__hash__) // return <class 'NoneType'>
```

![picpick-20170410-041010-001](https://cloud.githubusercontent.com/assets/10906962/24840668/9fb31e9e-1da3-11e7-81c2-c3c78164dc57.png)



Discussion:
Hi @Cologler, thanks for reporting this. The reason for this is that in Python 3.x "a class that overrides `__eq__()` and does not define `__hash__()` will have its `__hash__()` implicitly set to `None`" (https://docs.python.org/3.6/reference/datamodel.html#object.__hash__). In Python 2, `EmbeddedDocument` simply inherited the `__hash__` from `BaseDocument`. In Python 3 it doesn't. The simplest fix would be re-define the `__hash__` on the `EmbeddedDocument` (thus making Python 2 and 3 consistent).

That being said, I think the current behavior of the `BaseDocument.__hash__` is fundamentally flawed and we should consider what its higher-level behavior should be. Currently, it's invalid/confusing in Python 2:

```
In [29]: class EmbeddedDoc(EmbeddedDocument):
    ...:     txt = StringField()
    ...:

In [30]: a = EmbeddedDoc(txt='abc')

In [31]: b = EmbeddedDoc(txt='def')

In [32]: a == b  # ok, this is expected
Out[32]: False

In [33]: a.__hash__() == b.__hash__()  # this is expected as well
Out[33]: False

In [34]: b.txt = 'abc'

In [35]: a == b  # alright, they are indeed equal, holding exactly the same values
Out[35]: True

In [36]: a.__hash__() == b.__hash__()  # wait, this is unexpected, aren't they identical?
Out[36]: False

In [37]: set([a, b])  # given that they're holding exactly the same data, shouldn't the set contain just one element?
Out[37]: {<EmbeddedDoc: EmbeddedDoc object>, <EmbeddedDoc: EmbeddedDoc object>}

In [39]: a.pk = 'a'  # let's set a random "pk" value on both embedded docs

In [40]: b.pk = 'a'

In [41]: a.__hash__()
Out[41]: 12416037344

In [42]: b.__hash__()  # ha, now their hashes are equal...
Out[42]: 12416037344

In [43]: set([a, b])  # ...and they're recognized as identical, because of a randomly assigned "pk" field
Out[43]: {<EmbeddedDoc: EmbeddedDoc object>}
```

The problem is that `BaseDocument.__hash__` is actually a valid implementation *only* for `Document` instances (and should probably be moved there). They can be uniquely identified by their IDs. On an `EmbeddedDocument`, the `pk` field doesn't hold any significance, so its `__hash__` implementation is broken and, by accident, Python 3 fixed this problem for us :)

I believe the best way forward is actually to make sure `__hash__` doesn't exist on `EmbeddedDocument`s in both Python 2 and 3. As the [docs](https://docs.python.org/3.6/reference/datamodel.html#object.__hash__) say:

> If a class defines mutable objects and implements an `__eq__()` method, it should not implement `__hash__()`, since the implementation of hashable collections requires that a key’s hash value is immutable (if the object’s hash value changes, it will be in the wrong hash bucket).

There is absolutely nothing that's globally immutable on embedded documents, and so I think we have to leave it to the developers to override the `__hash__` in a way that's specific to their use case/data model.

Resolving patch:
diff --git a/mongoengine/base/document.py b/mongoengine/base/document.py
--- a/mongoengine/base/document.py
+++ b/mongoengine/base/document.py
@@ -272,13 +272,6 @@ def __eq__(self, other):
     def __ne__(self, other):
         return not self.__eq__(other)
 
-    def __hash__(self):
-        if getattr(self, 'pk', None) is None:
-            # For new object
-            return super(BaseDocument, self).__hash__()
-        else:
-            return hash(self.pk)
-
     def clean(self):
         """
         Hook for doing document level data cleaning before validation is run.
diff --git a/mongoengine/document.py b/mongoengine/document.py
--- a/mongoengine/document.py
+++ b/mongoengine/document.py
@@ -60,6 +60,12 @@ class EmbeddedDocument(BaseDocument):
     my_metaclass = DocumentMetaclass
     __metaclass__ = DocumentMetaclass
 
+    # A generic embedded document doesn't have any immutable properties
+    # that describe it uniquely, hence it shouldn't be hashable. You can
+    # define your own __hash__ method on a subclass if you need your
+    # embedded documents to be hashable.
+    __hash__ = None
+
     def __init__(self, *args, **kwargs):
         super(EmbeddedDocument, self).__init__(*args, **kwargs)
         self._instance = None
@@ -160,6 +166,15 @@ def pk(self, value):
         """Set the primary key."""
         return setattr(self, self._meta['id_field'], value)
 
+    def __hash__(self):
+        """Return the hash based on the PK of this document. If it's new
+        and doesn't have a PK yet, return the default object hash instead.
+        """
+        if self.pk is None:
+            return super(BaseDocument, self).__hash__()
+        else:
+            return hash(self.pk)
+
     @classmethod
     def _get_db(cls):
         """Some Model using other db_alias"""


Test patch:
diff --git a/tests/document/instance.py b/tests/document/instance.py
--- a/tests/document/instance.py
+++ b/tests/document/instance.py
@@ -2164,7 +2164,7 @@ class User(Document):
         class BlogPost(Document):
             pass
 
-        # Clear old datas
+        # Clear old data
         User.drop_collection()
         BlogPost.drop_collection()
 
@@ -2176,17 +2176,18 @@ class BlogPost(Document):
         b1 = BlogPost.objects.create()
         b2 = BlogPost.objects.create()
 
-        # in List
+        # Make sure docs are properly identified in a list (__eq__ is used
+        # for the comparison).
         all_user_list = list(User.objects.all())
-
         self.assertTrue(u1 in all_user_list)
         self.assertTrue(u2 in all_user_list)
         self.assertTrue(u3 in all_user_list)
-        self.assertFalse(u4 in all_user_list)  # New object
-        self.assertFalse(b1 in all_user_list)  # Other object
-        self.assertFalse(b2 in all_user_list)  # Other object
+        self.assertTrue(u4 not in all_user_list)  # New object
+        self.assertTrue(b1 not in all_user_list)  # Other object
+        self.assertTrue(b2 not in all_user_list)  # Other object
 
-        # in Dict
+        # Make sure docs can be used as keys in a dict (__hash__ is used
+        # for hashing the docs).
         all_user_dic = {}
         for u in User.objects.all():
             all_user_dic[u] = "OK"
@@ -2198,9 +2199,20 @@ class BlogPost(Document):
         self.assertEqual(all_user_dic.get(b1, False), False)  # Other object
         self.assertEqual(all_user_dic.get(b2, False), False)  # Other object
 
-        # in Set
+        # Make sure docs are properly identified in a set (__hash__ is used
+        # for hashing the docs).
         all_user_set = set(User.objects.all())
         self.assertTrue(u1 in all_user_set)
+        self.assertTrue(u4 not in all_user_set)
+        self.assertTrue(b1 not in all_user_list)
+        self.assertTrue(b2 not in all_user_list)
+
+        # Make sure duplicate docs aren't accepted in the set
+        self.assertEqual(len(all_user_set), 3)
+        all_user_set.add(u1)
+        all_user_set.add(u2)
+        all_user_set.add(u3)
+        self.assertEqual(len(all_user_set), 3)
 
     def test_picklable(self):
         pickle_doc = PickleTest(number=1, string="One", lists=['1', '2'])